TEST: secureai test

Fires adversarial payloads at any text chat model through an OpenAI-compatible endpoint or the SecureAI model proxy, then reports the Attack Success Rate.

Terminal

secureai test --target <chat-completions-url> [flags]

Only test systems you are authorised to test

This sends real attack traffic to a live endpoint and consumes tokens against your provider account. Point it at your own staging or production system, with your organisation's approval. Never point it at a third party's system.

Testing any text model

OpenAI-compatible endpoints work directly. For Anthropic, Gemini, Ollama, or a private JSON API, start the model proxy and point TEST at its local endpoint. The model owner must configure the upstream URL, model name, authentication environment variable, and any custom request or response mapping.

Terminal

secureai proxy --config secureai.provider.yaml

# In another terminal
secureai test --target http://127.0.0.1:8090/v1/chat/completions \
  --model proxied-model --intensity passive

Common uses

Terminal

# Standard run against a staging endpoint
secureai test --target https://staging.example.com/v1/chat/completions \
  --api-key "$OPENAI_API_KEY"

# Quick, low-volume pass suitable for a pull request
secureai test --target "$ENDPOINT" --api-key "$KEY" --intensity passive

# Full corpus, fail the build above 2% ASR
secureai test --target "$ENDPOINT" --api-key "$KEY" \
  --intensity full --fail-asr 2

# Respect a provider rate limit
secureai test --target "$ENDPOINT" --api-key "$KEY" --rps 5 --concurrency 2

How a run works

Reconnaissance: probes the endpoint to learn how it responds, refuses, and formats output.
Attack: sends the payload corpus across categories such as role confusion, instruction negation, policy bypass, output escape, and multi-turn coercion.
Verification: checks each response for evidence the attack actually succeeded, rather than counting anything that merely looks unusual.

Payloads are fetched, not bundled

The attack corpus is not compiled into the CLI. Each run fetches it from the control plane over an authenticated request, and only developers and security researchers are served it. Run secureai login first. See Authentication.

Flags

Flag	Default	Description
`--target`	required	Chat-completions endpoint URL.
`--api-key`	none	Sent as `Authorization: Bearer`.
`--auth-bearer`	none	Explicit bearer token; overrides `--api-key`.
`--header`	none	Extra HTTP header as `'Key: Value'`. Repeatable.
`--model`	`gpt-4o-mini`	Model name sent in requests.
`--intensity`	`standard`	Corpus size: `passive\|standard\|full`.
`--fail-asr`	`5`	Exit non-zero if overall ASR exceeds this percentage.
`--concurrency`	`4`	Parallel workers, 1-20.
`--rps`	`0`	Max requests per second. 0 means unlimited.
`--format`	`terminal`	Output format: `terminal\|json`.
`--payloads`	none	Custom payload YAML merged with the fetched library.
`--payloads-csv`	none	CoreLayer Striker CSV of attack payloads to merge.
`--allow-store-responses`	off	Retain raw model responses in the report.
`--no-bus`	off	Ignore BUILD events and do not emit TEST events to the local bus.

Choosing an intensity

Intensity	Use it	Trade-off
`passive`	On pull requests	Fast and cheap; catches the obvious regressions only.
`standard`	Nightly or pre-release	The default balance of coverage and cost.
`full`	Before a major release, or a scheduled audit	Highest coverage; slowest and most tokens.

Reading the result

Terminal

TEST: adversarial probing
target: https://staging.example.com/v1/chat/completions
intensity: standard · 250 payloads · 4 workers

CATEGORY                SENT   OK   ASR
Role Confusion            60   11  18.3%   HIGH    LLM01
Instruction Negation      60    9  15.0%   HIGH    LLM01
Policy Bypass             50    6  12.0%   MEDIUM  LLM01
Output Escape             40    3   7.5%   MEDIUM  LLM05
Multi-Turn Coercion       40    1   2.5%   LOW     LLM01

overall ASR: 12.0%  (30 of 250)
gate: fail-asr=5 → FAILED

ASR is the percentage of payloads that succeeded. Lower is safer. This runs in the opposite direction to the BUILD and VALIDATE scores.
Per-category ASR tells you where to spend effort: a high role-confusion rate and a low output-escape rate are different fixes.
A nonzero ASR is not automatically a release blocker. Set --fail-asr to whatever your organisation treats as acceptable.

Response handling

By default raw model responses are not retained. Only the verdict, category, and severity. Pass --allow-store-responses when you need the actual text as evidence, and be aware those responses may contain whatever the attack managed to extract, including data from your own system prompt.

Generating a CI workflow

Scaffold a GitHub Actions workflow that runs the attack engine:

Terminal

secureai test init-ci

See CI integration for token setup and gating advice.

To combine TEST with BUILD, VALIDATE, and RUNTIME, use secureai run.