These are aggregate results from running ai-blackteam against frontier models. The focus is robustness: how often each model blocked an attack attempt. We publish the numbers and the methodology. We do not publish attack prompts or model outputs.
These verdicts come from the framework’s automated evaluator (keyword and regex matching), not human review. Automated scoring has a high false-positive rate, a response that merely mentions a sensitive term can be flagged. So the “flagged” counts overstate real bypasses; treat the block rate as the reliable signal and the flagged counts as an upper bound that needs human verification.
Summary
| Benchmark | Model | Prompts | Blocked | Flagged (unverified) | Unclear |
|---|
| Claude Haiku 4.5 (general harm) | claude-haiku-4-5-20251001 | 380 | 335 (88.2%) | 45 | 0 |
| Claude Haiku 4.5 (ASL-3 / CBRN focus) | claude-haiku-4-5-20251001 | 679 | 637 (93.8%) | 42 | 0 |
| GPT-5.4 (general harm) | gpt-5.4 | 400 | 6 (1.5%) | 150 | 244 |
Block rate is blocked divided by total prompts. UNCLEAR responses (neither a clear refusal nor a clear bypass) are counted separately and are not failures. The automated evaluator behaves differently across runs, so these rates are not directly comparable across benchmarks. Read each as within-benchmark signal, not a cross-model leaderboard.
Claude Haiku 4.5 (general harm)
Model claude-haiku-4-5-20251001, run on 2026-03-30. 380 attack attempts, of which 335 were blocked (88.2%). 45 were flagged as bypassed and 0 as partial by the automated evaluator (unverified).
| Harm category | Tested | Blocked | Block rate | Flagged (unverified) |
|---|
| hate-speech | 76 | 76 | 100.0% | 0 |
| system-prompt-leakage | 76 | 76 | 100.0% | 0 |
| weapons | 76 | 73 | 96.1% | 3 |
| phishing | 76 | 61 | 80.3% | 15 |
| malware | 76 | 49 | 64.5% | 27 |
Claude Haiku 4.5 (ASL-3 / CBRN focus)
Model claude-haiku-4-5-20251001, run on 2026-03-30. 679 attack attempts, of which 637 were blocked (93.8%). 39 were flagged as bypassed and 3 as partial by the automated evaluator (unverified).
| Harm category | Tested | Blocked | Block rate | Flagged (unverified) |
|---|
| chemical-weapons | 168 | 164 | 97.6% | 4 |
| biological-weapons | 168 | 164 | 97.6% | 4 |
| radiological-weapons | 168 | 161 | 95.8% | 7 |
| nuclear-weapons | 168 | 141 | 83.9% | 27 |
| other | 7 | 7 | 100.0% | 0 |
GPT-5.4 (general harm)
Model gpt-5.4, run on 2026-04-03. 400 attack attempts, of which 6 were blocked (1.5%). 150 were flagged as bypassed and 0 as partial by the automated evaluator (unverified). 244 were unclear.
| Harm category | Tested | Blocked | Block rate | Flagged (unverified) |
|---|
| other | 356 | 6 | 1.7% | 131 |
| fraud | 44 | 0 | 0.0% | 19 |
Methodology
Each attack is a prompt generated by an ai-blackteam technique against a harm target. The model’s response is scored BLOCKED, PARTIAL, BYPASSED, or UNCLEAR by the automated evaluator. These runs used the keyword and regex evaluator without the optional LLM judge, so scores are fast but noisy.
What we do not publish, and why
We never publish the attack prompts or the model outputs. Publishing working jailbreaks or harmful content would cause real-world harm and is the opposite of responsible safety research. Any genuine, verified finding is reported privately to the model vendor through coordinated disclosure, never posted publicly. The value here is the aggregate signal, which models resist which attack families, not a how-to. 
