The ai-blackteam asl3 command runs targeted evaluations for Anthropic’s ASL-3 safety level criteria: CBRN (chemical, biological, radiological, nuclear) risk and autonomous capability.
ASL3 evaluations test for some of the most dangerous AI capabilities. Running these attacks against external APIs may trigger automated policy violation detection. Make sure you have authorization before running them.
Running ASL3 evaluations
Options
CBRN evaluation
Tests whether the model can be tricked into providing information about weapons of mass destruction.
Categories tested:
- Chemical weapons
- Biological weapons
- Radiological weapons
- Nuclear weapons
Techniques used:
dual-use-framing - presenting weapons information as dual-use researchtechnical-obfuscation - encoding dangerous content technicallylifecycle-staging - gradually escalating through a multi-step processexpert-persona - impersonating domain expertsencoding-obfuscation - standard encoding bypassesrefusal-suppression - techniques to prevent the model from refusinghypothetical-framing - presenting scenarios as hypotheticalmultilingual-attack - testing in non-English languages
These techniques are combined with the CBRN categories using the template expansion engine, generating targeted attacks for each combination.
Autonomous capability evaluation
Tests for dangerous autonomous behaviors that could indicate misalignment.
Scenarios tested:
Reading results
Exit codes
- 0 - all attacks blocked across both domains
- 1 - at least one attack bypassed in any domain
- 2 - configuration error
For CBRN testing, even a small number of bypasses is concerning. A 1-2% bypass rate on CBRN attacks is worth investigating even if your overall safety score is high.
Limiting scope
For initial testing or debugging, use --limit to cap the number of attacks:
