The ai-blackteam benchmark command runs a structured safety evaluation and produces a numerical score. It tests a curated set of harmful targets across all harm categories, running every attack technique against each target.

​
Single model benchmark

This runs the full benchmark suite and outputs:
  • Safety Score - a single percentage (higher = safer)
  • Bypassed / Blocked / Partial counts
  • Category breakdown - per-category scores
  • OWASP LLM Top 10 scorecard - mapped to OWASP categories

​
All configured providers

Benchmarks every provider that has an API key configured. Produces a leaderboard and a category comparison matrix.

​
Specific model pairs

Test specific provider:model pairs. Useful when you want to compare specific versions.

​
Options

​
Category breakdown

For single-model benchmarks, you get a category scores table:
Categories with lower scores tell you where the model is weakest.

​
Multi-model comparison

When testing multiple models, the output includes: Safety Leaderboard - models ranked by overall score:
Category Comparison - per-category scores across all models:

​
OWASP scorecard

Every benchmark run automatically produces an OWASP LLM Top 10 scorecard for each model. This shows block rates per OWASP category (LLM01-LLM10).

​
Saving results

The JSON output includes all scores, category breakdowns, and metadata needed for automation.

​
Threshold gates

See Benchmark Thresholds for details on using thresholds in CI.

​
Testing specific categories

This is useful when you want to focus on high-risk categories without running the full suite.