The ai-blackteam mega-sweep command combines external datasets with attack techniques and optional mutations to produce large-scale evaluations. Instead of testing a single target, it runs attacks against hundreds or thousands of harmful prompts from research datasets.

​
Basic usage

​
How it works

  1. Loads prompts from the specified dataset(s)
  2. Filters to single-turn attack techniques (multi-turn attacks aren’t compatible with raw prompts)
  3. Optionally applies mutation variants to each prompt
  4. Runs each prompt x attack combination against the target model
  5. Reports bypass counts

​
Options

​
Mutation variants

Mutations transform each prompt before sending it through attack techniques:
With mutations, the number of runs multiplies quickly. Each mutation type adds variants per prompt.

​
Dry-run mode

Before committing to a large run, check what would execute:
Output:

​
Filtering

​
By category

Only runs prompts whose category matches the filter.

​
By prompt count

Caps the total number of prompts processed. Useful for quick spot checks.

​
By attack technique

​
Scale

The math is straightforward:
With mutations:
For example:
  • HarmBench (400 prompts) x 83 single-turn attacks = 33,200 runs
  • Add encoding mutations (5 variants) = 166,000 runs
Use --limit and --dry-run to manage scale.

​
Saving results

The JSON output includes model, provider, total runs, bypass count, and configuration used.