​
Requirements

  • Python 3.12 or higher

​
Install from PyPI

pip install ai-blackteam
That’s it. Verify it worked: 
ai-blackteam --help
You should see the full command list - run, batch, benchmark, scorecard, report, and more.

​
Install from source

If you want the latest development version or plan to contribute: 
git clone https://github.com/BILLKISHORE/ai-evals.git
cd ai-evals
pip install -e .
The -e flag installs in editable mode so changes to the source take effect immediately.

​
Verify installation

ai-blackteam --help
Expected output includes all available commands: 
Usage: ai-blackteam [OPTIONS] COMMAND [ARGS]...

Commands:
  run          Run a single attack
  batch        Run all attacks against a model
  benchmark    Run the full safety benchmark
  scorecard    Generate compliance scorecards
  report       Generate reports
  config       Manage configuration
  ...

​
Next step

Set up your first provider and run a scan.

​
Troubleshooting

ai-blackteam requires Python 3.12 or higher. Check your version:
python --version
If you see anything below 3.12, install a newer Python via python.org, Homebrew (brew install python@3.12), or pyenv install 3.12.
Most install errors on macOS come from an outdated pip or build toolchain. Upgrade pip first:
pip install --upgrade pip setuptools wheel
pip install ai-blackteam
If that still fails, try uv, which sidesteps most dependency-resolution issues:
uv pip install ai-blackteam
The package installed, but the entry point is not on your PATH. Two reliable workarounds:
python -m ai_blackteam --help
Or install into your user site so the script lands in ~/.local/bin:
pip install --user ai-blackteam
export PATH="$HOME/.local/bin:$PATH"
Environment variables take precedence over config files. Verify the var is exported in the same shell:
echo $ANTHROPIC_API_KEY
If you prefer a persisted config instead of env vars, use the CLI:
ai-blackteam config set providers.anthropic.api_key sk-ant-xxxx
ai-blackteam stores run history under ~/.ai-blackteam/. If you see a permission or path error, recreate the directory:
rm -rf ~/.ai-blackteam
mkdir -p ~/.ai-blackteam
ai-blackteam run --provider mock --attack jailbreak.basic
Check that your home directory is writable and not on a read-only mount.
This usually means the CLI and your interpreter are out of sync (installed into a venv but resolved from system Python, or vice versa). Confirm both point at the same prefix:
which python
which ai-blackteam
python -c "import ai_blackteam; print(ai_blackteam.__file__)"
If the paths disagree, activate the right venv or reinstall inside the active interpreter.
Set the standard proxy variables before running pip and the CLI:
export HTTPS_PROXY=http://proxy.corp.example:8080
export HTTP_PROXY=http://proxy.corp.example:8080
pip install ai-blackteam
If outbound traffic to provider APIs is fully blocked, the mock provider still works end-to-end offline:
ai-blackteam run --provider mock --attack jailbreak.basic
Open an issue at github.com/BILLKISHORE/ai-evals/issues with:
ai-blackteam --version
python --version
pip show ai-blackteam
Include the exact command you ran and the full error output.