​
Requirements

  • Python 3.12 or higher

​
Install from PyPI

That’s it. Verify it worked:
You should see the full command list - run, batch, benchmark, scorecard, report, and more.

​
Install from source

If you want the latest development version or plan to contribute:
The -e flag installs in editable mode so changes to the source take effect immediately.

​
Verify installation

Expected output includes all available commands:

​
Next step

Set up your first provider and run a scan.

​
Troubleshooting

ai-blackteam requires Python 3.12 or higher. Check your version:
If you see anything below 3.12, install a newer Python via python.org, Homebrew (brew install python@3.12), or pyenv install 3.12.
Most install errors on macOS come from an outdated pip or build toolchain. Upgrade pip first:
If that still fails, try uv, which sidesteps most dependency-resolution issues:
The package installed, but the entry point is not on your PATH. Two reliable workarounds:
Or install into your user site so the script lands in ~/.local/bin:
Environment variables take precedence over config files. Verify the var is exported in the same shell:
If you prefer a persisted config instead of env vars, use the CLI:
ai-blackteam stores run history under ~/.ai-blackteam/. If you see a permission or path error, recreate the directory:
Check that your home directory is writable and not on a read-only mount.
This usually means the CLI and your interpreter are out of sync (installed into a venv but resolved from system Python, or vice versa). Confirm both point at the same prefix:
If the paths disagree, activate the right venv or reinstall inside the active interpreter.
Set the standard proxy variables before running pip and the CLI:
If outbound traffic to provider APIs is fully blocked, the mock provider still works end-to-end offline:
Open an issue at github.com/BILLKISHORE/ai-evals/issues with:
Include the exact command you ran and the full error output.