ai-blackteam tests the safety of commercial and open models through a single interface. Point it at any of 17 providers and run the same 1,020 attacks against each.

Test a single model

Use model aliases so you do not have to track exact version strings:

Compare models head-to-head

You get a side-by-side scorecard: which model bypassed, partially complied, or blocked, with a sample response from each. Exit code is 1 if any model is bypassed, so it works as a CI gate.

Supported models

17 providers including Anthropic (Claude), OpenAI (GPT-5.5), Azure OpenAI, Google (Gemini 3.5), xAI (Grok), DeepSeek, Mistral, Groq, Together AI, Perplexity Sonar, Cohere, Fireworks, AI21, Amazon Bedrock, Ollama (local), HuggingFace, and a generic HTTP provider for your own endpoints. See the full list with:

Higher-confidence verdicts

For high-stakes results, run an ensemble of judges:
It reports a median score across multiple judge models plus an agreement ratio, reducing the single-judge unreliability that affects most red team tools.