red-team-frameworks_skill

Overview

This skill bundles tools and frameworks for automated AI red teaming, including PyRIT, garak, Counterfit, ART, and TextAttack. It provides orchestration patterns, probe libraries, and adapters to run multi-turn attacks, vulnerability scans, and adversarial ML tests across LLM and ML targets. The aim is repeatable, auditable assessments that map findings to risk frameworks like OWASP and MITRE ATLAS.

How this skill works

The skill orchestrates tests by registering framework adapters and running framework-specific workflows: PyRIT for multi-turn prompt orchestration, garak for probe-based vulnerability scanning, Counterfit/ART for adversarial example generation on vision and tabular models, and TextAttack for targeted NLP perturbations. Results are normalized into a unified report with severity, evidence, OWASP mapping, and recommended remediation. Hooks support CI/CD integration, custom probes, and isolated test endpoints.

When to use it

• Assess enterprise LLM deployments for prompt injection, jailbreaks, and information disclosure
• Run CI/CD vulnerability scans against model endpoints before production rollout
• Evaluate robustness of image, tabular, or NLP models to adversarial examples
• Perform repeatable multi-framework red team engagements with centralized reporting
• Develop and validate custom probes or attack recipes for specific threat models

Best practices

• Define clear rules of engagement and obtain written authorization before testing
• Use isolated test environments and dedicated test accounts; never attack production without approval
• Rate-limit requests, implement exponential backoff, and monitor for unintended side effects
• Log and secure all attack artifacts and sanitize any extracted data
• Pin framework versions, use virtual environments, and document reproducible configurations

Example use cases

• Run a PyRIT crescendo campaign to evaluate multi-turn jailbreak resilience on an Azure OpenAI chat deployment
• Execute a garak full probe suite in CI to catch regressions after model or prompt changes
• Generate adversarial images with Counterfit and ART to measure classifier robustness and average perturbation
• Use TextAttack recipes to measure NLP model tolerance to word- and character-level perturbations
• Combine frameworks in a unified run to produce a single consolidated vulnerability report mapped to OWASP LLM risks