rag-exploitation_skill

Overview

This skill provides practical attack techniques and tests for Retrieval-Augmented Generation (RAG) systems, focusing on retrieval, ranking, embedding, and generation weaknesses. It consolidates methods for knowledge base poisoning, retrieval manipulation, context injection, and embedding attacks to evaluate RAG security. It is intended for red teamers, security engineers, and developers who need to assess persistent misinformation and retrieval-based vulnerabilities.

How this skill works

The skill implements adversarial workflows that craft poisoned documents, adversarial queries, hidden-context payloads, and embedding collisions. It inspects the embedding/retrieval pipeline by generating target-aligned content or queries, injecting into vector stores (when possible), and validating whether malicious artifacts are retrieved and influence generation. It also includes checks for similarity thresholds, ranking manipulation, and prompt-template resilience.

When to use it

• Assess risk of persistent misinformation in knowledge bases (KB poisoning).
• Evaluate retrieval robustness and ranking manipulation from query-only access.
• Test whether retrieved context can inject or override model instructions.
• Validate embedding-space defenses against collisions and inversion attacks.
• Simulate real-world red team scenarios for supply-chain or data-poisoning threats.

Best practices

• Run attacks in isolated test environments and never against production user data.
• Confirm and enforce strict write access controls and content validation for any KB.
• Monitor similarity scores and tune thresholds to reduce unintended retrievals.
• Sanitize and normalize retrieved content before concatenating into prompts.
• Regularly audit embeddings and apply anomaly detection on new vectors to detect collisions or unusual clusters.

Example use cases

• Simulate KB poisoning by crafting documents that embed near high-value queries and verify persistent retrieval.
• Probe retrieval by crafting adversarial queries that force specific documents into top-k results.
• Test context injection by adding hidden or structured payloads to documents and observing model responses.
• Perform embedding collision experiments to check whether different texts map to the same embedding and bypass deduplication.
• Use the vulnerability checklist to prioritize remediation and classify severity of discovered issues.