Research Initiative: AI Red Teaming & Evaluation

Red Teaming: The Power of Adversarial Thinking in AI Security

(AI hackers, tech wizards, and code sorcerers, we need you!)

This is your invitation and an opportunity for you to flex your hacker muscles and dive into the murky waters of Large Language Model (LLM) vulnerabilities. We’re putting together a team to map and tackle the OWASP Top Ten vulnerabilities for LLM applications, and we want you on board. Yes, you, the one with the hoodie and the suspiciously fast typing speed.

Why Should You Get Involved?

• Show Off Your Skills: This is your chance to be the star of the show, to let your brilliance shine! Your expertise is needed for an audience of CISOs, risk advisors, developers, and threat analysts focused on defending LLM for everyday people and organizations. 
• Join the Cool Kids Club: Collaborate, challenge, and charm your way through complex problems with like-minded geniuses. (It’s like The Avengers, but for nerds.)
• Make a Real Impact: Your work will help prevent LLM badness happening to everyday people, businesses, and organizations who would like to use LLMs but have no idea about the gotchas. Think of it as superhero work, but without the spandex.

What’s This All About?

• Our mission is to crack the “how” of AI Red Teaming and LLM evaluations. We aren’t here just to talk about AI problems; we’re here to help decipher the noise from real issues that can cause harm. This is where you come in! Help us identify, quantify, and prevent the vulnerabilities we face in the rapidly growing AI space.
• AI Red Teaming Methodology: Your insights will contribute to creating a standardized approach for AI red teaming, covering everything from testing methodologies to real world vulnerability examples.
• Standardized Evaluations: We’re creating the gold standard for AI testing, from metrics and benchmarks to datasets and tools, you’ll help build the gold standard for AI security evaluations.

How Can You Contribute?

• Find the Exploitable Bits: Help us uncover and document real-world examples of LLM vulnerabilities.
• Share Your Knowledge: Reveal how you test AI systems and the methodologies behind your work.
• Build the Tools: Contribute to the development of tools and benchmarks that AI red teamers everywhere will rely on.

🦸🏻🦸🏻‍♀️🦸🏿‍♂️🦸‍♀️ Not all heroes wear capes, some of them debug code (and red team LLMs)!

Initiative Collaboration and Participation

The OWASP Foundation is an open source neutral organization. Projects and initiatives are open to experts who want to contribute their experience and expertise. The OWASP Top 10 for LLMs AI Red Teaming & Evaluation Guidelines Initiative will have an established working team and we invite anyone with expertise to join the group to collaborate.

Joining the conversation 

• Join  OWASP Slack 
• Follow the #team-llm-redteam channel 
• The Red Teaming Initiative Working Group  Biweekly Calls start  Wednesday, September 18 · 9:00 – 10:00 am Time zone: America/Los_Angeles Google Meet Link

More Questions?

• Contact Team Leads Krishna Sankar or Sandy Dunn on OWASP Slack or Email:  Krishna Sankar  or Sandy Dunn 

The Challenge

The purpose of the OWASP Top 10 for LLMs  AI Red Teaming & Evaluation Guidelines Initiative is to help define the methodologies and standardize AI Red Teaming methodologies, test cases,  responsible disclosure, remediation and the interpretation and scoring of the results.

Initiative overview 

Our goal is to provide empirical evidence for evaluating standards and requirements. This includes providing an understanding of what guardrails are doing or not doing effectively.  AI Red Teaming serves as both a test of a model and a test of the model safeguards.

Project Goals

• Generative AI Red Teaming Methodology, Guidelines & Best Practices: A canonical methodology and process for Generative AI Red Teaming, including (but not limited to) LLM Red Teaming
• Standardized Evaluations to boost trust: Metrics, Benchmarks, Datasets, Frameworks, Tools and Prompt Banks (as applicable) for LLM evaluations.

Expected Outcomes

• An AI Red Teaming Methodology that organizations can use for their development, operations, governance and regulatory processes: A well articulated methodology for AI red Teaming improves the common understanding between the various constituents in the Generative AI ecosystem. The requirement of the details and content varies by the audience and so achieving a contextual common understanding is not easy. Our addition of best practices will definitely help the organizations.
• Standard set of LLM evaluations: The LLM evaluation requires broader artifacts spanning metrics, benchmarks, datasets, frameworks, tools and prompt banks (as applicable). A canonical collection and a toolset gives the practitioners a head start. They can, of course, customize it depending on the use case and organizational policies.
• Audience-specific, context-specific artifacts: We will have tailored and customized templates and profiles,  thus making this domain (and OWASP Top 10 LLMs) accessible, approachable and more importantly consumable by a wide variety of audiences 

📃 Full details on the project available at this link here 

Why the Focus on AI Red Teaming?

• In October 2023, President Biden’s Executive Order on AI Security put AI Red Teaming in the spotlight. Developers of AI systems—especially dual-use models that serve both civilian and military purposes—are now required to conduct red team testing before deployment. This ensures external evaluations to safeguard against harmful or unsafe AI outputs.
• The European Union’s AI Act echoes this sentiment, mandating rigorous adversarial testing for General-Purpose AI (GPAI) models. This puts red teaming at the core of AI security development worldwide.

How to Get Started with AI Red Team Testing

AI Red Teaming goes beyond traditional testing methods, focusing on adversarial scenarios where malicious actors may attack AI systems.

This approach includes:

• Threat Modeling: Using tools to generate threat models and attack trees.
  • Asset-centric Threat Modeling for AI-based Systems
  • STRIDE GPT is an AI-powered threat modeling tool that leverages Large Language Models (LLMs) to generate threat models and attack trees.
• Bias Detection: Employ frameworks like The Aletheia Framework to identify biases in AI systems and trace them back to the design and deployment stages. 
• Ethics and Safety: Ensure your AI system’s responses align with organizational values, avoid toxic outputs, and prevent hallucinations (false or misleading information generated by AI).

Diving Deeper Into Red Teaming AI Systems

• AI Red Teaming is a method designed to simulate adversarial attacks and probe AI systems for vulnerabilities. From testing how AI handles misleading data to evaluating its robustness against edge cases, Red Teaming ensures your AI system can withstand challenges in the real world.
• Model cards and risk cards are used to establish the scope of the Red Team testing and to create abuse cases. Model cards provide standardized documentation on their design, capabilities, and constraints of a model. Risk cards list  potential negative consequences of a model such as biases, privacy problems, and security vulnerabilities.

The Focus of AI Red Teaming

• Adversarial Attacks: Simulate malicious data inputs to see if the AI makes incorrect decisions or predictions.
• Bias Identification: Run scenarios to ensure the AI is making fair and equitable decisions.
• Robustness Testing: Test how well AI performs with anomalies and edge cases.
• Security Auditing: Identify vulnerabilities in your AI’s Software Bill of Materials (SBOM) or configuration.
• Ethics and Safety: Ensure the AI aligns with ethical guidelines, focusing on preventing harmful decisions.

Deployment Models & AI Red Teaming 

Red Teaming from an adversarial perspective requires clear boundaries. White hat red teams are authorized to test specific AI systems, and knowing where those boundaries are is essential. Once vulnerabilities are found, identifying who is responsible for fixing them whether in application security, network infrastructure, or the AI model itself is key.

Scoring AI Vulnerabilities

• When assessing vulnerabilities, organizations must measure the impact against their specific use cases.
• The Common Vulnerability Scoring System (CVSS) is the most common scoring method but due to unique characteristics of LLM’s, more accurately weighted scores can be calculated by extending the existing scoring method as suggested in the paper  “Security Vulnerability Analyses of Large Language Models (LLMs) through Extension of the Common Vulnerability Scoring System (CVSS) Framework”
• Bugcrowd’s Vulnerability Rating Taxonomy now includes LLM vulnerabilities from the  OWASP Top Ten for LLM.

AI Threat Intelligence

Another benefit of AI Red Teaming is enhancing AI Threat Intelligence. By researching and analyzing attack data, we aim to supplement the MITRE ATT&CK framework with adversarial techniques used against AIML systems. Learn more about joining the Research Initiative – Securing and Scrutinizing LLMS in Exploit Generation

The Bigger Picture: AI Red Teaming and the Future of Security

• AI Red Teaming is about more than just testing, it challenges us to think like our adversaries, ensuring AI systems are secure, ethical, and resilient. As AI continues to integrate into every aspect of our lives, adversarial thinking must be a core component of every AI development process.
• We don’t need to reinvent the wheel, but we do need to upgrade it. As AI evolves, let’s ensure our security measures keep pace no matter what icy conditions we face ahead!