BruCON 0x10

Over the last 12 months Microsoft’s AI Red Team (AIRT) has conducted nearly 100 assessments of AI systems including comprehensive reviews of foundation models, multiple reviews of Copilot features, and in-depth reviews of AI systems in sensitive domains such as health care. From this work AIRT has developed deep knowledge of the most impactful security, safety, and privacy risks that the usage of AI systems in the real world can cause, the techniques and tooling needed to elicit them, and approaches to prevent or detect these risks.

In this presentation we will cover what AI Red Teaming is, the processes and tooling AIRT has developed, and most interesting what the key trends have been in terms of techniques and weaknesses identified during our many assessments. We will discuss how AI security issues are tightly connected with traditional cybersecurity, but also how the safety aspect of AI introduces new and exciting challenges to our work. We will also touch on how AIRT’s work has informed the development of new defenses for AI systems and security professionals should approach defending the AI systems that they use.

We will also look ahead to next year and where the risks might go next, and how we might want to prevent them in a world where AI system capabilities are evolving at an extremely rapid pace.