Compliance Challenges of the EU AI Act

Translating Regulatory Requirements into Technical Benchmarks

This event will present the first comprehensive evaluation framework for generative AI models (COMPL-AI), developed by researchers at ETH Zurich, INSAIT, and LatticeFlow AI to bridge the gap between the EU AI Act’s regulatory requirements and technical realities. Mark Vero, a project collaborator and co-author of the paper, will discuss how the COMPL-AI framework translates the AI Act’s principles and requirements into concrete, measurable technical standards, with a focus on large language models (LLMs). Mark will also present the results of evaluating 12 state-of-the-art LLMs, highlighting their shortcomings, particularly in robustness, safety, diversity, and fairness. The session will explore the challenges and opportunities of aligning AI regulation with technical implementation, contributing to the EU’s broader efforts, including the drafting of the General Purpose AI Code of Practice.

Mark Vero is working on his PhD at the Secure and Reliable Intelligent Systems Lab (SRI Lab) under the supervision of Professor Martin Vechev at ETH Zurich. His research concerns the privacy and security of LLMs, with a focus on uncovering safety risks in user-facing applications of LLMs. His work has been highlighted at top conferences and workshops in spotlight and oral presentations, has won the Privacy Papers for Policymakers award, and has been featured in international popular media. Prior to his PhD, he completed his Master’s degree with distinction at ETH Zurich in electrical engineering.