Check Point & UK AI Security Institute Launch Open-Source Benchmark to Test LLM Security in AI Agents

Check Point Software Technologies, Lakera, and researchers from the UK AI Security Institute (AISI) have announced the launch of the backbone breaker benchmark (b3) — the world’s first open-source security evaluation framework designed specifically to test vulnerabilities in large language models (LLMs) embedded inside AI agents.

Unlike traditional red-teaming methods, b3 introduces a new concept called “threat snapshots.”

These snapshots isolate the most critical decision points within an agent’s workflow — the moments where security failures are most likely to happen. This allows developers and model providers to assess security weak spots without rebuilding or simulating an entire agent system.

“We built the b3 benchmark because today’s AI agents are only as secure as the LLMs that power them,” said Mateo Rojas-Carulla, Co-Founder and Chief Scientist at Lakera, a Check Point company. “Threat Snapshots allow us to systematically surface vulnerabilities that have until now remained hidden in complex agent workflows. By making this benchmark open to the world, we hope to equip developers and model providers with a realistic way to measure, and improve, their security posture.”

The benchmark includes 10 representative threat snapshots and a dataset of 19,433 adversarial attacks, crowdsourced via Gandalf: Agent Breaker — a gamified red-teaming platform.

It measures susceptibility to attacks ranging from system prompt exfiltration and phishing link insertion to malicious code injection, denial-of-service attempts, and unauthorised tool calls.

Initial testing of 31 popular LLMs produced notable insights:

• Enhanced reasoning significantly boosts security performance.
• Model size does not predict security strength.
• Closed-source models currently outperform open-weight models — though the gap is narrowing.