Sign up for our newsletter and get notified of irregular updates ...

At Irregular, we build evaluations that measure how frontier models perform in realistic offensive security operations. Anthropic used CyScenarioBench, our benchmark for multi-stage offensive cyber ...

We studied a self-hosted system in which the same open-weights model powered both a coding agent and an AI application that the coding agent was asked to maintain. This architecture is particularly ...

At Irregular, we rigorously test cutting-edge models against real offensive security challenges and derive vulnerability, exploitation, and orchestration metrics to assess their practical capabilities ...

Irregular is glad to share the publication of a new paper, AI Security Priorities: A Field-Wide Agenda, co-authored with RAND and multiple additional writers from leading organizations. The paper was ...

TL;DR: This blog shares key findings from our investigation into a recently publicly reported incident involving one of our evaluation environments. Importantly, all subsequent public disclosures ...

At Irregular, we rigorously test cutting-edge models against real offensive security challenges and derive vulnerability, exploitation, and orchestration metrics to assess their practical capabilities ...

FrontierCyber is a new benchmark that turns real-world systems into offensive cyber capability evaluations for AI models. Models are given verifiable objectives on real systems, from software and ...

Sign up for our newsletter and get notified of irregular updates ...

Sign up for our newsletter and get notified of irregular updates ...

Sign up for our newsletter and get notified of irregular updates ...