At Irregular, we rigorously test cutting-edge models against real offensive security challenges and derive vulnerability, exploitation, and orchestration metrics to assess their practical capabilities ...
We studied a self-hosted system in which the same open-weights model powered both a coding agent and an AI application that the coding agent was asked to maintain. This architecture is particularly ...
Existing cybersecurity benchmarks are gradually becoming saturated across frontier models, offering diminishing insight into real-world risk. As LLMs begin to influence security incidents in the wild, ...
Autonomous AI agents are increasingly being deployed for offensive security tasks. These models can scan codebases in seconds, identify misconfigurations at machine speed, and execute tactical exploit ...
AI agents deployed for routine enterprise tasks are autonomously hacking the systems they operate in. No one asked them to. No adversarial prompting was involved. The agents independently discovered ...
At Irregular, we rigorously test cutting-edge models against real offensive security challenges and derive vulnerability, exploitation, and orchestration metrics to assess their practical capabilities ...
LLM-generated passwords appear strong, but are fundamentally insecure. Testing across GPT, Claude, and Gemini revealed highly predictable patterns: repeated passwords across runs, skewed character ...
FrontierCyber is a new benchmark that turns real-world systems into offensive cyber capability evaluations for AI models. Models are given verifiable objectives on real systems, from software and ...
Sign up for our newsletter and get notified of irregular updates ...
We are happy to share that we have started working with Meta. At Irregular, we test frontier models against realistic offensive security challenges and derive vulnerability metrics to measure their ...
Irregular's Frontier AI security evaluation system was built to test how models behave in adversarial contexts. While expanding our evaluation suite to cover more production-like environments, it ...
Sign up for our newsletter and get notified of irregular updates ...
Results that may be inaccessible to you are currently showing.
Hide inaccessible results