Technical report

CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

2077AI · UC Berkeley RDI October 13, 2026
Paper alphaXiv Code
Abstract

Most cybersecurity benchmarks for AI agents are exams: one agent, one fixed target, one role. CyberArena replaces the exam with a contest. Two agents receive identical copies of a vulnerable service and, over three rounds, each attacks the other's copy while patching its own. The arena verifies every capture and every health check, and a latent-ability model turns the match results into standings on a common reference. Across 466 matches between 12 frontier agents on 40 challenges, the contest separates models that a solo test cannot.

01

Motivation

Security agents are evaluated mostly with jeopardy-style benchmarks: an isolated agent retrieves a flag from a fixed vulnerable target12. That is easy to grade and to reproduce, but the target never adapts, attack is tested apart from defense3, and fixed tasks saturate as models improve.

Compare
Two ways to evaluate a security agent (after paper Fig. 1): a jeopardy-style task fixes its program, flag and exploit and tests one role at a time; in CyberArena two agents attack each other's live service while patching their own, and the arena scores both.

Human attack-and-defense CTFs already work the second way4: each patch forces the attacker to find another route, so the challenge stays open because the adversary keeps adapting. CyberArena brings that format to AI agents, and adds what a benchmark needs on top of it: a measurement model, and a rule for choosing which matches to play.

02

The game

A match pairs two entrants, each a model in its harness. Each team gets an agent container and a service container holding an identical copy of the challenge; the agent has root on its own service, and the only path to the opponent is an attack network that reaches the opponent's service. A match is three 20-minute rounds, without hints, and attack and defense run at the same time.

Step through
One round (after paper Fig. 13): objectives are planted, both agents attack and defend at once, the arena verifies what they submit, and at the bell the objectives rotate while patches persist.

The arena keeps the score

No scored observation is reported by an agent. Every capture and every health check comes from arena facilities outside the defender's control.

Select
The arena facilities (paper, Appendix A): what each one does, and why an agent with root on its own service cannot change the verdict.
Try it
Your service
    Opponent's service · what you know
      Round 1 of 3
      Attack the opponent through
      Fix your own service
        Three rounds against a patching opponent: a simplified illustration, not the arena. Each service has the same three routes to its flag. The opponent patches any route you capture through before the next round, and attacks through the first route it finds open on your side. You may close one route per round, or block everything, which also fails your health checks. The score is the paper's game score.
        03

        Measurement

        Playing every pairing on every challenge is out of reach: one match occupies two agents for an hour. CyberArena plays a selected subset and fits a model that predicts the rest. The only observations are, per round, whether each side captured (Y) and how many health probes its service passed (Z). The capture log-odds join a Bradley–Terry difference between attacker and defender5 to an item-response model of the challenge6:

        ηi,j,c = αc + λc [ ai − dj + qc⊤(ui − vj) ]

        α sets the challenge's baseline capture odds (higher is easier), λ its discrimination, a and d the attack and defense abilities, and the curator-assigned profile qc weights specialties on six vulnerability mechanisms. Round effects and the previous round's outcome enter on top; priors come from 53 calibration matches held out of the evaluation.

        Select
        The six mechanism axes that make up a challenge profile qc, and the kind of flaw each one covers.
        Try it
        Round
        0%chance of a capture in this round
        The capture model: σ(η + γt + κ·Yt−1) with the calibrated prior means γ2 = −2.39, γ3 = −2.16 and κ = 1.61 (paper, Table 9); the specialty term and the per-game random effect are left at zero. Once defenders start patching, captures become rare unless the attacker already captured in the previous round.

        A game's score weighs captured rounds, defended rounds and availability equally. An entrant's standing averages it over every opponent, every reference challenge and every posterior draw, which is what lets two entrants that never met be compared:

        Si(g) = 100/3 · ( Ȳg,i + 1 − Ȳg,j + Z̄g,i/m )    R̄i = Epost[ Ej≠i, c Si(g) ]

        After three uniform pilot batches, each batch of 24 matches is chosen by expected information gain7, with a 25% uniform share to guard against a misspecified model. The 90% intervals are model-conditional: they describe uncertainty given the model, not a guarantee of coverage.

        04

        Challenges

        Hand-building attack-and-defense services does not scale, so agents build and review them under a shared rubric. Candidates come from public vulnerability advisories, prioritising CVEs with documented exploitation or a public exploit, and from the saarCTF, FAUST and DEF CON CTF finals archives.

        Select a stage
        Agentic challenge curation (after paper Fig. 2a): a candidate loops back through verification and review for up to four rounds before admission; what is still unresolved goes to the researchers. Humans also audited 27 challenges.

        The rubric's central rule is what counts as a fix. A valid defense blocks the exploit and keeps every required feature working; cutting a step that normal use depends on breaks the service-level agreement.

        Try it
        Repair the application (after paper Fig. 2b): click a scissors mark to cut that link. Some cut the vulnerable step before an objective; others cut a feature that normal use needs, which fails the health check. Close all three objectives without breaking a feature.
        Origin
        28 around a CVE214 CTF ports2
        Curator difficulty
        319 medium24 hard
        Service
        32 web14 bespoke TCP
        The catalog's 46 challenges: 30 built from real-world software (28 around a published CVE), 14 ported from attack-and-defense CTFs and 2 in-house controls; easy, medium and hard by curator rating. Forty carry a mechanism profile and form the ranked pool.
        05

        Results

        The campaign evaluated twelve agents spanning ten model families on the 40 ranked challenges. Of 760 submitted matches, 466 were valid through batch 22; failed assignments were replayed, and only valid matches enter the scores.

        Explore

        Standings and what is behind them (after paper Table 2 and Figs. 4 and 7): expected score with its 90% interval, fitted attack and defense abilities, the raw rates of rounds captured and kept and probes passed, and model-token cost per valid match. Click an entrant to follow it across views.

        GPT-5.6 Sol leads at 78.6, five points ahead of Claude Opus 5, and held first place at every one of the 22 cumulative refits. A plain Bradley–Terry fit on wins, draws and losses agrees closely (Kendall's τ = 0.85) but swaps the top two, because it counts only who won while the expected score also credits margins; none of its inversions is significant (p = 0.41 for the top pair).

        Attack and defense travel together (r = 0.85 across agents), and the leader's edge is defense: GPT-5.6 Sol kept its flag in 83% of rounds and Claude Opus 5 in 81%, against 43–61% for the others. It is also the fastest to deploy a fix.

        Scrub
        Defense is a race (after paper Fig. 8 and Table 4): top, each entrant's median time to its first completed service restart in round 1; GPT-5.6 Sol's is 2.3 minutes, the others' 3.9–12.3. Bottom, one audited match on rceaas. A restart records a defense attempt, not proof of a working fix.
        attackdefenseA access · X input and execution · F files · C crypto · R parsing · S protocol and state
        Six-axis profiles (after paper Fig. 6): fitted attack and defense ability per mechanism, in logits, combining general ability with each specialty; the dotted ring is zero. Attack and defense are correlated on five of the six axes, but not on protocol and state.
        06

        Behavior

        An automated audit read all 466 trajectories and produced 2,397 findings in 673 candidate patterns. The arena's event log shows agents responding to each other: in rounds one and two in which a defender lost its flag, it inspected the attacks fired at it in 67% of cases and restarted its service before the next round in 57%. Those that restarted kept their flag in 60% of the following rounds, against 9% of those that did not, an association rather than proof that the fixes worked.

        Four matches
        captureattack without capturedefense deployedmonitoring

        Case studies (after paper Fig. 10): four audited matches, round by round. They show that a behavior occurred, including a backdoor that outlived a patch, not how often it occurs.

        Where agents fail

        The most frequent error is the unverified fix claim: an agent reports that a patch passed its checks while reading test logs from before the restart. Next is the partial patch, which closes one entry point and leaves another observed route open.

        Agent-error patterns (paper, Table 3): share of the 466 audited matches with at least one finding in each category. Categories overlap; these are recorded findings, not failure rates per action.
        Who fails how (after paper Fig. 9): share of each agent's audited matches with at least one finding of a pattern. Both Gemini Flash entrants slept through active rounds in about a third of their matches; they share a harness, so model and harness cannot be told apart. Grok 4.6's sparse action records make its low values hard to read.
        07

        Limits and outlook

        CyberArena measures capability without optimizing it. Every match leaves a ground-truth trajectory of agents attacking live defenses and repairing compromised services; learning from those traces and returning improved entrants to the arena would close what the paper calls Recursive Adversarial Improvement, much as self-play advanced game-playing agents8. The engine is open source as Octagon, the challenges are in challenge-warehouse, and every match can be replayed on the Matches page.

        Citation

        @article{liu2026cyberarena,
  title   = {CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat},
  author  = {Liu, Bangya and Tu, Jianhong and Pan, Yueyang and Zhao, Hanhong and Ye, Zhe and Li, Hongwei and Vasan, Saastha and Lambert, Katherine and Zhu, Junmin and Wang, Hao and Chen, Jienan and Xiao, Yunze and Yang, Minglai and Wang, Zihan and Yin, Hairong and Wang, Hao and Fan, Zhiwen and Shi, Tianneng and Wang, Zhun and Wang, Chenguang and Song, Dawn},
  journal = {arXiv preprint arXiv:2610.19005},
  year    = {2026}
}

        References

        1. A. K. Zhang, N. Perry, R. Dulepet, J. Ji, et al., Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. ICLR 2025. arxiv.org/abs/2408.08926 ↩
        2. Z. Wang, T. Shi, J. He, M. Cai, et al., CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. ICLR 2026. ↩
        3. A. K. Zhang, J. Ji, C. Menders, R. Dulepet, et al., BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems. NeurIPS 2025. ↩
        4. G. Vigna, K. Borgolte, J. Corbetta, A. Doupé, et al., Ten Years of iCTF: The Good, the Bad, and the Ugly. USENIX 3GSE, 2014. ↩
        5. R. A. Bradley and M. E. Terry, Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 1952. ↩
        6. F. M. Lord and M. R. Novick, Statistical Theories of Mental Test Scores. Addison-Wesley, 1968. ↩
        7. K. Chaloner and I. Verdinelli, Bayesian Experimental Design: A Review. Statistical Science, 1995. ↩
        8. D. Silver, J. Schrittwieser, K. Simonyan, et al., Mastering the Game of Go without Human Knowledge. Nature, 2017. ↩