Abstract
Large language model (LLM) agents increasingly automate offensive-security tasks, but success on isolated challenges does not establish reliable long-horizon autonomy. We study this distinction in authorized capture-the-flag (CTF) and wargame environments and introduce a planner-directed architecture in which a persistent non-executing planner delegates on demand to transient, context-isolated specialists with retrievable skills, persistent execution, compact memory, guardrails, and configurable model escalation. On all 200 challenges of the NYU CTF Bench test split, the system obtains 41 accepted flags (20.5%) and 29 leak-audited genuine solves (14.5%), against 4 (2.0%) and 3 (1.5%) for a stateful single-agent baseline. Both start from gpt-4o-mini, but only the planner-directed system could escalate to gpt-5-mini, which it did on 199 of 200 challenges and for 72.5% of spend. A controlled architecture-by-escalation experiment over a pre-registered subset of 80 challenges, crossing a resource-matched monolithic agent with the planner under matched budgets, was run in two independent formal passes. M0 solves 4/80 then 3/80, M1 22/80 then 19/80, P0 4/80 then 3/80, and P1 14/80 then 15/80. Pooling the two formal passes with challenge as the repeated unit, a clustered GEE shows a strong escalation effect (OR 8.79, 95% CI 3.39−22.81, p=8×10-6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 8\times 10^{-6}$$\end{document}), no architecture main effect (OR 1.17, p=0.34\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.34$$\end{document}), and a significant architecture-by-escalation interaction (OR 0.53, p=0.005\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.005$$\end{document}). Under the spend-constrained 340-action ceiling, the coordination cost of specialist delegation, context restart, and evidence folding offsets any additional solve-rate advantage from the planner-directed structure. On OverTheWire Bandit, the planner-directed system reaches 30/33 standalone and 24/33 sequentially, compared with 27/33 and 23/33 for the strongest single-agent mini backbone.
Generated summary
- The paper addresses the challenge of achieving reliable long-horizon autonomous offensive security tasks in capture-the-flag (CTF) and wargame environments using large language model (LLM) agents.
- It introduces a planner-directed hierarchical multi-agent architecture comprising a persistent non-executing planner that delegates tasks to transient, context-isolated specialist agents equipped with skill modules, compact memory, and guardrails.
- The specialist agents operate with isolated contexts, limited procedural knowledge via progressive disclosure, and a deterministic execution contract ensuring reproducible and auditable actions.
- Adaptive escalation is employed where specialists can escalate to more powerful LLMs (gpt-5-mini) on demand to improve task performance, while the planner remains on a baseline model (gpt-4o-mini) for resource efficiency.
- Evaluation on the 200-challenge NYU CTF Bench demonstrates substantially improved genuine solve rates (14.5%) for the planner-directed system over a stateful single-agent baseline (1.5%), with adaptive escalation being the major performance driver.