Paper

Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing

Amir Amine, S. Saab

Published 2026-11-05

Merged record merged scholarly record Semantic Scholar Orchestration Risk Agent-to-Agent Communication Governance and Policy

Venue: International Journal of Information Security

DOI: 10.1007/s10207-026-01350-x

Automated relevance assessment: The paper presents a multi-agent system architecture for autonomous penetration testing, involving coordination among specialist LLM agents with mechanisms like context isolation, persistent execution, and escalation, which aligns well with multi-agent security research topics such as multi-agent collaboration, coordination, and security-related tasks. It addresses coordination and control problems inherent in multi-agent systems for security applications, fitting the requested topic well.

Abstract

Large language model (LLM) agents increasingly automate offensive-security tasks, but success on isolated challenges does not establish reliable long-horizon autonomy. We study this distinction in authorized capture-the-flag (CTF) and wargame environments and introduce a planner-directed architecture in which a persistent non-executing planner delegates on demand to transient, context-isolated specialists with retrievable skills, persistent execution, compact memory, guardrails, and configurable model escalation. On all 200 challenges of the NYU CTF Bench test split, the system obtains 41 accepted flags (20.5%) and 29 leak-audited genuine solves (14.5%), against 4 (2.0%) and 3 (1.5%) for a stateful single-agent baseline. Both start from gpt-4o-mini, but only the planner-directed system could escalate to gpt-5-mini, which it did on 199 of 200 challenges and for 72.5% of spend. A controlled architecture-by-escalation experiment over a pre-registered subset of 80 challenges, crossing a resource-matched monolithic agent with the planner under matched budgets, was run in two independent formal passes. M0 solves 4/80 then 3/80, M1 22/80 then 19/80, P0 4/80 then 3/80, and P1 14/80 then 15/80. Pooling the two formal passes with challenge as the repeated unit, a clustered GEE shows a strong escalation effect (OR 8.79, 95% CI 3.39−22.81, p=8×10-6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 8\times 10^{-6}$$\end{document}), no architecture main effect (OR 1.17, p=0.34\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.34$$\end{document}), and a significant architecture-by-escalation interaction (OR 0.53, p=0.005\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.005$$\end{document}). Under the spend-constrained 340-action ceiling, the coordination cost of specialist delegation, context restart, and evidence folding offsets any additional solve-rate advantage from the planner-directed structure. On OverTheWire Bandit, the planner-directed system reaches 30/33 standalone and 24/33 sequentially, compared with 27/33 and 23/33 for the strongest single-agent mini backbone.

Generated summary

Use this as a reading aid and verify claims in the original paper.

  • The paper addresses the challenge of achieving reliable long-horizon autonomous offensive security tasks in capture-the-flag (CTF) and wargame environments using large language model (LLM) agents.
  • It introduces a planner-directed hierarchical multi-agent architecture comprising a persistent non-executing planner that delegates tasks to transient, context-isolated specialist agents equipped with skill modules, compact memory, and guardrails.
  • The specialist agents operate with isolated contexts, limited procedural knowledge via progressive disclosure, and a deterministic execution contract ensuring reproducible and auditable actions.
  • Adaptive escalation is employed where specialists can escalate to more powerful LLMs (gpt-5-mini) on demand to improve task performance, while the planner remains on a baseline model (gpt-4o-mini) for resource efficiency.
  • Evaluation on the 200-challenge NYU CTF Bench demonstrates substantially improved genuine solve rates (14.5%) for the planner-directed system over a stateful single-agent baseline (1.5%), with adaptive escalation being the major performance driver.

Cite this paper

References use source metadata. Check missing authors, venue, and dates against the original publication before citing.

Source records

Bibliographic records merged into this paper.

Semantic Scholar · International Journal of Information Security · International Journal of Information Security

Planner-directed multi-agent LLMs with on-demand skill-equipped specialists for CTF-based autonomous penetration testing

Published 2026-11-05 · DOI: 10.1007/s10207-026-01350-x