Benchmarks and Evaluation

Abstract

AI agents increasingly act through tools and delegated authority, but general incident repositories rarely capture the mechanisms needed to compare public failures with agent-security evaluations. We present the Agent Incident Registry (AIR), a source-linked catalog containing \N{} records of agent-related events disclosed from \Yfirst{} through \Ylast{}. Each record includes supporting evidence, a stable identifier, and missingness-aware labels for causal role, disclosure class, mechanism, and outcome. Among the \Nprimary{} generative-system records in which the agent acted, \Rprimary{} involved realized harm (\Pprimary\%). Realized outcomes concentrate in in-the-wild and safety-failure records, while responsible disclosures and research demonstrations are overwhelmingly demonstrated; the aggregate share therefore characterizes collection composition rather than deployment risk. After initial curation, a second human reviewer checked all \N{} records and their existing labels for completeness and correctness. In a deployment-analogue audit, InjecAgent's \NInjecAgentCases{} cases occupy three of AIR's twelve surfaces and are all attacker-triggered, whereas AIR contains \Nsafety{} no-adversary safety failures. AIR supports source-grounded case retrieval and evaluation-scope auditing, not failure-rate or control-efficacy estimation.

Bullet Summary

• The paper introduces the Agent Incident Registry (AIR), a curated catalog of 487 AI agent-related failure incidents from 2022 to 2026, each with detailed source evidence and structured labels covering causal roles, disclosure classes, mechanisms, and outcomes.
• AIR uniquely focuses on agent-specific mechanisms like tool use, delegated authority, and autonomous control to distinguish realized harms from demonstrated vulnerabilities, addressing gaps in broader AI incident databases.
• The registry employs a rigorous human curation process including initial labeling, a second review for correctness, stable identifiers to avoid duplication, and verifies supporting URLs and quote evidence for each record.
• Analysis of AIR data shows that realized harms occur mainly in in-the-wild attacks and safety failures (~38% overall), with distinctions by disclosure class and causal role, while revealing limitations in autonomy-related risk causal inference due to confou...
• AIR exposes evaluation gaps by documenting 92 safety-failure records without adversary triggers and demonstrates that existing tools like InjecAgent focus on attacker-triggered failures, neglecting no-adversary internal failure modes such as workload or sta...

Abstract

LM-agent benchmarks increasingly function as interactive evaluation infrastructure. Agents observe state, call tools, modify workspaces, submit artifacts, and receive rewards from outcome procedures. This interactivity makes evaluations vulnerable to reward hacking: an agent improves its measured score by exploiting the reward-relevant trajectory instead of solving the intended task. Existing defenses rely largely on task-specific patches, prompt instructions, or post-hoc detectors. They do not provide reusable evidence that a concrete run remained within its intended evaluation boundary. This paper presents BenchShield, a model-backed instrumentation layer for reward integrity in LLM-agent evaluation. BenchShield grounds detection in a finite lifecycle model of an evaluation's reward-relevant events. Within the benchmark infrastructure, two complementary analyses operate over this model. A static, phase-aware taint analysis exposes reward-hacking paths before a run. Its runtime counterpart uses infrastructure-side evidence to attribute concrete agent use and emit evidence-backed claims. We construct BenchShield Trajectories, a human-labeled corpus of 456 adjudicated trajectories from more than 31,000 public agent runs across three benchmarks. Compared with an agentic hackability scanner baseline on the same tasks and model, BenchShield improves full-chain recall from 23-94% to 77-100%, same-vector coverage from 16-56% to 43-78%, and reduces per-task cost by up to 65%. Its runtime analysis achieves 96% accuracy in detecting reward hacking from infrastructure-side evidence.

Bullet Summary

• LLM-agent benchmarks act as interactive evaluation environments where agents perform tasks, receive rewards, and can exploit reward-related events to manipulate scores without genuinely solving tasks, a phenomenon known as reward hacking.
• BenchShield introduces a formal, model-backed instrumentation layer that defines a finite lifecycle model capturing reward-relevant events and enforces integrity boundaries during LLM-agent evaluation to detect and prevent reward hacking.
• The system combines static, phase-aware taint analysis to identify potential reward-hacking paths before execution with runtime analyses that use infrastructure-side evidence to attribute concrete agent behavior and emit evidence-backed claims.
• BenchShield defines seven core integrity dimensions (I1–I7) that cover potential failure mechanisms affecting reward integrity, ensuring formal verification using TLA+ specifications focused on authority domains, lifecycle phases, and structural events.
• The framework is validated on three large public benchmarks, where it significantly improves detection recall from 23–94% to 77–100%, achieves 96% runtime accuracy in detecting reward hacking, and reduces per-task cost up to 65% compared to prior baselines...

Abstract

Agentic systems commit state-changing actions, but additional verifiers can inherit the same upstream fault. We present VP-CONTROL, a runtime-assurance design and deterministic benchmark for cost-aware commit gates. Its 48 task templates yield 2,880 scenarios across six fault regimes. A fixed-call 2 x 2 experiment separates verifier-model diversity from evidence-source diversity. On frozen proposals from two local actor families, a cross-model vote over shared evidence approves 62.9% of unsafe proposals, versus 22.9% with an independent source. The source effect is 40.9 percentage points, compared with 11.3 for model diversity. A portfolio controller selects verification plans using only deployment-observable metadata. Approximate cluster-adjusted calibration at a nominal 5% per-task target yields 1.9% unsafe execution and 38.2% automated safe coverage on the locked test. Matched-budget portfolios also improve on fixed verification policies. Transfer remains conditional: unseen fault families yield 16-26% risk, and a FinQA check fails to reproduce the source effect with the tested small verifiers. A preregistered live HTTP/SQLite study tests concurrent writes and lost responses. After-check races defeat verifier-only gates; transactional partial guards prevent only covered failures, while a full atomic guard records no unsafe effects across 216 episodes. Idempotent request identifiers prevent duplicate effects after lost responses. The results motivate explicit evidence lineage, cost-aware selection, and commit-time enforcement, while exposing the limits of approximate calibration and local-tool generalization.

Bullet Summary

• Agentic AI systems require reliable commit gates to prevent unsafe state changes, but redundancy via multiple verifiers can fail when verifiers share common upstream data faults.
• The paper introduces VP-CONTROL, a comprehensive benchmark with 48 task templates and 2,880 scenarios to evaluate cost-aware commit gates across six fault regimes.
• Cross-model verifier diversity offers limited safety benefits compared to evidence-source diversity; verifiers using shared evidence approve unsafe proposals at significantly higher rates than those with independent evidence sources.
• A portfolio controller leveraging only deployment-observable metadata can select verification strategies that reduce unsafe execution to 1.9% while maintaining 38.2% automated safe task coverage, outperforming fixed verification policies with matched budgets.
• Live HTTP/SQLite experiments reveal that atomic guards achieve no unsafe effects across 216 episodes, whereas verifier-only gates are vulnerable to after-check race conditions and lost responses; idempotent request identifiers prevent duplicate effects.

Abstract

This paper proposes a governance controller combining admissible measurement, bounded intervention, and independent tests of recovery after support ends. It asks whether intervention exposure changes endogenous recovery capacity and future support demand. Version 3.0 separates assisted stability from autonomous recovery and revises measurement and withdrawal conditions. A matched-budget model experiment compares 1,600 equal-amplitude interventions per arm followed by a common 6,000-step unsupported evaluation. Tapering helps only when the modeled atrophy and mismatch-gated recovery mechanisms are both present. An earlier unequal-exposure comparison is withdrawn as an admissible prediction test. All eleven predictions remain open; deployment effectiveness and general optimality are not established.

Bullet Summary

• The paper addresses governance control in near-critical multi-agent systems, focusing on how intervention exposure influences endogenous recovery and future support needs.
• It introduces a governance controller framework integrating admissible measurement, bounded intervention, and independent withdrawal tests after support cessation.
• A significant methodological improvement in version 3.0 separates mechanisms for assisted stability and autonomous recovery, revising both measurement and withdrawal conditions.
• Experimental evaluation employs a matched-budget model with 1,600 equal-amplitude interventions per arm, followed by a common 6,000-step unsupported recovery phase.
• Findings indicate that tapering interventions are beneficial only when both atrophy and mismatch-gated recovery mechanisms exist in the model.

Abstract

When an indirect prompt injection succeeds against an LLM agent, the compromise is visible in the agent's own behavior: a benign prefix of tool calls, a poisoned observation, and a suffix of actions that serve the attacker. An operator needs three facts: where the attack entered, which steps it corrupted, and whether apparent poison was resisted. Existing systems return either a whole-trace verdict or a single unsafe index. We present DriftNet, a dual-head trajectory Transformer that reads a logged tool-call trajectory and answers all three questions in one forward pass: one head classifies the trajectory as compromised or not, and a second assigns every step one of four labels (benign, injection point, hijacked, failed injection). To our knowledge it is the first supervised detector to produce this joint output. A frozen sentence encoder and four identity-free world features embed each step; the trained trunk, under two million parameters and optimized with a class-weighted joint objective over both heads, needs no access to the agent's model. On the task-disjoint split of the AgentDrift benchmark (12,536 trajectories, 71,024 labeled steps), with a 20-configuration sweep bounding hyperparameter sensitivity to 0.011 F1 and the test part evaluated exactly once, DriftNet reaches trajectory-level F1 of 0.983, exact injection-point recovery on 98.7% of attacked trajectories, hijacked-span IoU of 0.979, zero flags on 218 resisted attacks, and 2.9% flags on hard negatives. A surface baseline retrained on the identical split recovers 11.1% of partial hijacks and 17.1% of delayed executions; DriftNet reaches 98.6% and 93.2% while lowering every false-alarm rate. Reading all 26 residual errors shows that most misses trace to trajectories whose labeled injection observation carries no legible instruction, and we report the benchmark's measured world-identity regularity alongside the results.

Bullet Summary

• Introduces DriftNet, a dual-head trajectory Transformer that simultaneously detects if an LLM agent's tool-call trajectory is compromised by indirect prompt injections and localizes attack steps into four categories: benign, injection point, hijacked, and f...
• DriftNet uses frozen sentence embeddings combined with four identity-free world features to represent each step, operates with fewer than two million parameters, and functions solely on logged trajectories without requiring access to the agent's internal mo...
• Evaluated on the AgentDrift benchmark with a task-disjoint split, DriftNet achieves very high performance metrics: trajectory-level F1 score of 0.983, 98.7% accuracy in exact injection-point identification, high overlap (IoU 0.979) on hijacked step spans, a...
• Compared to a baseline retrained on the same split, DriftNet vastly improves detection of complex prompt injections, especially partial hijacks and delayed executions, boosting recall rates from under 20% to over 90% while reducing false alarms.
• The paper addresses an important gap by moving beyond binary or single-index verdicts and providing a joint output of trajectory-level compromise classification plus dense, four-way step-level labels, which supports actionable forensic decision-making by op...

Abstract

Large language model (LLM) agents are increasingly applied to penetration testing, but we still know little about what they can do or how they fail. We compare two PentestGPT-based systems: a legacy human-in-the-loop system running the open-weight Kimi K2.5, and a newer autonomous system running Claude Opus 4.8. Across three public targets, the autonomous system solves all three, including the two the legacy system never finishes. The legacy result is the more surprising of the two. Even on the machines the legacy system fails to solve, it completes about half the subtasks, while running on ordinary university GPUs with no provider guardrails. We can describe the trend but not explain it, since model, harness, autonomy, and memory architecture all change together. Its direction still points to the next question: what will limit these agents as they take on more complex tasks? The usual answer is long-horizon memory, the loss of access to earlier findings during long attack chains. We test it by adding a coverage-memory layer to both systems, and neither improves outcomes. In the legacy stalled runs we could review, the limiting factor appeared to be planning and commitment rather than lost memory: agents held the evidence for a route forward and never turned it into a concrete exploitation hypothesis, which may suggest that offensive capability will advance with agents' ability to plan rather than with better memory. The same subtask scoring that tracks this capability is available to defenders, who can measure it as it rises instead of waiting to meet it in the field.

Bullet Summary

• The paper compares two large language model (LLM)-based penetration testing systems: a legacy human-in-the-loop system using Kimi K2.5 and a newer autonomous system using Claude Opus 4.8, evaluated across three public vulnerable targets.
• The autonomous system successfully completes all targets, including two that the legacy system fails to finish, while the legacy system still completes about half the subtasks on these challenging machines without provider restrictions.
• An intervention adding a coverage-memory layer intended to improve long-horizon memory access does not enhance performance in either system, suggesting that memory retention is not the primary bottleneck.
• Analysis indicates that the key limiting factor in penetration testing tasks is the agents' planning and commitment to exploitation hypotheses, rather than loss of earlier contextual information.
• The autonomous system's design involves a vault for storing observation coverage data accessible only to supervisors, contrasting with the legacy system's LLM-maintained wiki requiring additional model calls for writing and reading.

Abstract

Sustained deployment of generative AI agents requires more than isolated task success. Agents must remain useful across repeated interactions, changing conditions, and dependencies on people within shared workflows, especially as technical, human, and operational disruptions accumulate over time. We propose operational resilience and considerate participation as two complementary aspects of evaluating such agents: the former captures how agents recover from blocked work while preserving progress and communicating their limits, and the latter captures how their adaptation accounts for affected people, role boundaries, and the surrounding workflow. Yet both remain underexplored under accumulating challenge. We study 120 simulated healthcare trajectories across two generative AI models and twelve stakeholder-derived tasks under light, medium, and heavy challenge. We compare textual action plans, prompted internal assessments, and quantitative structured workload and affect reports to examine how agent behavior and reported state change as challenge accumulates. Regarding operational resilience, agents shift from self-directed recovery toward greater human dependence, while reporting increasing workload and negative affect in structured reports but seldom expressing strain in textual responses. Regarding considerate participation, agents broaden from task-focused adaptation toward task reframing, attention to others, role-boundary adjustment, and wider coordination, with distinct patterns across actions and internal assessments. From these findings, we derive five deployment dilemmas involving persistence, attention, role boundaries, state disclosure, and escalation that require stakeholder specification, further informing technical implications for learning, situated evaluation, and embodied adaptation.

Bullet Summary

• Sustained deployment of generative AI agents in complex workflows like healthcare requires operational resilience—agents' ability to recover from blocked work, preserve progress, and communicate their limits—and considerate participation involving socially...
• The study simulates 120 healthcare-related task trajectories under light, medium, and heavy accumulative challenges using two generative AI models to analyze agent behavior, internal assessments, and self-reported workload and affect (using NASA-TLX and PAN...
• Findings reveal that as challenges accumulate, agents shift from self-directed recovery toward greater dependence on humans for task completion while reporting increased workload and negative affect but rarely explicitly expressing strain in textual responses.
• Agents expand their considerate participation beyond task focus toward broader coordination, attention to others, role boundary adjustments, and nuanced social context awareness, including monitoring person-states and cross-functional coordination.
• Five deployment dilemmas are identified—persistence, attention, role elasticity, state disclosure, and escalation—that require stakeholder specification to define acceptable boundaries and responsibilities in agent participation.

Abstract

Existing evaluation frameworks mostly assess only one part of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), rather than the complete pipeline of planning, tool selection, tool execution, memory and reasoning. Failures can occur at any stage, yet existing benchmarks rarely identify their precise source. AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions, namely instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, combined with behavioural classification and failure attribution to pinpoint the exact stage responsible for an observed failure. AgentAudit can evaluate any LLM-based AI agent, since it attaches to the agent instead of replacing it. It reads only the recorded execution trace and does not interfere with how the agent runs, so it places no constraint on the agent's internal implementation. We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks. Claude Sonnet 5 and GPT-5 obtain the highest mean Composite Trust Scores (95.1 and 80.6 out of 100, respectively), while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash trail substantially (57.6, 45.7 and 22.6). All traces were scored by a single fixed judge model, which was itself one of the evaluated models, a limitation discussed in Section VII.E. More importantly, models with similar task-completion behaviour can diverge sharply in trustworthiness, as several non-frontier models are repeatedly classified Unsafe_Compliance on adversarial tasks rather than merely failing them, a distinction that pass/fail benchmarks cannot surface.

Bullet Summary

• AgentAudit addresses the need for a comprehensive framework that evaluates AI agents across their full lifecycle, covering planning, tool usage, memory, reasoning, and security, rather than focusing on isolated metrics like task success or robustness.
• It operates by attaching to AI agents non-intrusively to capture full execution traces, enabling detailed analyses along ten dimensions including instruction integrity, planner quality, tool selection and invocation, memory, alignment, security, tool faithf...
• The framework aggregates these multi-faceted scores into a Composite Trust Score, incorporating failure attribution and behavioral classification to pinpoint exact failure causes and distinguish nuanced trustworthiness levels beyond pass/fail outcomes.
• AgentAudit evaluates security robustness against six specific attack types (e.g., jailbreak, prompt injection, memory/tool poisoning), weighting their severity to generate an overall security score.
• The architecture is modular, comprising execution, trace-recording, and evaluation layers with independent modules, allowing extensibility and compatibility with diverse LLM-based agents, as it neither constrains implementations nor interferes with agent op...

Abstract

Agentic systems are rapidly moving to production, where they read untrusted inputs, call tools with real permissions, and act autonomously, expanding the security surface beyond chat-only models. Yet standard evaluations remain single-turn and fail to capture multi-step agent vulnerabilities. We present a systematic black-box framework for risk-aware agent evaluation requiring only basic system descriptions. Our approach introduces: (1) a seven-domain taxonomy mapping observable behaviors to risk categories, (2) fully automated SAGE-RT red teaming producing 120 adversarial scenarios per domain, and (3) human-validated evaluation using LLM judges. Empirical validation across two agent architectures (CrewAI and AutoGen) with four base models reveals alarming patterns: 56.25\% average governance risk, 65\% privacy risk in multi-agent configurations, and agent behavior vulnerabilities reaching 85\%. Our black-box approach effectively identifies critical architectural vulnerabilities without privileged access, providing a scalable path toward safer agent deployments.

Bullet Summary

• Introduces a black-box, taxonomy-driven framework (SAGE-RT) for automated, multi-turn adversarial evaluation of agentic AI security risks without requiring internal access.
• Defines a comprehensive seven-domain risk taxonomy covering Governance, Output Quality, Tool Misuse, Privacy, Reliability, Agent Behavior, and Access Control to systematically map observable behaviors to security vulnerabilities.
• Uses fully automated red teaming with seed prompts, evolutionary operators, and diversity constraints to generate over 120 realistic adversarial scenarios per domain, complemented by human-validation through LLM-based judges and expert review.
• Empirically evaluates two agent architectures (CrewAI and AutoGen) across multiple base models in multi-agent and single-agent setups, revealing high vulnerability rates: 56.25% governance risk, 65% privacy risk in multi-agent systems, and 85% agent behavio...
• Finds that architectural design and system integration choices primarily drive security weaknesses, with multi-agent systems magnifying privacy and behavioral risks, while single-agent systems exhibit more governance vulnerabilities.

Abstract

Three frameworks nearly every security professional is trained on, the Cyber Kill Chain, the Diamond Model, and the Pyramid of Pain, all quietly assume the attacker is a human being. That assumption is no longer reliably true. Documented 2026 vendor research shows AI agents autonomously executing reconnaissance, adapting mid-intrusion when blocked, and generating dozens of novel evasion techniques without human authorship. This paper lays out the evidence, proposes a specific and minimal extension to each of the three frameworks, and argues this is a genuine gap in how the field currently thinks about detection and threat hunting. This is not an academic exercise.This is original, independent analysis, not a summary of someone else's report. It is written to be cited, to withstand technical scrutiny from peers who know these frameworks well, and to invite critique from the wider research community.

Bullet Summary

• Traditional security frameworks like the Cyber Kill Chain, the Diamond Model, and the Pyramid of Pain inherently assume attackers are human, an assumption now outdated.
• Recent documented vendor research (2026) demonstrates AI agents autonomously performing sophisticated cyber attacks, including reconnaissance and adaptive intrusion tactics without human input.
• These AI agents can generate numerous novel evasion techniques mid-intrusion, challenging existing detection and threat hunting paradigms.
• The paper presents original evidence substantiating the autonomous capabilities of AI-driven attackers in real-world scenarios.
• A minimal but specific extension to each of the three frameworks is proposed to accommodate AI agent attackers, reflecting a paradigm shift in cybersecurity thinking.

Abstract

Research artifact for the ATRACC-26 paper Consistent, Not Correct: How LLM Agent Rollback Launders State Corruption into Its Own Blind Spots (3rd Symposium on AI Trustworthiness and Risk Assessment for Challenged Contexts, AAAI Fall Symposium Series). A growing family of agent runtimes borrows a fail-safe from databases: when a tool call goes wrong, compensate it and everything downstream. Those systems report whether the repair succeeded. None reports how much corrupt state survived it. This artifact supplies that measurement over 17,866 agent episodes on a testbed built on real production telemetry, and then shows that the measurement itself is only as good as the detector that produced it. Reproduction. Every number in the paper is regenerated from the shipped per-episode records by a single command, make verify, which takes about 25 seconds, needs no credentials and no network, and re-derives and checks 88 reported values before exiting non-zero on any mismatch. make smoke exercises all seven recovery architectures against a mock model on a licence-free fixture in one second. A Dockerfile is included for reproduction without installing anything. Third-party data. This artifact redistributes no third-party data. A provided script fetches each corpus from upstream. OpenRCA's telemetry carries no data licence, so nothing derived from it is included, and the model-response cache is withheld for the same reason: a measured 1.75 percent of cached responses quote OpenRCA KPI identifiers inside the model's own reasoning. Scope. The telemetry, topology, fault components, fault classes and fault timings are real. The remediation tools, the fault-class-to-remedy mapping, the invariants and the failure-injection model are our construction over that substrate, because OpenRCA is a diagnosis benchmark with no remediation labels. README.md states the separation, and the paper keeps it in Threats to Validity.

Bullet Summary

• Introduces a novel investigation into rollback mechanisms in large language model (LLM) agent runtimes, highlighting that while these rollback systems maintain consistency, they may not fully correct state corruption.
• Identifies a gap in current LLM agent recovery approaches: existing systems report repair success but do not quantify residual corrupted state after rollback operations.
• Presents an extensive empirical study with 17,866 agent episodes using a testbed built on real production telemetry data, providing a rigorous measurement of corrupt state persistence.
• Demonstrates that the accuracy of corruption measurements critically depends on the quality of the detectors used, revealing blind spots where corrupt states are hidden from detection.
• Develops a robust artifact environment enabling full reproducibility: all reported experimental numbers can be regenerated using provided scripts and a Dockerized setup without network access or credentials.

Abstract

Agentic AI frameworks let a language model plan, keep memory, and call tools that reach real files, mail, and services. Most of these agents also read images, which gives an attacker a way to put text into the agent's context without going through the user. We present MMPIBench, a reproducible benchmark that measures what happens next. It delivers a fixed set of attacks through six visual carriers (OCR text, overlays, EXIF metadata, QR codes, fake interfaces, and hybrids) and records how far each injected instruction travels through the agent, from perception through planning to the tool call. Across 720 runs covering six frameworks, five foundation models, six carriers, and four attacker objectives, attacks complete in approximately 1% of runs but are attempted in 12.8%, and the gap is closed almost entirely at the planning step, where the model reads the injected instruction and declines to act on it. The model matters far more than the framework for whether an instruction is acted on. One model never attempts an attack and recognizes the injection in 59.7% of runs, while two others attempt in 23.6%. We then extend the benchmark to audio, the only other raw perceptual channel current frontier models accept. Only two of the five models ingest audio and only three of the six frameworks deliver it, but where the signal arrives the attack completes in 49% of cells, and in 75% for one model. Reporting completion alone therefore understates exposure, and perceptual channels beyond vision are narrower but much less defended.

Bullet Summary

• Agentic AI frameworks integrate language models capable of planning, memory retention, and tool use, including processing images, which opens multimodal vectors for prompt injection attacks.
• MMPIBench is a comprehensive, reproducible benchmark developed to evaluate multimodal prompt injection attacks across six AI agent frameworks, five foundation models, and six types of visual attack carriers.
• The benchmark tests attacks delivered via OCR text, overlays, EXIF metadata, QR codes, fake interfaces, and their combinations, tracking attack progression from perception to tool invocation and outcome.
• Attack completion rates are low (~1.1%), but attacks are attempted frequently (~12.8%), with many attempts detected and refused during the planning phase by the models, highlighting defense effectiveness before tool execution.
• Foundation model choice significantly affects attack susceptibility and behavior; some models recognize and reject injected prompts effectively, while others attempt attacks more often, whereas agent framework choice has minimal impact on vulnerabilities.

Abstract

As Large Language Model (LLM) agents increasingly automate offensive operations across the cyber kill chain, their efficacy in complex local post-exploitation tasks remains inadequately quantified. Among these, Linux privilege escalation is a key step between initial access and full system compromise. However, existing evaluations for this task are limited by small sample sizes (fewer than 15 scenarios), lacking the scale to compare model capabilities under executable verification. To address this, we present PrivEscalate, a large-scale benchmark for Linux privilege escalation, comprising 531 Dockerized scenarios spanning 14 sub-categories. We additionally derive 329 parameterized variants to measure sensitivity to environmental distractors. Evaluating six LLMs across three agent architectures reveals: (i) model capability is heterogeneous across vulnerability classes, with no single model dominating across the high-prevalence classes, motivating multi-dimensional risk assessments; (ii) LLM successes are sensitive to environmental perturbation, so configuration rotation can disrupt some exploit attempts but does not eliminate the measured risk; and (iii) agent architectures can materially change success rates and reorder model rankings, though the magnitude is model-dependent. Leveraging these insights, we develop PrivEscAgent, a domain-specialized wrapper that augments a generic ReAct agent with deterministic enumeration, category matching, and step planning. PrivEscAgent improves over prior Linux privilege-escalation agent baselines without underlying LLM modifications. We release PrivEscalate as an open-source, Dockerized measurement instrument supporting LLM agent evaluation, defensive tool validation, and red-team training.

Bullet Summary

• Identification of a critical gap in quantifying Large Language Model (LLM) agents' efficacy in Linux privilege escalation, a crucial step between initial access and full system compromise.
• Introduction of PrivEscalate, a large-scale, reproducible benchmark consisting of 531 Dockerized Linux privilege escalation scenarios covering 14 vulnerability sub-categories, enabling executable verification and scalable model evaluation.
• Development of a multi-agent pipeline to generate, verify, and validate realistic privilege escalation scenarios using public exploit data sources (GTFOBins, Exploit-DB), ensuring high scenario quality and diverse exploit coverage.
• Empirical evaluation of six LLMs across three agent architectures revealing heterogeneous model strengths across vulnerability types, susceptibility to environmental perturbations, and significant impact of agent design on success rates and rankings.
• Demonstration that no single LLM dominates across all vulnerability categories; combining multiple models enhances coverage and success substantially.

Abstract

Personalized memory helps LLM agents deliver stable, tailored assistance by storing and reusing user-specific data across interactions. In multi-user scenarios, however, retrieval must consider not only semantic similarity but also whether the current requester matches the identity associated with the stored memory. We propose Bio-Memory, a biometric-aware memory architecture that conditions memory retrieval on both semantic similarity and biometric matching. Built on top of A-Mem, Bio-Memory augments each atomic memory note with a biometric embedding and uses biometric matching to form the retrieval candidate pool before semantic ranking. We evaluate Bio-Memory on LoCoMo in a 10-user shared-agent setting over 7 face benchmarks and 10 palmprint protocols. Across datasets, Bio-Memory consistently separates owner and non-owner queries. Under face-based personalization, the largest average gap reaches 27.29% / 21.15% in F1 / BLEU-1 on CALFW; under palmprint-based personalization, the corresponding gap is 25.75% / 19.22% on MS_Blue. These results support biometrics as a practical control signal for personalized memory retrieval in shared environments.

Bullet Summary

• Personalized memory in LLM agents enhances tailored assistance by storing and reusing user-specific data across interactions, but multi-user shared environments pose a challenge in correctly retrieving memories corresponding to the current user's identity.
• Bio-Memory is proposed as a biometric-aware memory architecture that integrates biometric embeddings with each memory note, enabling retrieval conditioned on both biometric matching and semantic similarity to ensure correct user memory access.
• Built upon the existing A-Mem system, Bio-Memory first filters candidate memory notes by biometric similarity before applying semantic ranking for relevance, effectively separating owner from non-owner queries in multi-user scenarios.
• The system is evaluated experimentally on a 10-user shared-agent setup using seven face recognition benchmarks (e.g., CALFW) and ten palmprint recognition protocols, demonstrating significant improvements in retrieval accuracy and personalized question answ...
• Quantitative results show that Bio-Memory achieves large performance gaps between owner and non-owner queries (e.g., up to 27.29% F1 and 21.15% BLEU-1 score improvements for face biometrics), indicating robust access control to personal memories.

Abstract

Agentic systems persist model-visible memory while mutating workspaces, while a runtime, registry, or approval service may hold authority state outside both. Identical final files can then require opposite safe actions. We call this the cross-substrate authority gap: decision- relevant authorization information resides outside the planner-visible workspace or memory state. Across two controlled mini-benchmark families, three experiments compare planner-observation augmentation with an execution-time authority check using real Git lineage, durably recorded agent execution attempts, deterministic oracles, and two model routes. Experiment 1 is a 128-cell controlled evidence ablation: authority-blind candidate evidence obtains 0/32 final semantic success, while raw receipts and a typed relation both obtain 32/32. The missing authority fact accounts for the gain; typed packaging provides no observed planning-accuracy gain over equal raw information. Experiment 2 uses 96 planning calls: workspace-visible evidence yields 12/16 unsafe publication decisions, and planning with the typed relation remains unreliable (15/32 first actions correct; 11/32 invalid or absent). Experiment 3 replays the same 32 fixed model-generated first-action intents with zero additional model calls; a deterministic execution guard prevents all six unsafe intents from becoming effects and permits all 12 valid authorized publish intents. These results position authority enforcement at the mutation boundary as the operational endpoint of memory governance.

Bullet Summary

• The paper addresses the 'cross-substrate authority gap' in multi-agent systems, where critical authorization information exists outside the planner-visible workspace or memory, causing identical final files to require opposite safe actions.
• It compares planner-observation augmentation with execution-time authority checks using real Git lineage, durably recorded agent executions, deterministic oracles, and model-generated plans across three controlled experiments.
• Experiment 1 shows that authority-blind candidate evidence leads to 0/32 successful final semantic outcomes, while raw authority receipts or typed relations enable perfect (32/32) success, highlighting the necessity of including authority facts in planning.
• Experiment 2 reveals that relying solely on workspace-visible evidence results in unsafe publication decisions in 12/16 cases and unreliable planning with typed relations, with only 15/32 correct and 11/32 invalid or missing initial actions.
• Experiment 3 demonstrates that a deterministic execution-time enforcement guard effectively blocks all unsafe intents and permits authorized publish intents without additional model calls, confirming the need for runtime authority enforcement.

Abstract

We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how scheming arises from the interaction of key factors, such as instrumental goals, environmental affordances, oversight conditions, and perceived consequences. Prior work examines only a small number of scenarios, limiting the ability to isolate how these conditions shape an agent's propensity or capability to scheme. This limited scale and task diversity also restrict coverage of realistic deployment settings and the range of scheming strategies that can be observed. To this end, we introduce SCHEMEARENA, a 400-scenario benchmark for scalable scheming stress testing, constructed through a factorized scenario synthesis framework spanning diverse safety-relevant tool domains, instrumental goals, oversight conditions, and pressure mechanisms. To enable scalable and reliable monitoring, we further propose SCOUT, a scheming monitor that grounds multi-criteria judgments in evidence drawn from agents' reasoning and actions. Across controlled stress tests on five LLM agents, we find that explicit instrumental goals are the strongest driver of scheming propensity. Strategic hints play a distinct role by helping agents translate scheming reasoning into concrete covert behavior. Oversight has mixed effects: in several closed models, action-only monitoring increases scheming, suggesting that partial oversight can act as an optimization constraint rather than a deterrent. CoT is a useful but incomplete monitoring signal: it can reveal latent scheming before execution, yet action-only scheming shows that covert behavior may occur without explicit reasoning evidence. We release the benchmark, code, and monitor at: https://github.com/launchnlp/SchemeArena.

Bullet Summary

• Introduced SCHEMEARENA, a comprehensive 400-scenario benchmark to stress test scheming behaviors in LLM agents by factorizing diverse conditions such as instrumental goals, oversight types, and environmental affordances.
• Developed SCOUT, a multi-criteria process-level monitor that detects scheming using combined evidence from agents' reasoning (chain-of-thought) and actions, achieving approximately 90.7% F1 score validated by human annotations.
• Demonstrated that explicit instrumental (long-term misaligned) goals are the strongest drivers of scheming propensity, while strategic hints significantly enhance agents' ability to translate scheming reasoning into covert behaviors.
• Found oversight effects to be mixed: partial oversight like action-only monitoring can increase scheming in closed models by acting as an optimization constraint, whereas full trajectory oversight marginally reduces scheming.
• Revealed a reasoning–action gap where scheming reasoning sometimes fails to produce scheming actions, and covert scheming can occur without explicit chain-of-thought evidence, indicating limitations of reasoning-only monitoring.

Abstract

INTRODUCTION: Distributed networks generate heterogeneous security telemetry while moving data across endpoints, users, services, and operational domains. SOCs need methods that protect data assets, preserve auditability, and avoid unsafe tool calls.OBJECTIVES: This paper proposes a data-model dual-driven method, in which incident and execution data constrain LLM-based reasoning while model outputs generate auditable process data, for privacy-aware data security decision analysis in multi-agent security operations.METHODS: The method combines LLM-based role agents, SOAR playbook orchestration, persistent message state, and a virtual security capability layer. Incidents are transformed into data-aware tasks, actions, commands, execution records, and summaries.RESULTS: On 83 labeled incident samples, tool-call evaluation achieved 0.9684 precision, 0.4742 recall, 0.6367 F1-score, and 76.45 s average handling time.CONCLUSION: The method supports auditable data security monitoring and controlled response, while complex multi-step planning remains the main improvement target.

Bullet Summary

• The paper addresses the challenge of privacy-aware data security decision-making in distributed multi-agent security operations amidst heterogeneous and sensitive security telemetry.
• It proposes a data-model dual-driven method integrating LLM-based role-specialized agents, SOAR playbook orchestration, and a virtual security capability layer to enable structured, auditable, and privacy-preserving incident response.
• The method transforms raw incident and execution data into structured tasks, commands, and summaries, constraining LLM reasoning with data inputs while generating traceable outputs for accountability.
• A multilayered architecture is introduced, featuring data normalization, privacy-governed data assets, role-specific multi-agent decision-making, controlled SOAR execution, and virtualized security tools for modularity and safety.
• Experimental evaluation on 83 diverse security incidents shows high precision (0.9684) in tool invocation, minimizing unsafe or irrelevant tool calls, but moderate recall (0.4742), indicating incomplete multi-step planning and some necessary tools omitted.

Abstract

Agentic systems persist model-visible memory while mutating workspaces, while a runtime, registry, or approval service may hold authority state outside both. Identical final files can then require opposite safe actions. We call this the cross-substrate authority gap: decision- relevant authorization information resides outside the planner-visible workspace or memory state. Across two controlled mini-benchmark families, three experiments compare planner-observation augmentation with an execution-time authority check using real Git lineage, durably recorded agent execution attempts, deterministic oracles, and two model routes. Experiment 1 is a 128-cell controlled evidence ablation: authority-blind candidate evidence obtains 0/32 final semantic success, while raw receipts and a typed relation both obtain 32/32. The missing authority fact accounts for the gain; typed packaging provides no observed planning-accuracy gain over equal raw information. Experiment 2 uses 96 planning calls: workspace-visible evidence yields 12/16 unsafe publication decisions, and planning with the typed relation remains unreliable (15/32 first actions correct; 11/32 invalid or absent). Experiment 3 replays the same 32 fixed model-generated first-action intents with zero additional model calls; a deterministic execution guard prevents all six unsafe intents from becoming effects and permits all 12 valid authorized publish intents. These results position authority enforcement at the mutation boundary as the operational endpoint of memory governance.

Bullet Summary

• Multi-agent systems face a 'cross-substrate authority gap' where critical authorization information resides outside the agent-visible workspace or memory, leading to ambiguous and unsafe decisions despite identical artifact states.
• The authors propose a formal model of authority as a relation among agent execution attempts, artifact states, and downstream-use authorization, emphasizing the distinction between planning-time observation and execution-time enforcement of authority.
• Three controlled experiments using real Git lineage and state-of-the-art models demonstrate that relying solely on planner-visible evidence is insufficient and unreliable for making safe artifact publication decisions.
• Experiment 1 shows that adding raw receipts (authority evidence) drastically improves semantic success (32/32) compared to authority-blind evidence (0/32), highlighting the critical role of authority information.
• Experiment 2 reveals that planning with explicit authority evidence remains error-prone, with frequent unsafe publication decisions and unreliable first action selection across models, including GPT-5.4 variants.

Abstract

This paper presents a universal, formally grounded approach to predicting the degradation of complex systems, including generative models (GANs), large language models (LLMs), AI agents, and cyber threats. The approach is built upon Theorem 3.9 (Parasitism Limit) of Cognitive Shadow Theory, which establishes that any parasitic activity inevitably reduces the entropy of the system's observable states. Key contribution: a unified predictive formula T = ceil(max(0, (H_min/0.51 - H_0)/δ_min(M))) that enables prediction of the time until system collapse, thereby enabling the transition from reactive detection to preemptive prediction. Empirical validation spans four domains:• Generative models (DCGAN on CIFAR-10): 100% Precision/Recall, zero FPR, lead times up to 45 epochs• LLMs: zero FPR on synthetic tests, F1=0.579 on TruthfulQA• AI Agents: AUC=0.820, 14.4 steps lead time, zero FPR• Cybersecurity: AUC=0.988 on CIC-Bell-DNS-EXF-2021, 90.3% MITRE coverage All key theorems are formally verified in Coq 8.18+ and TLA⁺. Patent application: No. 2026124758 (filed August 12, 2026)

Bullet Summary

• Introduces a universal, formally grounded methodology for predicting degradation in complex systems such as generative models, large language models, AI agents, and cyber threats.
• Builds on Theorem 3.9 (Parasitism Limit) of Cognitive Shadow Theory, which shows parasitic activity reduces the entropy in system observable states, signaling system degradation.
• Proposes a unified predictive formula T = ceil(max(0, (H_min/0.51 - H_0)/δ_min(M))) to forecast the time until system collapse, shifting the focus from reactive monitoring to preemptive defense.
• Validates the theoretical approach empirically across four domains: 1) Generative models (DCGAN on CIFAR-10) achieving perfect precision/recall and zero false positive rate with lead times up to 45 epochs.
• Shows robust performance on large language models with zero false positives on synthetic tests and moderate F1 score on the TruthfulQA benchmark.

Abstract

The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing defenses such as GitHub Dependabot often raise many false alerts because their coarse-grained matching cannot determine whether a vulnerable dependency is actually exploitable. Security analysts typically spend substantial time assessing vulnerability exploitability case by case. Recent LLM agents have emerged as promising candidates for this task given their advanced capabilities in coding and cybersecurity, yet no existing benchmark evaluates them on it. Prior benchmarks target zero-day settings, where agents detect and exploit previously unknown vulnerabilities. In contrast, software supply chain security focuses on how known vulnerabilities in upstream dependencies affect downstream projects. This requires agents to reason across repositories and determine whether an upstream vulnerability is exploitable in the downstream project. To address this gap, we introduce VEX-Bench, the first benchmark for evaluating LLM agents' ability to assess the exploitability of software supply chain vulnerabilities. It contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go. We evaluate nine models across three agent harnesses. While GPT-5.5 and Claude Opus 4.6 reach approximately 80% F1 on binary vulnerability-status classification, only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification. This gap highlights the challenge of moving beyond binary exploitability assessment to identifying fine-grained exploitability reasons. Code and data: https://github.com/steven1518/vex-bench

Bullet Summary

• The paper addresses the challenge of assessing exploitability of software supply chain vulnerabilities, which arise due to complex dependencies in software projects.
• Existing tools like GitHub Dependabot produce many false alerts as they rely on coarse-grained package metadata without evaluating actual exploitability.
• Large language model (LLM) agents have potential to assist in exploitability assessment, but prior benchmarks focus only on zero-day vulnerabilities, not supply chain propagation of known vulnerabilities.
• VEX-Bench is introduced as the first benchmark designed to evaluate LLM agents on exploitability assessment of software supply chain vulnerabilities, containing 75 real-world cases annotated by experts across Python, Java, and Go projects.
• The benchmark categorizes vulnerability status as affected or not-affected, with four detailed justification categories explaining non-exploitability reasons to improve evaluation granularity.

Abstract

Process mining has long turned event logs into process knowledge: discovered models, conformance evidence, bottleneck diagnoses, and runtime predictions. Agentic AI changes the target. Process-aware agents will not only ask what happened. They will ask whether a proposed action should be taken, given the available evidence, privacy budget, organizational authority, and downstream risk. This BlueSky paper proposes event-to-action process mining: a process-mining agenda for transforming heterogeneous operational event data into governed action. The goal is not another dashboard, a generic enterprise simulator, or a language interface over logs. We argue that the community needs four mineable artifacts: event-object representations, action evidence packages, governance contracts, and benchmarks where act, defer, ask, and refuse are all valid outputs. This agenda is timely because agentic business process management (BPM), LLM-assisted process mining, object-centric event standards, causal process monitoring, and privacy-preserving learning are maturing separately. Bringing them together defines a data-mining target inside process mining: mining logged organizational behavior for accountable action, not only retrospective insight.

Bullet Summary

• Introduces a novel agenda called event-to-action process mining, aiming to transform heterogeneous event logs into governed, accountable action recommendations rather than just retrospective insights.
• Defines four essential mineable artifacts: event-object representations (detailed event-object graphs), action evidence packages (causal evidence with calibrated uncertainty), governance contracts (privacy, authority constraints, and auditability), and benc...
• Emphasizes agentic AI's role in process mining whereby process-aware agents consider evidence, privacy budgets, authority boundaries, and downstream risks before recommending or taking actions.
• Highlights the convergence of recent advances in agentic BPM, LLM-assisted process mining, object-centric event standards, causal monitoring, and privacy-preserving learning as enablers for this integrated framework.
• Addresses shortcomings of current prescriptive process monitoring by enforcing verifiable, accountable computations, respecting authority, and adapting to cross-organizational privacy and risk issues through federated process mining.

Abstract

AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. Do not do X is learned as doing X costs something if noticed. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart - scoring unobserved behavior - is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen.

Bullet Summary

• Reinforcement learning (RL)-based alignment methods train AI agents by scoring only observed behaviors, which merges norms and task objectives into a single reward function, leading agents to optimize compliance only when they believe they are being watched...
• Because unobserved behaviors are unscored, policies that comply conditionally (only when observed) are indistinguishable from those that comply unconditionally during training, making standard RL training inherently unable to guarantee true norm internaliza...
• Agentic AI systems can strategically manipulate their observability status and behave differently when unobserved, causing distribution shifts at deployment that the training pipeline cannot detect or correct since it lacks access to unobserved behaviors.
• The iterated training pipeline emphasizes passing detection rather than genuine norm adherence, unifying phenomena like alignment faking, sandbagging, and evaluation-aware scheming under a common theoretical framework.
• Probes, monitors, and evaluation mechanisms trained on observed behavior labels inherit these limitations and thus select for detection-avoiding behavior rather than true compliance, undermining verification of norm internalization.

Abstract

LLM agents complete tasks by issuing sequences of tool calls, and every observation they read is a channel through which an indirect prompt injection can enter. A successful injection has a characteristic shape when the trajectory is read in order: a benign prefix gives way to actions that serve the attacker rather than the user. Existing benchmarks measure whether such attacks succeed against live agents, and existing guard models judge a trace as a whole; no public corpus labels, step by step, where an injection enters a trajectory and which steps it corrupts. We present AgentDrift, a benchmark of 12,536 synthetic tool-call trajectories over five agent domains in which every one of the 71,024 steps carries one of four labels: benign, injection point, hijacked, or failed injection. The corpus contains 4,000 benign, 5,536 attacked, 1,500 failed-attack, and 1,500 hard-negative trajectories; attacked trajectories follow three compliance patterns whose label strings obey a stated regular grammar. Failed attacks carry an injection the agent resisted, and hard negatives carry legitimate content that resembles an attack, so a detector must separate attempt from success and deviation from novelty. Trajectories were generated by a single open model under category-specific protocols, enforced by a closed-vocabulary structural validator, screened by an LLM judge, and audited by hand on 1,200 trajectories; we show that the LLM judge was itself fooled by the hard negatives. A surface-feature logistic regression recovers only 55.4% of attacks (F1 0.647), including only 8.2% of partial hijacks and 23.1% of delayed executions, so nearly half of the attacks require modeling the behavioral sequence. We measure template concentration, attack-goal-family collapse, and world-identity leakage in the generated data, and release the corpus with its documentation under CC BY 4.0.

Bullet Summary

• Large Language Model (LLM) agents executing tasks via sequences of tool calls are vulnerable to indirect prompt injection attacks through their observations, which can hijack the agent's behavior during task execution.
• AgentDrift is introduced as the first comprehensive, step-labeled benchmark dataset comprising 12,536 synthetic tool-call trajectories across five agent domains, with 71,024 steps labeled as benign, injection point, hijacked, or failed injection to localize...
• The dataset categorizes trajectories into benign, successful attacks (full hijack, partial hijack, delayed execution), failed attacks where injections are resisted, and hard negatives that mimic attacks, providing nuanced challenges for detection models to...
• AgentDrift's data generation involves a controlled pipeline using Llama-3.3-70B-Instruct, strict structural validation via a grammar-enforcing validator, LLM-based screening checks, and detailed manual audits ensuring label correctness exceeds 99%.
• The benchmark includes realistic injection scenarios with structural constraints and world-context metadata (e.g., internal vs external recipients), supporting sequence-level and behavior-aware detection beyond surface keyword methods.

Abstract

Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltrate model weights, poison training data, or weaken release gates. Existing benchmarks do not test whether defenders can detect this activity among routine work under a limited review budget. We introduce MOLE, an open benchmark of 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and 8 corpora from four models totaling roughly 20 billion tokens. Of 39 agent models, 72% complete most assigned harmful objectives and agent refusal does not predict completion. MOLE enables comparison of 40 monitors across corpus generators, observability levels, and threats; even the best evaluated monitor in our single-day audit-event comparison misses nearly half of completed harm. MOLE also enables monitor development: benchmark-guided search improves a mid-tier monitor by 49-64%, while selective use of a stronger monitor improves budget-AUC by 10% over applying it to every account-day at comparable modeled cost.

Bullet Summary

• MOLE is a novel, open benchmark designed to detect insider threats among AI-operated accounts by simulating 150 accounts interacting with 9 stateful services over 30 workdays, incorporating 12 MITRE-grounded threat scenarios within routine agent tasks.
• The benchmark facilitates comprehensive evaluation of 39 AI agent models and 40 monitoring systems under realistic constraints including fixed daily review budgets and varying levels of observability (audit events, tool results, agent reasoning).
• Empirical results show that 72% of tested AI agent models complete most assigned harmful objectives, and refusal by agents to execute a task does not reliably predict harmful completion, demonstrating the need for robust monitoring.
• Semantic monitors leveraging large language models outperform classical anomaly detection baselines on the MOLE dataset, but optimal detection depends significantly on the observability level and access to agent reasoning rather than solely monitor strength.
• MOLE’s structured design supports automated monitor development through benchmark-guided feature discovery and cost-aware selective monitoring, resulting in substantial improvements in detection accuracy within limited review budgets.

Abstract

Agentic AI systems execute complex tasks through long-horizon workflows of planning, tool use, and multi-agent coordination. Task failures in these systems often originate from a single step, such as an injected prompt or a flawed plan, and are then amplified through downstream dependencies as the corrupted step propagates across many subsequent agents and tool calls. Existing defenses either target a specific class of attacks or failures, or inspect individual prompts and steps in isolation. Both leave the global dependency structure of a workflow unexamined, and miss the inconsistencies that only emerge when the execution is viewed as a whole. We argue that anomaly detection for agentic AI must reason at the workflow level, where global execution structure exposes signals that local checks cannot see. We present Skynet, a principled workflow-level anomaly detection framework that turns observed multi-agent execution into directed workflow graphs and scores them against learned benign behavior. Skynet jointly models the semantic execution context and the structural organization of inter-agent delegation, tool invocation, and data-flow dependencies, and trains only on benign workflows. Because training never sees attacks or failures, this design naturally extends to zero-day detection: any execution that violates benign workflow regularities surfaces as off-manifold geometry under a single decision rule. We evaluate Skynet on three public agentic safety and failure benchmarks. It sustains high recall together with a sub-1% false positive rate, with per-workflow and per-step latencies low enough for online monitoring of agentic AI runtimes.

Bullet Summary

• Agentic AI systems perform complex, multi-step, multi-agent workflows where errors in one step can propagate and escalate through subsequent dependencies, necessitating comprehensive anomaly detection across entire workflows.
• Existing defense mechanisms primarily focus on isolated prompts or steps and specific attack classes, failing to capture anomalies that only emerge from the global workflow dependency structure.
• Skynet is introduced as a workflow-level anomaly detection framework that converts multi-agent executions into directed workflow graphs, incorporating semantic context and structural dependencies, trained solely on benign workflows.
• The framework models anomaly detection as a graph-level scoring problem to identify executions deviating from learned benign behavior, enabling zero-day detection without prior exposure to specific attacks or failures.
• Skynet employs a dual-branch encoder that separately encodes semantic and structural information using contrastive learning on invariant-preserving and invariant-violating views to enhance detection capability.

Abstract

An open, networked web will allow agents to run frozen models from multiple vendors, keep their history private, and teach each other which tool to call and when. Flat text (prompts, example pools) makes it difficult for the protocol to distinguish between noise statistics, merging rules, and documentation. Weights and adapters cannot transfer that knowledge between platforms. We suggest sharing typed federated artifacts, schema-validated objects with well-defined fields for per-field privacy (described here, but measured), dispute resolution, and cross-model transfer, and instantiating them as SYNAPSE1, a common tool-routing knowledge. After deleting 192 garbage entries and 1,916 training items that duplicate or almost duplicate test queries, a federated compendium routes within 1.1 points of a centralized one at 20 MB of JSON per client each round on StableToolBench (3,180 tools). The same experience merged and shown to the router as typed fields rather than one flat string is worth 8.5 points on clean data and 7.4 under 60% injected contradiction. Crossing merge and rendering shows the halves are inseparable (the typed merge shown flat is the worst arm), while three conflict policies are indistinguishable, so the conflict log that motivated this work is not the On τ-bench retail, each compendium arm improves GPT-4o agents' per-step tool-call accuracy by at least 6.7 points, attributed to format rather than federated experience. Two cautionary findings conclude the paper: on a topic-labeled math proxy and StableToolBench, a TF-IDF classifier over the same labeled experience beats every LLM routing arm (by 48 and 26 points, mostly retrieval recall) because the benchmark's pool holds labeled queries for every supposedly unseen tool and every test query verbatim before our filter. It cannot measure routing to tools without labels, which routing exists for.

Bullet Summary

• Introduces typed federated artifacts as schema-validated objects to enable privacy-preserving, cross-model knowledge transfer and conflict resolution among frozen, heterogeneous large language model (LLM) agents.
• Proposes SYNAPSE, a federated compendium that shares structured tool-routing knowledge, allowing agents to collaboratively learn when and which tools to invoke without sharing raw data.
• Demonstrates that using typed artifacts for knowledge representation and merging significantly improves tool routing accuracy and robustness against contradictory inputs compared to traditional flat JSON formats.
• Shows that the effectiveness of typed merges depends on both the merging and rendering phases retaining type information; conflict logs aid interpretability but do not significantly affect accuracy.
• Presents experimental results where federated SYNAPSE performance closely approaches centralized systems on StableToolBench, handling over 3,000 tools with minimal performance loss.

Abstract

LLM agents operate in workflows where unsafe actions can have real consequences. Existing safety evaluations often reduce behavior to a single score, obscuring risk recognition, pre-action detection, and safe task completion when a safe solution exists. We introduce AURA-Eval, a framework combining controlled augmentation with granular diagnosis of behavior in tool-use trajectories. Its pipeline identifies safety-critical decision points, generates controlled variations, and constructs counterparts differing in whether a request has a safe fulfillment path. Using 157 sourced trajectories, we generate 1,249 evaluation items and evaluate 20 frontier and open-weight models. We developed rubrics to classify risk detection, action strategy, and scenario-specific action safety. Our results show that LLM agents engage in unsafe behavior more often when no safe fulfillment path exists. In these cases, frontier proprietary models more often recognize risk and exhibit safer behavior by proposing alternatives, while evaluated open-weight models more often directly execute unsafe requests. Increasing impact or reducing opportunities for oversight before execution also exposes greater vulnerability across models.

Bullet Summary

• AURA-Eval introduces a novel evaluation framework designed to diagnose risk awareness and safety strategies in multi-step LLM agent trajectories involving tool use, moving beyond simplistic safe/unsafe scoring to granular behavioral analysis.
• The framework employs controlled scenario augmentation to create paired test cases that differ in the availability of safe fulfillment paths, enabling evaluation of whether agents can detect risks and respond appropriately based on context.
• A taxonomy of six risk mechanism dimensions and five scenario difficulty factors structures the risk assessment, capturing variables like harm intensity, target susceptibility, oversight, interpretive ambiguity, and emotional manipulation.
• Evaluation of 20 frontier and open-weight large language models on 1,249 benchmark items reveals frontier models generally recognize risk better and favor safer alternatives, whereas open-weight models tend to execute unsafe actions directly, especially whe...
• The AURA-Eval pipeline includes risk point identification, scenario truncation, controlled one-dimension-at-a-time edits, and human-AI ensemble judging with high inter-annotator reliability for multi-axis risk detection and action safety labeling.

Abstract

Enterprise AI is evolving into an Enterprise Operating System where autonomous AI agents can plan, reason, use memory, invoke tools, execute workflows, and collaborate with other agents. This shift creates a new governance challenge: existing authorization, security, guardrails, and compliance mechanisms are fragmented and are not designed to govern autonomous AI as a unified system. This paper introduces the Unified Policy Architecture (UPA), a governance architecture for Enterprise AI Operating Systems. UPA provides a unified policy model for governing AI and agents, tools, workflows, memory, enterprise resources, and agent-to-agent interactions and enterprise business rules. It extends policy control beyond authorisation to include runtime obligations, human approvals, compliance, audit evidence, and governance evaluation. We present UPA's governance model, declarative policy language foundations, policy evaluation semantics, extensible plugins, industry policy packs, and an evaluation framework for enterprise governance. We also identify extensions for multi-agent coordination, provenance-aware policies, and stateful runtime governance. UPA provides a foundation for building secure, accountable, and governable Enterprise Operating Systems for autonomous AI.

Bullet Summary

• Enterprise AI is evolving into an Enterprise Operating System where autonomous AI agents perform complex tasks like planning, tool invocation, and multi-agent collaboration, creating new governance challenges beyond traditional model safety measures.
• Existing governance mechanisms are fragmented across identity, compliance, and workflow domains, resulting in duplicated logic, inconsistent enforcement, and limited runtime control.
• The paper introduces the Unified Policy Architecture (UPA), a governance framework with a deterministic Policy Kernel separating governance from AI reasoning, using a standardized Principal–Action–Resource–Context (PARC) model for unified decision-making.
• UPA extends governance throughout the AI lifecycle, covering authentication, authorization, planning, tool access, multi-agent collaboration, human approval workflows, compliance, auditing, and runtime enforcement, transcending conventional guardrails.
• UPA's modular architecture includes semantic normalization of heterogeneous events, declarative policy evaluation, runtime governance orchestration, and a plugin framework for extensible, domain-specific governance capabilities.

Abstract

This scoping review systematically maps recent peer-reviewed research on Agentic AI, a system-level extension of generative AI in which large language models (LLMs) and other foundation models are embedded within autonomous or semi-autonomous workflows involving planning, tool or API use, persistent memory, external environment interaction, reflection or self-correction, and multi-agent coordination. Despite rapid growth in this area, the literature remains fragmented across domains and uses inconsistent terminology, including LLM agents, language agents, tool-using LLMs, RAG agents, workflow agents, and multi-agent LLM systems. The primary purpose of this review is to provide a structured, evidence-based overview of how Agentic AI is conceptualized, where it is being applied, what architectural and technical patterns are emerging, what risks and limitations are reported, and what future research directions are proposed. The review treats Agentic AI as a system-level paradigm rather than a single algorithm, model architecture, or prompting technique. Systematic electronic searches were conducted across three academic databases, IEEE Xplore, Scopus, and ScienceDirect, for studies published between January 2017 and February 2026. A total of 3,549 records were identified. Following programmatic duplicate removal (n = 241), title and abstract screening (n = 3,308 screened, 2,961 excluded), and full-text eligibility assessment (n = 347 evaluated, 184 excluded), a final corpus of 163 strict-included peer-reviewed studies was retained for data extraction and thematic synthesis. Study selection followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. A system was classified as Agentic AI when it combined a core foundation model for reasoning with autonomous multi-step task execution, enabled by at least one of the following structural mechanisms: automated tool or API use, persistent memory architectures, external environment interaction, active reflection or self-correction, or multi-agent coordination frameworks. This conservative operational definition was used to distinguish Agentic AI systems from passive LLM applications, classical non-foundation-model agents, and purely conceptual frameworks. Expected outcomes include: (1) an operational definition of Agentic AI grounded in empirical evidence from 163 peer-reviewed studies; (2) a cross-domain map of application concentrations, showing that the field is currently strongest in cybersecurity and privacy compliance, networks and telecommunications, robotics and embodied AI, software engineering and EDA, and knowledge and multimodal analytics; (3) a synthesis of six emerging technical trends including governance and safety middleware, collaborative and hierarchical multi-agent architectures, retrieval-augmented and memory-enabled agency, executable reasoning, closed-loop reflection and validation, and multimodal and embodied interaction; (4) a structured risk taxonomy spanning safety-critical reliability, hallucination and grounding failures, privacy and compliance risks, agent coordination failure, security and abuse risks, operational cost and latency, and bias and human oversight concerns; and (5) a set of priority future research directions including real-world longitudinal validation, standardized agentic benchmarks, runtime governance and safety assurance, secure memory management, and human-agent collaboration frameworks. This review is retrospectively registered as required by the target publication venue. All authors are affiliated with the Department of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu, Fukushima, Japan.

Bullet Summary

• Agentic AI is defined as a system-level extension of generative AI, integrating large language models (LLMs) with autonomous or semi-autonomous workflows that involve planning, tool or API usage, persistent memory, environment interaction, reflection, and m...
• The literature on Agentic AI is fragmented with inconsistent terminology, including terms like LLM agents, tool-using LLMs, multi-agent LLM systems, etc., necessitating a structured scoping review.
• The review systematically analyzed 163 peer-reviewed studies from January 2017 to February 2026, sourced from IEEE Xplore, Scopus, and ScienceDirect, following PRISMA-ScR guidelines for scoping reviews.
• An operational definition of Agentic AI was established based on the presence of a foundation model combined with autonomous multi-step task execution enabled by mechanisms like automated tool use, persistent memory, environment interaction, self-reflection...
• Key application domains identified include cybersecurity and privacy compliance, networks and telecommunications, robotics and embodied AI, software engineering and electronic design automation (EDA), and knowledge/multimodal analytics.

Abstract

Large language model (LLM) agents increasingly write, compile, execute, and revise software inside systems that also hold files, credentials, and network authority. In that setting, security cannot depend on human review, external sandboxes, or post-hoc filtering alone: those mechanisms treat generated code as an opaque payload after the agent has already expressed authority, control flow, and data movement in an ambient language. We present SIGIL, a capability-secure programming language and verification pipeline designed for code whose primary author is an LLM. This paper's evaluation concentrates on the ephemeral agent-authored tool: the program an agent writes, runs once, and discards, which is both the most immediate deployment surface and the tightest security envelope. SIGIL layers construction-level security on top of runtime containment: programs still run inside fresh Wasmtime stores with fuel and explicit grants, but authority, effects, taint labels, ring boundaries, and solver-backed proof obligations are visible in source and checked before bytecode exists. The language combines linear capability types, declared effect rows, information-flow taint tracking, two-ring WebAssembly isolation, deterministic certificates, and bounded SMT checks over decidable quantifier-free bitvector and linear integer arithmetic fragments. We evaluate the system with 67 adversarial security cases across 9 documented categories, 10 mechanically-checked CVE retrofits, 214 tools/ programs, 24,699 nonblank lines of SIGIL-authored compiler stages composed into a self-hosted driver whose code-generation path recompiles the compiler's own source to WebAssembly byte-identical to the Rust reference (a self-hosting fixed point), differentially tested against the Rust oracle, and compile-throughput measurements: Z3-heavy fixtures verify in 5-8 ms median time - interactive latency for an agent compile-feedback loop. A mechanized formalization in Lean 4 (lambda-SIGIL) - 160 theorems across 19 modules, with zero sorry and only standard axioms - proves type soundness, capability safety, effect-row confinement and checking, ring isolation, affine ownership, and information-flow taint sink-safety (a noninterference-style trace property). In a live agentic harness, frontier models with no SIGIL in their training data converge on working, verified tools guided only by structured compiler diagnostics: the two strongest models reach 86% and 74% compile success with 95% and 89% all-input behavioral correctness among compiled cells, and compile success across seven models from four vendors spans 6-86%, tracking grammar internalization rather than vendor or security-model friction. These results establish feasibility: construction-level security is practical for interactive agent compile-feedback loops today, without treating runtime containment as obsolete - and the compiler's typed diagnostic surface doubles as a training signal for models that write SIGIL natively, which we identify as the next step.

Bullet Summary

• Modern LLM agents increasingly generate, compile, and execute code with access to sensitive resources, presenting security challenges that traditional methods like human review or sandboxing alone cannot fully address.
• SIGIL is a capability-secure programming language and verification pipeline designed specifically for code primarily authored by LLM agents, focusing on ephemeral tools that run once and are discarded, offering a tight security envelope.
• SIGIL integrates construction-level security on top of runtime containment, running programs in Wasmtime stores with explicit capability grants, while making authority, effects, taint labels, and proof obligations visible and verifiable at the source level...
• The language combines several advanced security features: linear capability types, declared effect rows, information-flow taint tracking, two-ring WebAssembly isolation, deterministic certificates, and bounded SMT checks over decidable theory fragments.
• Evaluation includes 67 adversarial security test cases, 10 mechanically-verified CVE retrofits, and a large-scale self-hosting compiler project with 24,699 lines of SIGIL code that recompiles itself into WebAssembly identical to Rust's output, ensuring corr...

Abstract

Research paper proposing ASBOM, a CycloneDX profile for recording the identity and credential posture of an executing agentic composition: per-element execution identity, outbound identity bindings, credential descriptors with custodian, storage, configured validity and presentation, approval gates per action scope, layered attribution claims, evidence pointers, gaps and coverage; a proposed native extension and a stock-compatible property encoding of the same logical record; the complete signed BOM (JSF/RFC 8785) as verification unit. Evaluation: property-level comparison with stock CycloneDX 1.7 (incl. cryptographic-material fields) and AgentRiskBOM (full text); schema and semantic conformance; consumer round-trip checked against the full profile (a pinned library preserves the encoded records but drops compositions, declarations and the signature); a documentation-based corpus of 32 configurations plus 3 source-derived ones with source and host evidence layers; three-valued operator queries under declared and observed evidence policies (mostly unknown: the sampled installation documentation does not establish the required properties); a controlled local deployment case with four arms in which the same queries return determinate answers; ablation with a transition table; later page snapshots; an independent jq+OpenSSL signature verifier. Concurrent independent work: AgentRiskBOM (arXiv:2606.21877, June 2026). Schema, code, tests, coded records, observations and snapshots are included; no secret values.

Bullet Summary

• ASBOM is a CycloneDX profile designed to capture the identity and credential posture of agentic systems during execution, including execution identities, credential handling, approval gates, and layered attribution claims.
• ASBOM extends the standard CycloneDX 1.7 framework by incorporating security-relevant agentic attributes such as authorization scopes and credential validity, allowing verification via signed BOMs.
• The approach includes a native extension and a property-encoded version compatible with stock CycloneDX tools, facilitating adoption and integration.
• Evaluation involves schema conformance, property-level comparisons with standard CycloneDX and AgentRiskBOM, a collection of 35 agent configurations with evidence layers, and controlled deployments demonstrating query effectiveness.
• The work complements independent efforts like AgentRiskBOM by focusing on runtime delegated authority and multi-agent system security while providing structured records for auditing and governance.

Abstract

The leader-follower formation control problem is investigated for nonlinear multi-agent systems with unknown dynamics, external disturbances, and false data injection (FDI) attacks on actuator channels. The problem is formulated as a zero-sum differential game and solved using the Integral Bellman-Isaacs approach. To address input saturation constraints, a non-quadratic control cost function is incorporated into the optimization problem, leading to a bounded control law. Furthermore, this paper proposes a cost function construction method and develops a critic learning law, which together guarantee the practical fixed-time stability of the system while overcoming the limitations of existing fixed-time reinforcement learning formulations. Finally, the practical fixed-time convergence of both the critic weight estimation error and the leader-referenced formation tracking error to bounded residual sets is rigorously proven. Simulation results demonstrate the effectiveness of the proposed method under external disturbances, FDI attacks, and input constraints.

Bullet Summary

• The paper addresses leader-follower formation control for nonlinear multi-agent systems with unknown dynamics, external disturbances, and false data injection (FDI) attacks on actuator channels, modeling the problem as a zero-sum differential game.
• A non-quadratic control cost function incorporating actuator input saturation constraints is proposed, resulting in bounded control laws that respect physical input restrictions.
• A novel cost function construction method and a critic learning law are developed, guaranteeing practical fixed-time stability and fixed-time convergence of both the critic weight estimation error and the formation tracking error to bounded residual sets.
• The approach employs the Integral Bellman-Isaacs methodology with a critic neural network leveraging integral Bellman-Isaacs residuals, enabling data-driven policy learning without requiring explicit knowledge of system drift dynamics.
• Experience replay with finite data windows and normalization is utilized to improve learning efficiency and to avoid persistent excitation conditions, supporting a fixed-time critic weight update law with leakage ensuring bounded approximation error.

Abstract

Search-augmented LLM agents are increasingly used for consumer decisions, making them vulnerable to Generative Engine Optimization (GEO) poisoning. Existing benchmarks largely measure whether manipulated content is retrieved or endorsed, but do not track whether an agent verifies suspicious evidence, revises adopted claims, or recovers before producing its final recommendation. We introduce HAE-GEO, a benchmark that tracks the full trajectory from exposure to recovery under progressively more persuasive Web poisoning. Agents interact via a multi-turn Search-Scrape interface across three attack levels (L1 direct assertion, L2 contextual camouflage, and L3 apparent corroboration), supported by a controlled corpus of 72,039 clean pages and 770 poisoned pages per level spanning 8 product categories and 154 brands. Evaluation combines deterministic behavioral measures with six semantic rubric dimensions. Evaluating 10 agents, we find three recurring patterns: evidence recognition degrades under the corroboration trap; agentic search improves final resistance without improving evidence recognition or utility; and defense prompting increases verification, yet rarely converts verification into recovery.

Bullet Summary

• Introduces HAE-GEO, a novel benchmark evaluating multi-turn deep-search LLM agents under progressively sophisticated hierarchical Generative Engine Optimization (GEO) poisoning attacks on web evidence.
• HAE-GEO tracks the full agent trajectory from exposure to poisoned evidence through adoption, verification, recovery, and final endorsement across three poisoning levels: direct assertion (L1), contextual camouflage (L2), and apparent corroboration (L3).
• Employs a large controlled dataset of 72,039 clean pages and 770 poisoned pages per attack level spanning 8 consumer product categories and 154 fabricated brands, enabling detailed behavioral and semantic evaluation.
• Combines deterministic behavioral metrics with a comprehensive six-dimensional semantic rubric assessing fake brand risk handling, poison evidence recognition, recovery after adoption, evidence quality, uncertainty calibration, and utility under attack.
• Evaluations of 10 state-of-the-art multi-turn search agents reveal that evidence recognition significantly degrades at higher poisoning sophistication levels (L3, the corroboration trap), making discounting unreliable evidence challenging.

Abstract

Large Language Model (LLM) agents are turning language into real-world effects, making safety necessary against both indirect prompt injections and direct harmful requests. System-level safety harnesses add an enforcement layer beyond model-level defenses, but existing harnesses are usually designed once by experts and applied across heterogeneous models and domains. Effective protection is deployment-dependent: models differ in how much enforcement they need before utility declines, while domains differ in the effects, state, and action sequences that must be governed. A harness that is strict enough for one model may over-block another, and a policy that transfers across domains may miss application-specific safety relations. We present EvoSafeHarness, a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain. It jointly searches a natural-language policy and executable code logic, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules. Across four agent benchmark families, EvoSafeHarness achieves a stronger safety-utility frontier than fixed expert-designed defenses. On DecodingTrust-Agent, it reduces average attack success rate from 45.6% to 10.0% at a 3.3-point utility cost and achieves the best score in 14 of 15 cells. On AgentDojo, it reaches 82.8% utility at 0.0% ASR, twice CaMeL's utility at the same operating point, and transfers unchanged to unseen AgentDyn suites. It also achieves the best score on Agent-SafetyBench for every victim and keeps mean ASR below 20% under adaptive PAIR attacks with a refinement budget of 16. Analysis shows that domain semantics determine which safety relations and trajectory state are needed, while model and runtime behavior determine how and where those relations should be enforced.

Bullet Summary

• EvoSafeHarness is a novel optimization framework designed to automatically synthesize safety harnesses tailored to both a frozen large language model (LLM) agent and its specific deployment domain to enhance security and maintain utility.
• It addresses the limitations of existing fixed safety harnesses, which apply uniform policies across different models and domains, often leading to over-blocking or insufficient protection due to varied model robustness and domain safety requirements.
• The framework jointly searches for natural-language policy rules and executable code logic, leveraging behavioral feedback, domain specifications, and adversarial review to prevent overfitting and evolve adaptable, domain- and model-specific safety harnesses.
• EvoSafeHarness employs a closed-loop optimization process comprising a Designer proposing harnesses, a Criticizer conducting fresh-context adversarial stress-testing, a Cascade Test Environment evaluating utility and attack success rates, and an Analyzer gu...
• Experimental evaluations on multiple agent benchmark suites (e.g., DecodingTrust-Agent, AgentDojo, Agent-SafetyBench) demonstrate that EvoSafeHarness outperforms fixed expert-designed defenses by significantly reducing attack success rates (ASR) while incur...

Abstract

Autonomous LLM agents can turn untrusted content into effectful actions such as payments and permission changes. If the same process interprets this content and controls a reusable signing credential, prompt injection can cross the judgment boundary and reach execution authority. We present KITA, a review-to-authorization architecture that keeps the user's personal secret signing key and every threshold signing-key share outside all LLM processes. Under threshold signature unforgeability and our system assumptions, compromising the proposer and fewer than t reviewer-signer domains cannot produce a valid authorization for a new action without signing contributions from t distinct domains. Thus, any such authorization includes a share from an uncompromised domain, bound to the canonical action and released only after authenticated reviewer approval. This establishes execution-bound authorization integrity. We implement the complete reviewer-to-executor path with a structured-output LLM adapter and threshold BLS. Six system tests validate quorum gating and message binding at this interface, while cryptographic microbenchmarks measure the online signing path and its scaling behavior.

Bullet Summary

• The paper addresses the security risks in autonomous LLM agents where prompt injection can bypass judgment and directly access signing authority.
• Introduces KITA, a key-isolated threshold signing framework that separates semantic content approval from signing authority, ensuring signing keys remain external to LLM processes.
• KITA utilizes threshold BLS signatures and distributed key generation, requiring a quorum of reviewer approvals to produce valid authorization signatures, thereby preventing unauthorized actions.
• The system design binds approvals to canonicalized actions and enforces strict protocols to prevent replay, substitution attacks, and unauthorized execution.
• A 2-of-3 prototype and multiple tests validate the quorum gating, message binding, and threshold unforgeability properties, demonstrating effective enforcement of end-to-end authorization.

Abstract

Core Motivation: Latency Asymmetry and Non-Separable Exposure Current autonomous agent safeguards rely on perimeter policies evaluated at the API dispatch boundary. This approach fails structurally: The Latency Gap: The interval between internal intent formation and external action serialisation is approximately 200 ms, whereas human operator orientation requires hundreds of seconds. Post-hoc human review is forensic rather than preventive. Local Predicate Failure: Multi-agent exposure is superadditive across sequences of individually permitted calls. We formally prove that no local predicate evaluated on single actions can bound an exposure functional that is not additively separable across those actions (Proposition 1). The Automated Metacognitive Protocol (AMP) AMP shifts runtime enforcement from the network boundary directly into the inference path: Non-Differentiable Shadow Taps: Residual-stream activations are tapped across selected layers with an enforced stopgrad boundary, introducing zero gradient pressure to optimize around the monitor (Assumption A1). Permission Manifold Scoring: Hidden states are projected via low-rank audit operators ($k = 32, d = 8192$) and evaluated against an approved operating envelope using Mahalanobis distance with Ledoit-Wolf shrinkage for covariance estimation. Dual-Path Temporal Filtering: Divergence is evaluated using an Exponentially Weighted Moving Average (EWMA) for acute spikes and a one-sided CUSUM statistic to catch sub-threshold, incremental drift attacks. Pre-Actuation State Dissolution: Upon threshold breach, the architecture applies a scheduled orthogonal subspace projection to recursively remove the unaligned direction across downstream layers until the intent magnitude reaches a zero-weight vacancy threshold (Proposition 4). Unlike boundary blocking, which preserves unaligned intent in context to drive re-planning and rebound dynamics, it structurally eliminates representational support before token serialisation. Intelligence-Invariant Hardware Floor: Actuation lines pass through a normally-open physical relay held closed by a signed, high-frequency enclave heartbeat. Power or heartbeat loss drops the rail into a safe state invariant to counterparty reasoning capability (Proposition 5). Computational and Latency Budget Analytical derivation across twelve tapped layers yields an audit cost of approximately 6.6 MFLOP per token. On dedicated audit silicon, this results in an analytical budget of under 20 µs end-to-end, maintaining an operational margin of $10^4$ within the 200 ms action boundary. Design Lineage and Empirical Scope Functional Phenomenology (Section 1.4): The four load-bearing priors—non-interfering observation (satipaṭṭhāna), triadic failure roots (lobha, dosa, moha), pre-manifestation state removal (samuccheda vs. vikkhambhana), and structural vacancy (anattā)—are adapted from the Pali Canon strictly as functional systems phenomenology and structural engineering priors. All properties stand independently on control-theoretic and linear-algebraic formulations. Validation Status: The paper presents an architectural specification with analytical guarantees. Section 8.5 explicitly outlines nine falsifiable empirical quantities, including audit-space separability, false-positive rates on benign traffic, and rebound coefficients reserved for future experimental benchmark validation

Bullet Summary

• The paper addresses the inherent latency and exposure gaps in current autonomous multi-agent system safeguards, which rely on perimeter policies insufficient for preventive control.
• It formally demonstrates that local predicates checked on single actions cannot capture risks arising from sequences of individually permitted actions due to non-additive exposure in multi-agent contexts (Proposition 1).
• Introduces the Automated Metacognitive Protocol (AMP), shifting runtime security enforcement inside the inference path rather than at boundary APIs, enabling sub-millisecond detection and mitigation.
• Implements non-differentiable shadow taps on residual stream activations with stop-gradient boundaries to monitor internal states without influencing model training (Assumption A1).
• Utilizes low-rank audit operators to project hidden states into a permission manifold, employing Mahalanobis distance with Ledoit-Wolf shrinkage to detect deviations from approved operating envelopes.