The Australian Signals Directorate (ASD) has released guidance for enterprises, highlighting a significant security flaw in agentic artificial intelligence (AI) that cannot be rectified within the model itself. Instead, the agency recommends implementing controls in the software layer surrounding the models to mitigate this vulnerability.
Deciphering the Agentic AI Harness
According to the ASD, an agentic AI harness refers to the layer above the model, encompassing all components of an agentic system except the large language model (LLM). This includes elements such as connectors, tool registries, memory stores, and permission systems. The agency emphasizes that the harness is the component that organizations can actually control, and it will outlast multiple generations of models.
The guidance focuses on prompt injection, a vulnerability that arises when agents interpret content as instructions. Since language models process instructions and information in the same context window, they are unable to distinguish between the two reliably. The ASD notes that “no fully reliable technical mitigation currently exists” for this vulnerability.
Historical Precedents and Inherent Limitations
Telecommunications engineers may recognize the issue, as it bears similarities to the conditions that enabled phone phreaking. In this scenario, signalling tones and conversations shared the same channel, allowing individuals to instruct the switch by reproducing the tones. A notable example is John Draper, who used a toy whistle from Cap’n Crunch cereal boxes to generate the tone. This problem was resolved by separating signalling from conversations, an approach that is not feasible within a language model.
Read Also: Optus Mulls Smart Glasses Restrictions Amid Privacy Fears
In December last year, the United Kingdom’s National Cyber Security Centre (NCSC) reached a similar conclusion, which is referenced by the ASD. The NCSC suggested that prompt injection may never be fully mitigated, much like SQL injection was eventually addressed. The agency framed an AI model as an inherently confusable entity rather than a system with a fixable input handling bug. The NCSC also advised that if a system’s security cannot tolerate residual risk, the use case may not be suitable for a language model.
Given the inherent nature of the weakness in how models process context, mitigations must be applied in the harness. This involves restricting an agent’s access and actions, implementing least privilege access, requiring human approval for high-impact actions, verifying outputs before use, and logging prompts, tool invocations, and configuration changes.
Operational Safeguards and Board Oversight
Multi-agent systems should be treated as a single entity, as a compromise in one component can spread through shared context and trust relationships. The ASD emphasizes that a model’s safety controls are not a substitute for controls enforced by the harness. Additionally, the agency recommends deleting stale agent context rather than summarizing it, as this can introduce new errors. Organizations should maintain a persistent rules file that the harness reads at the start of each session.
The ASD guidance provides seven questions for board directors, including one that asks about the worst possible outcome if the harness is compromised, misconfigured, or manipulated, and which controls would mitigate it. Answering this question is challenging, particularly when the harness is part of a commercial product rather than an in-house development. The ASD acknowledges this challenge without providing guidance on how customers can inspect a vendor’s implementation.
