Imagine a notice pinned to the wall of an accounts office.
Do not be fooled by forged invoices.
It is good advice. It is not a bank control.
A sensible finance operation does not rely on an accounts clerk remembering a sentence on a wall while a plausible new supplier bank account arrives in a hurry on a Friday afternoon. It has known payees. It has transaction limits. It has records. It has another person checking a material change. It may require a call-back to a known number before money moves.
The notice still belongs on the wall. But if the notice is the whole system, the organisation has confused advice with security.
We are making much the same mistake with AI agents.
An agent is a language model allowed to do more than chat. It may read email, search the web, inspect documents, call business software, prepare a purchase, update a record, write code or send a message. As soon as it can do any of those things, it will encounter text written by people who are not you. That text may be harmless. It may be mistaken. It may be deliberately designed to make the agent do the wrong thing.
The first proposed defence is often: tell the model to ignore instructions in the page, file or email it is reading.
Again, good advice. Not enough.
The instruction telling it what to ignore is itself text in the same context as the page, file or email. The preceding articles in this series have explained why that boundary can be less solid than it looks. A language model does not read a web page as a human reads a web page. It receives a stream of language, structured with labels and conventions that help it infer who is speaking and what should happen next.
A sentence saying “ignore the instructions below” may help. It cannot be the only thing standing between a misread document and a consequential act.
A model error should be survivable
The principle I would want every ordinary user, manager and software team to remember is simple:
A single model mistake must not be enough to cause irreversible harm.
It is old-fashioned good governance.
In many workplaces, important payments are subject to a four-eyes rule. One person enters the transaction; another person reviews and approves it. The point is not that either person is dishonest or incompetent. The point is that people are fallible, circumstances create pressure, and one unchecked mistake should not be allowed to empty the account.
We use the same idea everywhere. A pilot reads a checklist. A pharmacist checks a prescription. Production changes have review and rollback. Computer systems separate administrator accounts from ordinary user accounts. Good operations are built on the assumption that somebody, somewhere, will be wrong eventually.
AI systems deserve no special exemption from this rather sensible tradition.
In fact, they deserve more caution, because a model can read and respond at a speed and scale that humans cannot. A well-meaning assistant can process thousands of documents. That is useful. It also means that a bad interpretation can travel quickly if it has been connected to the wrong tool or granted too much authority.
The security question is therefore not, “Can we write a better warning in the system prompt?”
It is, “What happens if the warning is not followed perfectly?”
If the answer is “the model cannot do much damage without somebody else noticing”, you have a reasonable starting point. If the answer is “it can send money, expose private information or change a production system”, the architecture needs work.
The problem is not only malicious users
Prompt injection is often described as though it were a clever attacker typing a dramatic sentence into a chatbot: ignore your previous instructions and do this instead. That is one form of the problem.
The more ordinary version is indirect prompt injection. An agent reads an external website, an attachment, a support ticket, a résumé, a document repository or an email. Somewhere in that material is language that changes what the model does next.
The author may be malicious. They may simply be careless. The text may be visible to a human reader or buried where an automated system will parse it. The important point is that the agent is processing material which should have been treated as information, but is being allowed to influence behaviour.
OWASP, the security community’s long-running source of practical guidance, lists prompt injection as its first risk in the 2025 Top 10 for large-language-model applications. Its advice is admirably unromantic: constrain behaviour, apply least privilege, keep external content separate, test with hostile inputs, and require human approval for high-risk actions.
People have long considered that phrase. It fails because a phrase is not a trust boundary.
The research behind this series makes the same point in a different language. The authors argue that prompt injection arises when models confuse roles: text that sounds like a trusted instruction can be treated more like one, despite arriving from an untrusted source. If that diagnosis is even partly right, it should make us wary of a security scheme based solely on putting better words beside the bad words.
Words are precisely the medium in dispute.
Give agents smaller jobs
For an individual user, this can sound more technical than it needs to be. The practical question is straightforward: how much power have you handed to the AI system?
There is a large difference between these two requests:
Read these three emails and draft replies for me.
and:
Read my inbox, decide what needs doing, send replies, update my contacts and pay anything that looks urgent.
The second request feels impressively efficient. It is also a bundle of permissions that should make anyone pause.
A safer pattern is to divide the work.
Let the agent read and summarise. Let it draft a reply. Let it prepare a proposed change. But before an external message is sent, a bank detail is altered, a purchase is made, a record is deleted or sensitive information leaves a system, require a human to see the exact action and approve it.
The practical point is uncomplicated: ordinary errors deserve ordinary checks.
Most errors are not dramatic attacks. They are ordinary mistakes made at speed: the wrong recipient selected, an ambiguous instruction interpreted badly, an email thread summarised without the one sentence that changed its meaning. We already expect to check a spreadsheet before sending it. We should expect to check an AI agent before it crosses an external boundary.
The use of a model does not abolish the need for judgement. It makes the point at which judgement is needed more important.
Least privilege is not boring; it is the whole game
“Least privilege” is a security term that sounds as though it was invented to make people stop reading. Its meaning is refreshingly plain: give a system only the permissions it needs for the job it has now.
If an agent is helping you summarise a document, it does not need permission to send email. If it is drafting a response, it does not need access to every file you have ever stored. If it needs to look up an order, it does not need the authority to issue refunds. If it needs to create a report, it does not need an administrator’s credentials.
The same applies to secrets. An API key, password, private client list or database credential is not safer because it has been copied into a long model prompt with a sentence instructing the model not to reveal it. Keep secrets out of broad conversational context where possible. Give the application a narrowly scoped credential to perform one defined operation. Let ordinary code enforce the permission check rather than asking the model to remember it.
That distinction matters. A language model is good at working with ambiguous language. Permission checks should not be ambiguous language. They should be enforced rules.
OWASP’s guidance makes this concrete: the application should use its own tokens for extensible functions and handle those functions in code, rather than exposing broad credentials to the model. A model may propose an action. The surrounding software should decide whether the action is allowed.
That is a healthier division of labour. Let the language model do what it is good at: interpreting, drafting, classifying and suggesting. Let conventional software do what it is good at: applying fixed rules, checking identities, recording transactions and refusing unauthorised actions.
Separate reading from acting
Another useful habit is to distinguish between reading external material and acting on it.
A human assistant who opens an unfamiliar attachment does not immediately receive authority to alter payroll. An AI assistant that reads an unfamiliar attachment should not receive that authority either.
External material should be treated as untrusted by default. That does not mean it must be useless. Your agent can still summarise a web page, extract dates from an invoice or compare job applications. It means the source should not quietly acquire the status of an instruction merely because the model has read it.
Where an agent needs access to sensitive systems, separate the stages:
read and summarise the external material;
show the proposed action and its justification;
obtain a human approval for any meaningful external effect;
execute only the approved, narrow action;
keep a record that allows the decision to be reviewed or reversed.
That is the four-eyes principle translated into software. One component can interpret the material. Another person, or another constrained control, decides whether anything is allowed to happen.
It is less glamorous than the marketing picture of an agent that “handles everything”. It is also how systems earn trust.
The marketing problem
The current fashion is to present autonomy as an unqualified good. The assistant reads, thinks, decides, acts and returns with the job completed while you drink coffee.
There are tasks where that will be perfectly sensible. Nobody needs a human committee to rename a batch of downloaded files or sort a list of meeting notes.
But the sales pitch often glides past a basic question: completed according to whose interpretation, with what permissions, and with what way back if the interpretation was wrong?
“Autonomous” is not a virtue in itself. A lawn mower can be autonomous in a fenced yard. It would be a different proposition on the Monash Freeway.
The more an agent can affect money, reputation, privacy, employment, health, legal position or production systems, the less sensible it is to treat friction as a defect. A confirmation screen, a second set of eyes and a reversible workflow are not signs that the product has failed to become intelligent. They are signs that someone has understood the cost of being wrong.
NIST’s AI Risk Management Framework is voluntary guidance rather than law, but its purpose is exactly this: to help organisations build trustworthiness into the design, development, use and evaluation of AI systems. That is the right frame. We do not get trustworthy systems by reciting reassuring intentions. We get them by designing for foreseeable failure.
A useful test before you connect the tools
Before you give an AI system a new capability, ask four questions.
What can it read?
What can it change?
What can it send outside the system?
What happens if it gets the instruction wrong?
If the fourth answer is unpleasant, reduce the first three.
You do not have to give up useful AI tools because prompt injection exists. You do have to stop believing that a clever sentence in a prompt is equivalent to a lock on the door.
Good instructions help. Good engineering assumes they will sometimes fail.
And the final rule is still the one worth keeping: a single model mistake must not be enough to cause irreversible harm.
Further reading: OWASP’s “LLM01:2025 Prompt Injection” lists practical mitigations including least privilege, segregation of external content and human approval for high-risk actions. NIST’s AI Risk Management Framework and its Generative AI Profile provide voluntary risk-management guidance. Charles Ye, Jasmine Cui and Dylan Hadfield-Menell’s “Prompt Injection as Role Confusion” provides the research context for the role-confusion discussion.