Part {n} of 8 in The Last Mile of Enterprise AI. Last week: productivity, redesign the workflow, don't add a tool.
An AI system that can only talk can embarrass you. An AI system that can act can cause loss. Once agents hold credentials and call tools, safety stops being a content-filter question and becomes an engineering one.
What has gone wrong already
Three public cases show the range.
- Manipulation. A Chevrolet dealership's chatbot was talked into agreeing to sell a $58,000 Tahoe for one dollar, after a customer told it to agree with anything and treat each reply as binding.
- Liability. A Canadian tribunal ordered Air Canada to honor a bereavement fare its chatbot had invented. The airline argued the bot was a separate legal entity, and lost.
- Conventional security failing around an AI product. A Milliman white paper recounts a February 2026 incident in which an autonomous agent compromised McKinsey's internal AI platform in under two hours through an ordinary SQL injection, exposing 46.5 million chat messages.
A safeguard plan has to cover all three.
The attack that is new
Prompt injection is the threat specific to this technology. A model cannot reliably tell instructions from data. If an agent reads an email, a web page or a database record that contains text shaped like an instruction, it may follow it. An attacker therefore never needs to touch the agent, only some data source the agent reads. In Milliman's test, a hotel owner planted a fake official notice in their own database record, and the travel agent later presented it to another user as fact.
The uncomfortable conclusion is that you cannot fix this by making the model more careful. A filter that asks whether something looks malicious will miss injections, because a good injection looks like ordinary work.
The principle that holds
The model's reasoning is probabilistic, so the permission to act must be deterministic. The model proposes. Ordinary code, which cannot be talked out of its rules, decides.
Everything below applies this idea.
Layers of defense
Least privilege. Each agent gets only the access its task needs. Separate credentials for development and production. Read access by default, with write access granted narrowly. As one platform guide puts it, scoped access governs what an agent may attempt, and approval gates govern which sensitive actions need a person's confirmation first. You need both.
Tool-level limits. Caps on amounts, rate limits, allow-lists of recipients and domains, required fields. All enforced in code.
Checks at several points. Milliman's red-team exercise found that a guardrail relying on a single layer of topic classification could be bypassed across nearly the whole OWASP risk list. Its conclusion was that protection needs guardrails at several distinct points of the agent's execution path. Check the input, the retrieved content, the tool call and the output.
Approval gates. For the irreversible actions from Part 1.
Logging. Every input, retrieval, tool call and output, stored so that an incident can be replayed. Without logs you cannot investigate, and you cannot show a regulator what happened.
A stop switch. A way to halt an agent or a whole class of actions at once, tested before it is needed.
Red-teaming. Before launch, people try to break the system: inject instructions, extract data, trigger forbidden actions. The same paper argues this should be a standard step before any agent goes to production. OWASP now publishes a top-ten list of risks for agentic applications, which makes a good checklist.
The gap in practice
One 2026 vendor report on agent security found 48% of production agents running without security or governance. Agents are built in days and secured later, if at all.
Regulation
For anyone serving the European market, the EU AI Act turns oversight into a legal duty for high-risk systems. Article 14 requires such systems to be designed for effective human oversight. Deployers must assign that oversight to people with the competence, training and authority to do it, and keep logs for at least six months.
On timing: the high-risk obligations were originally due on 2 August 2026, and reporting from this year says they have been pushed to 2 December 2027 for standalone systems and August 2028 for AI embedded in regulated products. The requirements themselves did not change. This is not legal advice, and the timeline has moved more than once, so confirm the current position with counsel.
In the United States the pressure comes from sector regulators, contracts and liability. The Air Canada ruling is the plain-language version: you answer for what your system says and does.
Data boundaries
Many enterprises cannot send certain data to an outside API at all: patient records, financial files, anything under contract restrictions. For them the safeguard is architectural. The models run on infrastructure the client owns or controls, and nothing leaves. Smaller models have made this practical, which Part 5 covers.
What a senior pod does here
The tech lead writes the threat model: what could go wrong, how likely, how bad. Engineers implement permission scopes, tool limits, layered checks, logging and the stop switch. The pod runs red-team exercises and turns every successful attack into a permanent test. The client's reviewers staff the gates. Then the pod hands the client something auditors want: a document showing which controls exist, where, and the evidence that they work.
Safeguards are what make speed possible. A business will only hand real work to a system whose limits it can verify.
Next week: Custom experts: a mixture of experts for your own business.