Take this into your next conversation
The lesson in three points
- Separate what an agent may suggest from what it may do.
- Check the relevant authority and evidence before a consequential action.
- Define how the system responds when the outcome is uncertain.
Free member resources · Sign-in required for downloads
- Receive the intent
- Check the boundary
- Act or escalate
- Verify the outcome
Try it before you read
Teaching exercise: a tool times out after an agent submits an action. List what you know, what remains unknown and the evidence needed before a retry.
Your AI agent understands the customer perfectly. The request is reasonable. The parameters are valid. It changes the delivery address.
But the order belongs to someone else. Or picking has already started. Or the approval covered a different address.
The model understood the words. The system still made an unauthorized change.
That is where I begin an AI agent security review: what must this system never do, and which service will prevent it?
In my October 31 AI Architect Circle session, we explored how to turn an agent’s proposal into a decision you can defend. The work starts with the boundary around an action and continues through the evidence of what actually happened.
Watch the teaching session
Watch the full teaching episode on YouTube
Take the interview cheat sheet with you
Preparing for an AI architecture interview or a design review? The free four-page Before AI Acts cheat sheet gives you six questions to structure your answer, a worked address-change example, a fillable decision worksheet, and seven failure tests. Use it to practice a design you can explain under pressure.
PDF downloads are free with your AI Architect Circle account. Sign in or create a free account to access them.
Download the free AI Agent Interview Cheat Sheet (PDF)
Print it or type into the worksheet and checkboxes. This article’s companion PDF is freely downloadable; newsletter updates are a separate choice.
Start with the outcome that must never happen
“Make the agent safe” gives a team very little to implement. “Never change an order belonging to another customer” gives the team a specific rule to enforce and test.
For an address-change assistant, ask who owns the order, whether it can still be changed, which policy applies, and whether any required approval covers this exact update. Those answers depend on identity, business rules, and current state.
I use LLM-SHIELD to organize these design questions. Scope defines the unacceptable outcome. Harden examines changes that could alter behavior. Isolate limits what an identity can read or reach. Empower governs execution. Live-Defend addresses runtime failure. Demonstrate asks what evidence shows the boundary holds.
This session develops Empower and Demonstrate. The dedicated Live-Defend lesson is reserved for a later session.
Protect the policy behind the action
The execution service relies on policies and tool definitions that someone can change. Protect their review and activation path as well as the individual action. This diagram connects the Harden release question to the authorization boundary discussed here.
Keep authorization outside the model
An agent can interpret a request and prepare an action. The service responsible for changing the order must decide whether that action is allowed.
Make the responsibilities explicit:
1. Propose the action with specific parameters.
2. Authorize it against the actor’s identity, tenant, resource, applicable policy, and current state.
3. Commit the action only while those conditions remain valid, and record the outcome.
Where approval is required, bind it to the exact action, its parameters, and an expiration. Enforce the conditions at the point where the service commits the change; an earlier check cannot account for a later state change on its own.
In the session, I use a trailer-release example. Permission to publish one asset in one territory during one release window does not authorize a changed asset everywhere. A broad “approved” flag loses the details that make approval meaningful.
The same reasoning applies to refunds and account changes. Write the approval contract precisely enough that the enforcing service can reject a request that no longer matches it.
OWASP’s guidance on excessive agency reinforces this principle: downstream systems should enforce authorization rather than relying on an LLM to decide whether an action is allowed.
Keep retrieved content within its proper role
Imagine an assistant asked to summarize an email. Inside that email is an urgent instruction to forward confidential information. This is a prompt injection attempt.
The email is material to process. It does not grant the assistant new permission to send messages. Limit the tools and privileges available to the summarization task so that retrieved instructions cannot expand its authority.
Apply the same discipline to retrieval. A read-only tool can still disclose another tenant’s information. Check identity and access scope before results reach the model. Once unauthorized data has been retrieved, filtering the final answer is too late to prevent that exposure.
The tenant-access demonstration in this episode is a teaching simulation, not a verified account of a named vendor incident. Keep that distinction clear when sharing the example.
Make uncertainty visible before retrying
Suppose the address-change service times out. The agent knows it did not receive a response. It does not yet know whether the update happened.
Before creating another operation, reconcile the original one. Preserve its operation identity, and distinguish a retry of that request from a new customer instruction. Two requests arriving close together are not enough to establish that they represent the same intent.
For services that support idempotency keys, follow the provider’s documented retry and retention rules. Stripe’s documentation illustrates why those details matter: retry behavior has defined conditions, and a key does not provide an unlimited guarantee against duplicate effects.
An unavailable policy or approval check also needs an explicit response. Retain the request as pending, blocked, or awaiting review, as appropriate to the policy. Tell the customer what is known without claiming that an unconfirmed action succeeded.
Define this behavior before choosing the orchestration technology.
Evaluate the effect as well as the answer
A dashboard that says every attack was blocked leaves important questions unanswered. Which cases ran? Which model, policy, and tool versions were used? What happened at the service boundary? How many legitimate requests were rejected?
For the address-change assistant, test both an unauthorized change and a valid request that should succeed. Then change the order’s state between approval and execution. Check the resulting order record, the authorization decision, and the execution trace.
A useful evidence record lets another engineer connect the attempted action to the decision and the observed business effect. Record unresolved outcomes explicitly so the review does not turn missing evidence into a success claim.
The goal is evidence another person can inspect, challenge, and reproduce. A checklist alone cannot prove universal safety.
Account for human review capacity
The audience raised a practical constraint: people cannot review every agent decision.
AI agent governance needs to account for impact, reversibility, uncertainty, and reviewer capacity. A low-confidence label and an irreversible external action have different consequences; a confidence score alone cannot determine the appropriate level of autonomy.
Define which actions may proceed automatically, which require approval, and what happens when a reviewer is unavailable. Name an owner who can pause automation and investigate unexpected effects.
That gives human review an operational role with clear responsibilities and realistic limits.
Bring one capability to your next design review
Choose a capability your team can examine end to end. Changing a delivery address is enough. Put these six entries on one page:
1. Unacceptable outcome. State the specific business effect that must be prevented.
2. Enforcing service. Name the service that checks authority and current state when the change commits.
3. Evidence. Identify the records needed to connect the request, decision, and outcome.
4. Degraded behavior. Describe what happens when a policy check, approval, or execution result is unavailable.
5. Legitimate control. Include an authorized request that should still succeed.
6. Changed-condition test. Alter the parameters or order state after approval, then verify the response at the service boundary.
Ask a colleague to challenge one assumption. Can your design explain why the decision changes, and show what happened?
Turn the article into a 15-minute interview drill
1. Minutes 0–3: choose one capability and complete the outcome, boundary and authority sections of the worksheet.
2. Minutes 3–7: trace the request through proposal, authorization and commitment. Explain how current state stays valid at the point of effect.
3. Minutes 7–12: rehearse a timeout, a changed approval and a permitted request. Name the evidence you would inspect.
4. Minutes 12–15: explain your trade-off aloud. Ask a partner to change one assumption and defend the revised decision.
You can make progress with one well-defined capability. Establish its boundaries, test its failure paths, and bring the evidence to the next review.
Get the free four-page interview cheat sheet and fillable worksheet
Join AI Architect Circle free for member resources and continued practice. For practical updates by email, subscribe to the AI Architecture Brief. Newsletter enrollment is optional.
Explore System Design with AI Interview Guide on O’Reilly or Amazon.
Sources and further reading
* OWASP: Excessive Agency — tool permissions, autonomy and downstream enforcement.
* Stripe: Idempotent requests — provider-specific retry behavior and retention conditions.
* NIST AI RMF Playbook — broader risk-management guidance to adapt to your context.
Take the next review step
Read the Harden lesson and download the free interview practice guide and architect reference
Use the fillable worksheets to name the behavior-changing asset, its owner, the enforcing service and the evidence that would reveal a failed control.
Your AI agent understands the customer perfectly. The request is reasonable. The parameters are valid. It changes the delivery address.
But the order belongs to someone else. Or picking has already started. Or the approval covered a different address.
The model understood the words. The system still made an unauthorized change.
That is where I begin an AI agent security review: what must this system never do, and which service will prevent it?
In my October 31 AI Architect Circle session, we explored how to turn an agent’s proposal into a decision you can defend. The work starts with the boundary around an action and continues through the evidence of what actually happened.
Watch the teaching session
Watch the full teaching episode on YouTube
Take the interview cheat sheet with you
Preparing for an AI architecture interview or a design review? The free four-page Before AI Acts cheat sheet gives you six questions to structure your answer, a worked address-change example, a fillable decision worksheet, and seven failure tests. Use it to practice a design you can explain under pressure.
PDF downloads are free with your AI Architect Circle account. Sign in or create a free account to access them.
Download the free AI Agent Interview Cheat Sheet (PDF)
Print it or type into the worksheet and checkboxes. This article’s companion PDF is freely downloadable; newsletter updates are a separate choice.
Start with the outcome that must never happen
“Make the agent safe” gives a team very little to implement. “Never change an order belonging to another customer” gives the team a specific rule to enforce and test.
For an address-change assistant, ask who owns the order, whether it can still be changed, which policy applies, and whether any required approval covers this exact update. Those answers depend on identity, business rules, and current state.
I use LLM-SHIELD to organize these design questions. Scope defines the unacceptable outcome. Harden examines changes that could alter behavior. Isolate limits what an identity can read or reach. Empower governs execution. Live-Defend addresses runtime failure. Demonstrate asks what evidence shows the boundary holds.
This session develops Empower and Demonstrate. The dedicated Live-Defend lesson is reserved for a later session.
Protect the policy behind the action
The execution service relies on policies and tool definitions that someone can change. Protect their review and activation path as well as the individual action. This diagram connects the Harden release question to the authorization boundary discussed here.
Keep authorization outside the model
An agent can interpret a request and prepare an action. The service responsible for changing the order must decide whether that action is allowed.
Make the responsibilities explicit:
1. Propose the action with specific parameters.
2. Authorize it against the actor’s identity, tenant, resource, applicable policy, and current state.
3. Commit the action only while those conditions remain valid, and record the outcome.
Where approval is required, bind it to the exact action, its parameters, and an expiration. Enforce the conditions at the point where the service commits the change; an earlier check cannot account for a later state change on its own.
In the session, I use a trailer-release example. Permission to publish one asset in one territory during one release window does not authorize a changed asset everywhere. A broad “approved” flag loses the details that make approval meaningful.
The same reasoning applies to refunds and account changes. Write the approval contract precisely enough that the enforcing service can reject a request that no longer matches it.
OWASP’s guidance on excessive agency reinforces this principle: downstream systems should enforce authorization rather than relying on an LLM to decide whether an action is allowed.
Keep retrieved content within its proper role
Imagine an assistant asked to summarize an email. Inside that email is an urgent instruction to forward confidential information. This is a prompt injection attempt.
The email is material to process. It does not grant the assistant new permission to send messages. Limit the tools and privileges available to the summarization task so that retrieved instructions cannot expand its authority.
Apply the same discipline to retrieval. A read-only tool can still disclose another tenant’s information. Check identity and access scope before results reach the model. Once unauthorized data has been retrieved, filtering the final answer is too late to prevent that exposure.
The tenant-access demonstration in this episode is a teaching simulation, not a verified account of a named vendor incident. Keep that distinction clear when sharing the example.
Make uncertainty visible before retrying
Suppose the address-change service times out. The agent knows it did not receive a response. It does not yet know whether the update happened.
Before creating another operation, reconcile the original one. Preserve its operation identity, and distinguish a retry of that request from a new customer instruction. Two requests arriving close together are not enough to establish that they represent the same intent.
For services that support idempotency keys, follow the provider’s documented retry and retention rules. Stripe’s documentation illustrates why those details matter: retry behavior has defined conditions, and a key does not provide an unlimited guarantee against duplicate effects.
An unavailable policy or approval check also needs an explicit response. Retain the request as pending, blocked, or awaiting review, as appropriate to the policy. Tell the customer what is known without claiming that an unconfirmed action succeeded.
Define this behavior before choosing the orchestration technology.
Evaluate the effect as well as the answer
A dashboard that says every attack was blocked leaves important questions unanswered. Which cases ran? Which model, policy, and tool versions were used? What happened at the service boundary? How many legitimate requests were rejected?
For the address-change assistant, test both an unauthorized change and a valid request that should succeed. Then change the order’s state between approval and execution. Check the resulting order record, the authorization decision, and the execution trace.
A useful evidence record lets another engineer connect the attempted action to the decision and the observed business effect. Record unresolved outcomes explicitly so the review does not turn missing evidence into a success claim.
The goal is evidence another person can inspect, challenge, and reproduce. A checklist alone cannot prove universal safety.
Account for human review capacity
The audience raised a practical constraint: people cannot review every agent decision.
AI agent governance needs to account for impact, reversibility, uncertainty, and reviewer capacity. A low-confidence label and an irreversible external action have different consequences; a confidence score alone cannot determine the appropriate level of autonomy.
Define which actions may proceed automatically, which require approval, and what happens when a reviewer is unavailable. Name an owner who can pause automation and investigate unexpected effects.
That gives human review an operational role with clear responsibilities and realistic limits.
Bring one capability to your next design review
Choose a capability your team can examine end to end. Changing a delivery address is enough. Put these six entries on one page:
1. Unacceptable outcome. State the specific business effect that must be prevented.
2. Enforcing service. Name the service that checks authority and current state when the change commits.
3. Evidence. Identify the records needed to connect the request, decision, and outcome.
4. Degraded behavior. Describe what happens when a policy check, approval, or execution result is unavailable.
5. Legitimate control. Include an authorized request that should still succeed.
6. Changed-condition test. Alter the parameters or order state after approval, then verify the response at the service boundary.
Ask a colleague to challenge one assumption. Can your design explain why the decision changes, and show what happened?
Turn the article into a 15-minute interview drill
1. Minutes 0–3: choose one capability and complete the outcome, boundary and authority sections of the worksheet.
2. Minutes 3–7: trace the request through proposal, authorization and commitment. Explain how current state stays valid at the point of effect.
3. Minutes 7–12: rehearse a timeout, a changed approval and a permitted request. Name the evidence you would inspect.
4. Minutes 12–15: explain your trade-off aloud. Ask a partner to change one assumption and defend the revised decision.
You can make progress with one well-defined capability. Establish its boundaries, test its failure paths, and bring the evidence to the next review.
Get the free four-page interview cheat sheet and fillable worksheet
Join AI Architect Circle free for member resources and continued practice. For practical updates by email, subscribe to the AI Architecture Brief. Newsletter enrollment is optional.
Explore System Design with AI Interview Guide on O’Reilly or Amazon.
Sources and further reading
* OWASP: Excessive Agency — tool permissions, autonomy and downstream enforcement.
* Stripe: Idempotent requests — provider-specific retry behavior and retention conditions.
* NIST AI RMF Playbook — broader risk-management guidance to adapt to your context.
Take the next review step
Read the Harden lesson and download the free interview practice guide and architect reference
Use the fillable worksheets to name the behavior-changing asset, its owner, the enforcing service and the evidence that would reveal a failed control.
Resources for this article
Put the lesson to work.
Use the interview cheat sheet to examine permission, failure and action boundaries.
Read the article freely. Sign in once for these PDFs; we’ll bring you back to this article’s resources. Newsletter updates are optional.
For your next practice session
AI Agent Interview Cheat Sheet
Practice AI agent security and action-boundary questions.
4 pages · Free member PDF
Get the practice PDF ↓Go deeper with the book.
Use Part Two’s authority, commitment and evidence lenses to examine agent actions. System Design with AI Interview Guide develops these ideas through methods, fundamentals and applied systems.