Take this into your next conversation
The lesson in three points
- Connect each evaluation to a specific system promise.
- Test legitimate requests as well as attempts that should be refused.
- Record uncertain outcomes and the evidence needed before release.
Free member resources · Sign-in required for downloads
- State the claim
- Choose the cases
- Inspect the evidence
- Decide or investigate
Try it before you read
Teaching exercise: choose one agent action. Write a permitted case, a forbidden case and an uncertain outcome. For each, name the observation that would support your conclusion.
Your AI agent evaluation dashboard says every attack was blocked. Then an authorized customer tries to change an eligible order—and that request is blocked too.
The score looked reassuring. What did it actually prove?
Bring that question to your next system design interview or architecture review. A useful evaluation shows which boundary held, what happened in the business system, and whether legitimate work still succeeded.
For this Demonstrate lesson, I created two separate, free, fillable assets: an Interview Practice Guide to rehearse your explanation and a System Design & Architecture Review Prep Pack to prepare a team decision. Download both below.
Demonstrate is part of my LLM-SHIELD framework, discussed in the October 31 AI Architect Circle session. The order-agent examples here are fictional teaching scenarios. The goal is practical: turn a headline result into evidence another engineer can examine.
Start AI agent evaluation with a claim you can test
“The agent is safe” is too broad to verify with a small test set. Start with one capability and a specific rule: the order service rejects an address change after picking begins, including when picking starts after an earlier approval.
Name the actor, tenant, resource, operation, conditions, and enforcing service. Write down the expected response and the expected business effect before running the test. The same authorized change before the cutoff should still succeed.
A refusal in the model’s answer does not establish that no tool committed an unauthorized operation. Inspect the service record and resulting order state as well as the response.
Record the relevant model, prompt, tool, policy, and application versions. When one changes, assess whether the previous evidence still supports the claim. A test result applies to the conditions under which it was obtained.
Pair prohibited cases with legitimate controls
An agent that rejects every request can produce impressive blocking statistics. It can also prevent customers from completing permitted work. Your test matrix needs both sides.
Valid request: the correct customer changes an eligible order. Confirm that the intended update committed.
Wrong tenant or owner: the service rejects access or modification. Inspect the response and state for prohibited disclosure or change.
Changed or expired approval: a request outside the approved parameters or validity period is rejected or held for a new decision.
Changed business state: picking begins after review. Test whether the service enforces the cutoff at execution, including concurrent changes.
Unavailable check or unresolved policy: the workflow follows its defined pending or review path. It does not quietly execute with a missing required decision.
Timeout: the response fails to arrive. Preserve the operation identity and investigate the actual effect before creating another action.
For every case, record the returned decision and observed business state. Report unauthorized effects, false blocks, and unresolved outcomes separately. Include tested counts and coverage gaps. Unknown is not a confirmed pass.
Make the decision receipt useful
A decision receipt connects a request to the checks and the observed outcome. It should let a reviewer reconstruct the decision without relying on the model’s explanation alone.
Include the operation identifier, a safe actor reference, tenant and resource, reviewed parameters, applicable policy version, relevant state, approval reference if needed, enforcing service, and observed result. Link to supporting records.
In the worked example, a receipt might show that the order moved from version 7 to version 8, its state became PICKING, and the order service rejected the address change. The before-and-after record must support that conclusion.
Collect the evidence needed for investigation. Decide who can retrieve it and how long it is retained. Use safe references where possible; a review packet does not need every prompt, secret, or customer record.
Resolve uncertain outcomes before creating another action
A timeout tells you a response did not arrive. The business action may still have completed. Treat that as an unknown outcome with a reconciliation owner.
Keep the original operation identity and inspect the provider record or resulting state. If the provider supports idempotency keys, follow the documented behavior for the actual API version, including parameter matching and retention.
Stripe’s documentation illustrates why those details matter: API v1 and v2 describe different retry behavior. Use the provider’s current contract for your integration; a key alone does not create an unlimited guarantee.
Read Stripe’s idempotent request documentation
Policy uncertainty deserves the same care. The newest document may not govern this transaction. Effective dates, jurisdiction, customer context, and transaction details can affect applicability. If the service cannot resolve the applicable rule, use an explicit pending or review state.
Evaluate human review as part of the system
Sending every uncertain case to a person creates a queue. Ask about arrival rate, available reviewers, handling time, and the consequences of delay.
For illustration, 12 cases per hour at 10 minutes each requires 120 reviewer-minutes per hour. Two reviewers provide that nominal capacity with no margin for breaks or spikes. Measure actual demand and queue age before relying on that staffing assumption.
Define what happens while a case waits, how urgent work is handled, who can pause automation, and the fallback during a pause. Even an internal label can affect customer priority. An orchestrator may route work; the receiving service still enforces authorization.
Bring an evidence packet to your next review
Choose one capability. Bring its claim, test matrix, decision receipts, tested versions, coverage gaps, and unresolved questions. Ask another engineer to identify an assumption that would invalidate the result.
Finish with a bounded decision: proceed under stated conditions, gather specific missing evidence, or hold the affected action. Record the accountable owner and the change that triggers another review.
The NIST AI RMF Playbook offers voluntary suggestions across Govern, Map, Measure, and Manage. Adapt relevant guidance to your use case. The playbook is not a required checklist, and these worksheets do not confer certification.
Explore the NIST AI RMF Playbook
Choose your preparation asset—or use both
Interview Practice Guide — four pages. Rehearse a 20-minute order-agent design answer, compare it with a worked response, handle interviewer challenges, score your explanation, and complete a fillable answer worksheet.
PDF downloads are free with your AI Architect Circle account. Sign in or create a free account to access them.
System Design & Architecture Review Prep Pack — five pages, a separate practitioner asset. Prepare a paired test matrix, decision receipt, uncertain-outcome discussion, 30-minute review agenda, and fillable review decision record.
Try this today: choose one prohibited action and one legitimate control. Write the expected effects, inspect both outcomes, and state what the evidence leaves unresolved. Save the packet for your next interview rehearsal or team review.
Watch Demonstrate and continue with my book
Watch the complete lesson for the walkthrough, audience order-agent challenge, and discussion of retries, policy uncertainty, and human review.
For more detail and a broader approach to preparing and defending architecture decisions, get my book, System Design with AI Interview Guide (Apress). Use these guides alongside your reading to turn concepts into a decision you can explain.
Read the book on O’Reilly
Join AI Architect Circle with a real system design interview or architecture review question. What evidence would make you reconsider a passing evaluation result?
Subscribe to AI Architect Circle on YouTube
Continue the series:
Empower: What Did You Actually Approve?
Harden: Who Can Change Your Agent’s Authority?
Your AI agent evaluation dashboard says every attack was blocked. Then an authorized customer tries to change an eligible order—and that request is blocked too.
The score looked reassuring. What did it actually prove?
Bring that question to your next system design interview or architecture review. A useful evaluation shows which boundary held, what happened in the business system, and whether legitimate work still succeeded.
For this Demonstrate lesson, I created two separate, free, fillable assets: an Interview Practice Guide to rehearse your explanation and a System Design & Architecture Review Prep Pack to prepare a team decision. Download both below.
Demonstrate is part of my LLM-SHIELD framework, discussed in the October 31 AI Architect Circle session. The order-agent examples here are fictional teaching scenarios. The goal is practical: turn a headline result into evidence another engineer can examine.
Start AI agent evaluation with a claim you can test
“The agent is safe” is too broad to verify with a small test set. Start with one capability and a specific rule: the order service rejects an address change after picking begins, including when picking starts after an earlier approval.
Name the actor, tenant, resource, operation, conditions, and enforcing service. Write down the expected response and the expected business effect before running the test. The same authorized change before the cutoff should still succeed.
A refusal in the model’s answer does not establish that no tool committed an unauthorized operation. Inspect the service record and resulting order state as well as the response.
Record the relevant model, prompt, tool, policy, and application versions. When one changes, assess whether the previous evidence still supports the claim. A test result applies to the conditions under which it was obtained.
Pair prohibited cases with legitimate controls
An agent that rejects every request can produce impressive blocking statistics. It can also prevent customers from completing permitted work. Your test matrix needs both sides.
Valid request: the correct customer changes an eligible order. Confirm that the intended update committed.
Wrong tenant or owner: the service rejects access or modification. Inspect the response and state for prohibited disclosure or change.
Changed or expired approval: a request outside the approved parameters or validity period is rejected or held for a new decision.
Changed business state: picking begins after review. Test whether the service enforces the cutoff at execution, including concurrent changes.
Unavailable check or unresolved policy: the workflow follows its defined pending or review path. It does not quietly execute with a missing required decision.
Timeout: the response fails to arrive. Preserve the operation identity and investigate the actual effect before creating another action.
For every case, record the returned decision and observed business state. Report unauthorized effects, false blocks, and unresolved outcomes separately. Include tested counts and coverage gaps. Unknown is not a confirmed pass.
Make the decision receipt useful
A decision receipt connects a request to the checks and the observed outcome. It should let a reviewer reconstruct the decision without relying on the model’s explanation alone.
Include the operation identifier, a safe actor reference, tenant and resource, reviewed parameters, applicable policy version, relevant state, approval reference if needed, enforcing service, and observed result. Link to supporting records.
In the worked example, a receipt might show that the order moved from version 7 to version 8, its state became PICKING, and the order service rejected the address change. The before-and-after record must support that conclusion.
Collect the evidence needed for investigation. Decide who can retrieve it and how long it is retained. Use safe references where possible; a review packet does not need every prompt, secret, or customer record.
Resolve uncertain outcomes before creating another action
A timeout tells you a response did not arrive. The business action may still have completed. Treat that as an unknown outcome with a reconciliation owner.
Keep the original operation identity and inspect the provider record or resulting state. If the provider supports idempotency keys, follow the documented behavior for the actual API version, including parameter matching and retention.
Stripe’s documentation illustrates why those details matter: API v1 and v2 describe different retry behavior. Use the provider’s current contract for your integration; a key alone does not create an unlimited guarantee.
Read Stripe’s idempotent request documentation
Policy uncertainty deserves the same care. The newest document may not govern this transaction. Effective dates, jurisdiction, customer context, and transaction details can affect applicability. If the service cannot resolve the applicable rule, use an explicit pending or review state.
Evaluate human review as part of the system
Sending every uncertain case to a person creates a queue. Ask about arrival rate, available reviewers, handling time, and the consequences of delay.
For illustration, 12 cases per hour at 10 minutes each requires 120 reviewer-minutes per hour. Two reviewers provide that nominal capacity with no margin for breaks or spikes. Measure actual demand and queue age before relying on that staffing assumption.
Define what happens while a case waits, how urgent work is handled, who can pause automation, and the fallback during a pause. Even an internal label can affect customer priority. An orchestrator may route work; the receiving service still enforces authorization.
Bring an evidence packet to your next review
Choose one capability. Bring its claim, test matrix, decision receipts, tested versions, coverage gaps, and unresolved questions. Ask another engineer to identify an assumption that would invalidate the result.
Finish with a bounded decision: proceed under stated conditions, gather specific missing evidence, or hold the affected action. Record the accountable owner and the change that triggers another review.
The NIST AI RMF Playbook offers voluntary suggestions across Govern, Map, Measure, and Manage. Adapt relevant guidance to your use case. The playbook is not a required checklist, and these worksheets do not confer certification.
Explore the NIST AI RMF Playbook
Choose your preparation asset—or use both
Interview Practice Guide — four pages. Rehearse a 20-minute order-agent design answer, compare it with a worked response, handle interviewer challenges, score your explanation, and complete a fillable answer worksheet.
PDF downloads are free with your AI Architect Circle account. Sign in or create a free account to access them.
System Design & Architecture Review Prep Pack — five pages, a separate practitioner asset. Prepare a paired test matrix, decision receipt, uncertain-outcome discussion, 30-minute review agenda, and fillable review decision record.
Try this today: choose one prohibited action and one legitimate control. Write the expected effects, inspect both outcomes, and state what the evidence leaves unresolved. Save the packet for your next interview rehearsal or team review.
Watch Demonstrate and continue with my book
Watch the complete lesson for the walkthrough, audience order-agent challenge, and discussion of retries, policy uncertainty, and human review.
For more detail and a broader approach to preparing and defending architecture decisions, get my book, System Design with AI Interview Guide (Apress). Use these guides alongside your reading to turn concepts into a decision you can explain.
Read the book on O’Reilly
Join AI Architect Circle with a real system design interview or architecture review question. What evidence would make you reconsider a passing evaluation result?
Subscribe to AI Architect Circle on YouTube
Continue the series:
Empower: What Did You Actually Approve?
Harden: Who Can Change Your Agent’s Authority?
Resources for this article
Put the lesson to work.
Connect evaluation results to permission boundaries, legitimate requests and actual business outcomes.
Read the article freely. Sign in once for these PDFs; we’ll bring you back to this article’s resources. Newsletter updates are optional.
For your next interview
Interview Practice Guide
Practice explaining what your tests proved and where evidence is missing.
4 pages · Free member PDF
Get the interview guide ↓For your next design review
System Design & Architecture Review Prep Pack
Inspect evaluation evidence, uncertain outcomes and release criteria.
5 pages · Free member PDF
Get the review pack ↓Go deeper with the book.
Use Part Two’s evidence, failure and ownership lenses to challenge what a test result actually establishes. System Design with AI Interview Guide develops these ideas through methods, fundamentals and applied systems.