Architecture challenge · Web edition · October 30, 2026
This is a hypothetical design exercise, not a report of a customer incident. An AI support assistant proposes a refund. A policy service checks eligibility, and the required approval is recorded. A worker calls the payment provider. The request times out before the worker receives a response.
The questionA second worker picks up the operation. Later, two copies of a success event arrive from the provider. The application database also becomes unavailable briefly. How do you resolve the operation without issuing another refund for the same business intent?
Assume the provider supports idempotency keys for this endpoint and offers a way to reconcile its recorded operations. Verify its documented retention and retry behavior before choosing a production policy. Distinct, separately approved partial refunds are allowed; duplicate execution of one approved refund is not.
Before reading the answer, sketch your operation record, state transitions, retry owner, and escalation condition. Decide what the customer sees while the result is unknown.
A worked answer1. Persist intent before calling. Create a durable refund-operation record with an opaque operation ID, approval reference, payment reference, amount, currency, and stable provider idempotency key. Use an atomic claim or lease to coordinate workers. If the record cannot be committed, do not call the provider.
2. Keep uncertainty explicit. Move the operation from approved to submitting. After a timeout, record outcome unknown. Show the customer that the request is being checked, with a reference they can use. Neither declare success nor ask them to submit another refund.
3. Retry the same intent. Let a single recovery component control attempts. Preserve the key and parameters when the provider contract permits a retry. Apply a bounded retry budget and backoff with jitter. A new worker must load the existing operation, not generate a new key.
4. Reconcile before starting over. Match provider records to the operation using supported identifiers. If a refund exists, attach its provider reference and follow its actual status. If the outcome remains unclear or the safe retry window has passed, escalate for review. Do not equate a missing local result with permission to submit again.
5. Make completion repeatable. Authenticate provider notifications. Deduplicate events and update the operation transactionally so repeated success events have no additional business effect. Preserve a pending event for later processing when the database is unavailable. Reconciliation must also recover cases where no usable notification arrives.
The trade-off to defendThis design adds durable state, recovery work, and some customer waiting time. It makes uncertainty visible and gives the team a controlled way to resolve it. It does not claim a universal exactly-once guarantee across your database and the payment provider.
AWS explains why a request identifier must represent caller intent. Stripe documents version-specific idempotency behavior, including API v1 key retention and saved responses. Those details determine which retries are safe.
Check your answerCan you explain what survives a worker crash, what prevents concurrent duplicate work, and what happens after the provider's retention window? Can you distinguish a new partial refund from a retry? If any answer depends on the model remembering what it did, revise the design.
Capture your decision in the Design Decision worksheet. Read Brief 001 for the permission, evidence, and recovery framework, and bring your alternative to a live session.
Architecture challenge · Web edition · October 30, 2026
This is a hypothetical design exercise, not a report of a customer incident. An AI support assistant proposes a refund. A policy service checks eligibility, and the required approval is recorded. A worker calls the payment provider. The request times out before the worker receives a response.
The questionA second worker picks up the operation. Later, two copies of a success event arrive from the provider. The application database also becomes unavailable briefly. How do you resolve the operation without issuing another refund for the same business intent?
Assume the provider supports idempotency keys for this endpoint and offers a way to reconcile its recorded operations. Verify its documented retention and retry behavior before choosing a production policy. Distinct, separately approved partial refunds are allowed; duplicate execution of one approved refund is not.
Before reading the answer, sketch your operation record, state transitions, retry owner, and escalation condition. Decide what the customer sees while the result is unknown.
A worked answer1. Persist intent before calling. Create a durable refund-operation record with an opaque operation ID, approval reference, payment reference, amount, currency, and stable provider idempotency key. Use an atomic claim or lease to coordinate workers. If the record cannot be committed, do not call the provider.
2. Keep uncertainty explicit. Move the operation from approved to submitting. After a timeout, record outcome unknown. Show the customer that the request is being checked, with a reference they can use. Neither declare success nor ask them to submit another refund.
3. Retry the same intent. Let a single recovery component control attempts. Preserve the key and parameters when the provider contract permits a retry. Apply a bounded retry budget and backoff with jitter. A new worker must load the existing operation, not generate a new key.
4. Reconcile before starting over. Match provider records to the operation using supported identifiers. If a refund exists, attach its provider reference and follow its actual status. If the outcome remains unclear or the safe retry window has passed, escalate for review. Do not equate a missing local result with permission to submit again.
5. Make completion repeatable. Authenticate provider notifications. Deduplicate events and update the operation transactionally so repeated success events have no additional business effect. Preserve a pending event for later processing when the database is unavailable. Reconciliation must also recover cases where no usable notification arrives.
The trade-off to defendThis design adds durable state, recovery work, and some customer waiting time. It makes uncertainty visible and gives the team a controlled way to resolve it. It does not claim a universal exactly-once guarantee across your database and the payment provider.
AWS explains why a request identifier must represent caller intent. Stripe documents version-specific idempotency behavior, including API v1 key retention and saved responses. Those details determine which retries are safe.
Check your answerCan you explain what survives a worker crash, what prevents concurrent duplicate work, and what happens after the provider's retention window? Can you distinguish a new partial refund from a retry? If any answer depends on the model remembering what it did, revise the design.
Capture your decision in the Design Decision worksheet. Read Brief 001 for the permission, evidence, and recovery framework, and bring your alternative to a live session.