ROBATZ editorial · · 5 min read
Start with a decision, not another demo
A successful demonstration answers a narrow question: can this approach work on these examples? A production decision asks something harder: can the business use it repeatedly, with acceptable errors, operating costs and oversight? Before extending the pilot, write down who uses it, which task it changes and who owns the result. A support assistant that drafts a reply has a different risk profile from a system that issues a refund.
Choose one measurable workflow. Record the current handling time, error types and review effort. Define what would justify continuing, revising or stopping. These thresholds should come from your business process; an impressive model benchmark does not establish the value of your product.
Turn examples into an evaluation set
Collect representative tasks before choosing more infrastructure. Include ordinary requests, incomplete inputs, conflicting instructions, outdated documents and cases where the correct response is to ask a person. Remove or protect sensitive information. Keep some examples separate from the cases used to improve prompts so that the final check measures more than familiarity.
For each task, define an acceptable outcome and an unacceptable failure. Evaluate the final result and relevant actions, such as which records were changed. Repeat variable tasks and retain failures as regression examples. Anthropic's evaluation guidance supports treating agent behavior as something to measure across trials, rather than approving it after one convincing run.
Give the system limited, explicit authority
Make a permissions table: what can the application read, suggest, change and send? Tie those permissions to the signed-in user and the business action. A model instruction is not a replacement for an authorization check in application code. For consequential actions, show the proposed change to a person and require approval before execution.
Consider a hypothetical invoice assistant. Extracting fields, comparing them with an order and preparing a discrepancy report can be separate steps. Paying the invoice requires a different control. This separation makes the workflow easier to test and gives the operations team a clear place to intervene when information is missing or inconsistent.
Budget for the whole workflow
Model tokens are one expense. Retrieval, document processing, storage, tool calls, retries and human review can also affect the cost of completing a task. Measure the full workflow with realistic inputs, including large documents and failed attempts. Compare cost per completed task with the value of that task, rather than treating the cheapest model as the cheapest solution.
Set limits for response time, tool calls and spending. Decide what the user sees when a provider is unavailable or a limit is reached. A useful fallback might be ordinary search, a saved draft or a manual queue. Reliability includes a sensible experience when the AI path cannot finish.
Release in stages that you can reverse
Begin with a narrow audience and a reversible workflow. A shadow mode can compare suggestions with existing decisions without allowing the system to act. A subsequent assisted mode can let people review every proposal. Broader automation should follow evidence that the agreed quality and operational thresholds are met.
Name the person who can pause the feature. Version prompts, model configuration and retrieval settings alongside the application release. Keep enough operational records to investigate failures while limiting sensitive content. A rollback plan should say what happens to unfinished jobs and pending approvals, not just how to restore the previous code.
Ask for a production readiness review
A practical handoff includes the workflow definition, evaluation results, data permissions, cost assumptions, monitoring, support ownership and recovery procedure. It also identifies unanswered questions. Request a demonstration of failure handling as well as the happy path: an unavailable source, a denied action and a task the model cannot complete.
Keep the architecture proportionate. Anthropic recommends starting with simple patterns and adding agent complexity when it earns its place. Your next investment might be better source documents, clearer approvals or a conventional integration. The objective is an operating capability that helps people finish work, with evidence to guide the next release.
Apply this to your product
Explore our AI consulting service for scope, deliverables and the decisions to make before starting.