The demo that was almost perfect
Brindlepath’s Horizon 3 team, introduced in Chapter 7, had a bold idea: an AI agent that would read a high-risk patient’s discharge summary, medication list and recent notes, then draft a post-discharge plan for the navigator, covering who to call, what to ask, which follow-up appointments were needed and which warning signs to watch for. If it worked, a navigator could prepare for a call in two minutes instead of fifteen.
The first demo, to the leadership team, was stunning. The agent produced clear, well-organized plans in seconds. The CEO wanted to announce it at the industry’s biggest conference, six weeks away.
Maya asked for one thing first: an evaluation. The team assembled 300 real, anonymized discharge summaries and asked two experienced nurses to review the agent’s plans against the source documents. The plans were good, most of the time. But roughly one plan in thirty contained something that was not in the record: a medication the patient was not taking, or a follow-up appointment that had never been ordered. Every one of those errors was written as confidently as everything else.
In a demo, one error in thirty is invisible. In a product used for thousands of patients a month, it is dozens of dangerous mistakes every week.
The team did not abandon the idea. They changed it. The agent now cited the source line for every statement, flagged anything it could not find in the record, and presented its plan as a draft the navigator reviewed and confirmed, never as an instruction. They built an evaluation set that ran before every change, and monitored errors in production. They launched four months later, with one customer, and expanded as the evidence came in.
This chapter is about building AI products that survive contact with reality, and about leading teams in which AI agents do real work.
Where organizations are
AI adoption has moved fast. In McKinsey’s 2025 global survey, 88 percent of respondents said their organizations used AI in at least one business function, and 62 percent said they were at least experimenting with AI agents. Far fewer had scaled them: 23 percent reported scaling an agentic AI system somewhere in their enterprise, and in no single business function did more than 10 percent report scaled use of agents.1
The gap between experimenting and scaling is where product leadership matters most. The technology to build an impressive demo is widely available. The discipline to build something reliable, safe, affordable and genuinely useful is not.
When AI is the right tool
AI is not the answer to every problem, and using it where it does not fit creates cost and risk for no benefit. Before building an AI feature, ask what kind of problem it is.
AI tends to fit well when:
- the work involves understanding or producing language, images or other unstructured content;
- inputs vary widely and cannot be handled by simple rules;
- a good draft saves significant time, even if a person reviews it;
- some errors are acceptable, or can be caught by review;
- there is enough data, or enough examples, to evaluate quality.
AI tends to fit poorly when:
- the answer must be exactly right every time and can be computed with rules;
- the stakes of an undetected error are severe and there is no practical review step;
- the task is simple enough that ordinary software does it cheaply and reliably;
- you cannot measure whether the output is good.
Brindlepath’s discharge-plan agent was a good fit for drafting, because summaries were long and varied and a draft saved real time, and a poor fit for acting alone, because an undetected error could harm a patient. The product design followed from that distinction.
How much should AI do on its own?
The most important design decision for an AI feature is its level of autonomy: how much it does without a person’s involvement. A useful way to think about it is as a ladder.
- Assist. AI answers questions or retrieves information when a person asks. The person decides what to do with it.
- Suggest. AI proposes a draft or recommendation; a person reviews, edits and accepts or rejects it.
- Act with approval. AI prepares an action, such as sending a message or updating a record, and a person approves it before it happens.
- Act within limits. AI acts on its own within defined boundaries, with monitoring and the ability for people to review and reverse.
- Act autonomously. AI acts on its own across a broad scope. Rarely appropriate in high-stakes domains today.
Match the step to the consequences of being wrong. Drafting a patient-education handout might reasonably sit at suggest. Updating the ownership of a patient in the queue might sit at act with approval. Changing a medication should not sit anywhere on this ladder without a clinician’s decision. Start lower than you think you need to, measure, and climb only when the evidence supports it.
Designing AI experiences people can trust
AI features behave differently from traditional software. They are probabilistic: the same input can produce different outputs, and the output is sometimes wrong in ways that look right. Good AI design makes those properties visible and manageable.
Show the sources. Let people see where an answer came from. Brindlepath’s agent cites the line in the discharge summary behind every statement, so a navigator can verify it with one click.
Signal uncertainty. Flag what the AI could not find or is unsure about, instead of letting it guess confidently.
Make correction easy. Editing a draft should be faster than writing from scratch. If it is not, people will stop using the feature.
Keep people in control. Make it clear what the AI has done and what it will do next, and let people stop, undo or override it.
Label AI output. People should always know when content was produced by AI.
Capture feedback. Every acceptance, edit and rejection is a signal about quality. Collect it, with appropriate privacy protections, and use it to improve.
Design for the reviewer’s time. As Chapter 12 noted, AI that saves the author’s time but costs the reviewer more is not a gain. Measure total time, not just generation time.
Evaluation: the discipline that separates demos from products
Traditional software is tested with expected outputs: given this input, the result must be exactly that. AI features need evaluation: systematic measurement of output quality across many realistic examples, against criteria people agree on.
A practical evaluation program has five parts.
1. An evaluation set. A collection of realistic inputs, including hard and unusual cases, ideally drawn from real data with sensitive details removed. Brindlepath’s started with 300 discharge summaries and grew to over 1,000, deliberately including the messiest records the team could find.
2. Clear criteria. What does good look like? For the discharge plan: every statement supported by the record; no missing high-risk medications; correct follow-up appointments; plain language. Criteria should be specific enough that two reviewers agree.
3. Scoring. Expert human review is the gold standard, especially in clinical settings. Automated checks, and AI models used as graders under careful supervision, can scale evaluation, but should be calibrated regularly against human judgment.
4. Thresholds. Decide in advance what quality is good enough to launch, and what change in quality blocks a release. For Brindlepath, any unsupported clinical statement in the evaluation set blocked a release until it was understood.
5. Production monitoring. Track quality in real use: acceptance and edit rates, user-reported errors, sampled expert review, cost and speed. Real-world inputs drift over time, and model updates can change behavior. Feed problems you find in production back into the evaluation set.
Evaluation is not a one-time gate. It is the core quality process for any AI product, and it deserves the same investment as testing does for traditional software. A team that cannot measure its AI feature’s quality cannot improve it, and should not ship it.
Grounding AI in your data
General-purpose models know a great deal about the world and nothing about your customer. Most useful AI features combine a model with the right context: the customer’s records, documents, policies and history. A common approach, often called retrieval-augmented generation, finds the relevant information and provides it to the model with each request, instructing it to answer from that information.
Grounding has several benefits: answers reflect the customer’s actual situation, sources can be cited, and errors are easier to detect. It also means your data quality, integrations and permissions become part of your AI quality. An agent that can see the wrong patient’s record, or none of the right ones, will produce the wrong answer however good the model is. The integration work in Chapters 15 and 16 is AI work too.
Cost, speed and model choice
Every AI request has a cost and a delay. At small scale, they are easy to ignore. At scale, they shape the product and its margins, as Chapter 6 warned.
- Use the smallest model that meets the quality bar. Larger models are often more capable and more expensive. Many tasks are handled well by smaller, faster, cheaper models.
- Don’t call a model when you don’t need to. Cache results, avoid regenerating unchanged content, and use ordinary code for anything deterministic.
- Design for latency. Show progress, stream results, or run work in the background so people are not left waiting.
- Keep the option to switch. Model providers change prices, capabilities and terms. An architecture that can swap models, with an evaluation set to confirm quality after a switch, protects you, as Chapter 15 recommended.
Safety, security and responsibility
AI features introduce risks that product leaders must own, building on Chapter 11.
Harmful errors. In domains like healthcare, wrong outputs can hurt people. Keep humans in the loop for consequential decisions, evaluate rigorously, and monitor in production.
Bias and fairness. Models can perform worse for some groups of people than others. Evaluate quality across the groups your product serves, including by age, language and other relevant characteristics, and address gaps before launch.
Manipulation and misuse. AI systems can be tricked through their inputs, for example by instructions hidden in a document an agent reads. Limit what agents can access and do, treat external content as untrusted, and require approval for consequential actions.
Privacy. Know exactly what data reaches each model, where it is processed, whether it is retained, and whether it is used for training. Get the agreements in place before data flows.
Regulation. Requirements are evolving: medical device rules for clinical AI, the EU AI Act’s obligations for high-risk systems, state laws and sector guidance. The NIST AI Risk Management Framework offers a practical structure for governing, mapping, measuring and managing AI risks.2 Involve legal and compliance partners early, as Chapter 11 recommended.
Leading teams that include AI agents
AI agents increasingly do real work inside product organizations: drafting research syntheses, triaging support tickets, monitoring dashboards, writing and reviewing code, preparing release notes. Treating them well is now part of leadership.
A practical way to think about it: manage an agent the way you would manage a new team member with specific, limited responsibilities.
- Give it a job description. What exactly does it do, for whom, with what inputs, and what does good output look like?
- Give it an owner. A named person is accountable for each agent’s instructions, permissions, quality and failures, as Chapter 8 recommended.
- Limit its permissions. Grant access only to the data and actions its job requires. Review those permissions regularly.
- Review its work. Sample its output, track its quality, and correct it. Agents do not improve by themselves; their owners improve them.
- Make it visible. Everyone should know which work is done by agents, and how to raise a concern about it.
- Keep humans accountable for decisions. Agents can gather, draft, recommend and, within limits, act. Accountability for outcomes, especially those affecting customers, money, safety or people, stays with people.
The product leader’s job is to design the system in which people and agents work together well: deciding which work suits agents, where human judgment must stay, and how quality is assured across both. That is the “orchestrator” role from Chapter 1, in practice.
From idea to AI product
Pulling it together, a practical path for building an AI product or feature looks like this:
- Start with the problem, using the discovery methods in Chapters 2 and 3, not with the technology.
- Check the fit: is AI the right tool, and what is the cost of an error?
- Choose the autonomy level based on the consequences of being wrong.
- Build the evaluation set early, before the feature, with experts defining good.
- Prototype quickly and measure against the evaluation set, not against a demo.
- Design the experience for trust: sources, uncertainty, easy correction, control.
- Pilot with a small group, monitor closely, and expand on evidence.
- Monitor quality, cost and safety in production, and feed what you learn back into evaluation.
Brindlepath’s discharge-plan drafts followed that path. In the story, navigators accepted about seven in ten plans with minor or no edits, preparation time per high-risk patient fell substantially, and the share of high-risk patients contacted within 24 hours rose at the pilot customer. Those results are part of the fictional example; your numbers will depend on your product and data. The method is what transfers.
Brindlepath in practice: the evaluation set
The evaluation set for the discharge-plan agent became one of the most valuable assets the company owned.
It began with 300 discharge summaries from two pilot customers, used with permission under the customers’ agreements, with names, dates and other identifying details removed. The team deliberately over-sampled hard cases: patients with long medication lists, summaries with contradictory information, handwritten notes scanned into the record, and patients transferred between hospitals.
Two experienced nurses wrote the quality criteria with the team. Every statement in a plan must be supported by the record. Every high-risk medication must be mentioned. Follow-up appointments must match what was ordered. Warning signs must be appropriate to the diagnosis. The language must be plain enough for a new navigator to act on. Each criterion was scored pass or fail, so that two reviewers would agree.
The nurses reviewed the first runs by hand. As the set grew, the team added automated checks, for example matching every medication in a plan against the medication list, and an AI-assisted grader for criteria that were harder to automate. The grader’s scores were compared each month against a sample reviewed by the nurses, and recalibrated when they drifted apart.
Every proposed change to the agent, whether a new instruction, a new model version or a new data source, ran against the full set before release. Twice, a change that looked better in a demo made the evaluation results worse, and was not released. Once, a model provider’s update changed behavior subtly; the evaluation caught it before any customer saw it.
Production monitoring fed the set. Every plan a navigator flagged as wrong was reviewed, and the most instructive cases, with details removed, were added to the evaluation set. After a year it held over 1,000 cases and reflected the real variety of the work far better than any test the team could have invented.
Common mistakes
Starting with the model instead of the problem. “We should use AI” is not a product idea. A validated customer problem is where every AI feature should begin.
Judging quality by the demo. Demos show the best case. Evaluation across hundreds of realistic examples shows what customers will actually experience.
Too much autonomy, too soon. Letting AI act on its own before the evidence supports it turns occasional errors into customer incidents.
Hiding the AI. Unlabeled AI output, without sources, makes errors harder to catch and erodes trust when they are found.
Ignoring cost until the invoice arrives. AI costs scale with use. A feature that is affordable in a pilot can be ruinous at scale.
Agents without owners. An AI agent with broad permissions and nobody accountable for it is an incident waiting to happen.
Try this
Pick one AI feature you have built, or are considering. Write its evaluation plan using the template in the Appendix: purpose, autonomy level, cost of an error, evaluation set, criteria, thresholds, fairness checks, monitoring and owner. If you cannot fill in the evaluation set and the criteria, stop and build them before you write another line of the feature. Then write a job description, from the Appendix template, for one AI agent your team uses, and check that its permissions match it.