Evals for Product Managers

Every AI feature I've worked on has had the same moment. The demo goes well. Someone types a question, the answer looks right, and the room nods. Then a stakeholder asks, "So is it good enough to ship?" and nobody can answer.

Engineers say it depends on the prompt. Data scientists talk about accuracy on a benchmark nobody in the room has seen. Sales wants a date. The real problem is that nobody wrote down what "good" meant before the work started.

In traditional software, the spec defines good. A button either submits the form or it doesn't. AI features don't work that way. The same input can produce different outputs, and "correct" is often a matter of degree. That's why the spec for an AI feature isn't a document. It's an evaluation set: a set of real examples, the criteria for a good answer, and the bar it has to clear.

Writing that is a product manager's job. Here's how I approach it.

1. Start from the job and the cost of being wrong

Before any examples or metrics, answer three questions in plain language:

• Who uses this, and what are they trying to get done? "A sales rep answering a security question during a customer call" is a job. "A RAG chatbot over our docs" is a technology.
• What happens after the AI responds? Does a person read it and decide, or does it go straight to a customer, a record or a payment?
• What does a wrong answer cost? Not all mistakes are equal.

The last question shapes everything that follows. I sort failures into tiers:

This table is the most useful conversation you can have with stakeholders early. It turns "we want it to be accurate" into a shared view of which mistakes are tolerable and which are not.

2. Collect examples before writing requirements

Abstract requirements like "answers should be accurate and concise" sound fine and mean almost nothing. Real examples force precision.

Before building, I gather 30 to 50 real inputs the feature will face: actual customer questions, support tickets, call transcripts, documents. Then I sit with the people who do this work today and write down what an ideal response looks like for each one. Not every word, but the facts it must contain, what it must not say, and when it should decline.

Good example sets cover more than the happy path:

• Common cases that make up most of the volume.
• Hard cases where experts disagree or the answer depends on context.
• Edge cases like ambiguous questions, missing information or outdated documents.
• Should-refuse cases where the right answer is "I don't know" or "I can't help with that."
• Adversarial cases where someone tries to make the feature do something it shouldn't.

This exercise often changes the product. Teams discover that many questions can't be answered from the available data, or that experts disagree on what a good answer is. Finding that out before building is much cheaper than after.

3. Read real outputs before you write criteria

There's a trap in "define good before you build." If you write a detailed rubric from a whiteboard, you'll measure the failures you imagined, not the ones that happen. People also change their minds about what good looks like once they see real outputs. Researchers call this criteria drift, and it's normal.

So before you build means: agree on the job, the stakes and the examples first. Then put together a rough prototype in days, not weeks, run your examples through it, and read every output. This is error analysis, and it's the most valuable hour a PM can spend on an AI feature.

1. Write open notes. For each output, jot down in plain words what's wrong, from the user's point of view. Don't diagnose causes yet; just describe what you see.
2. Group the notes. After a few dozen, similar problems cluster: wrong dates, missing sources, ignoring the user's plan tier, too long for a live call. These groups are your real failure modes.
3. Count them. How often each failure happens tells you what to fix first and which criteria actually matter.
4. Keep going until nothing new appears. When fresh outputs stop revealing new kinds of failure, you've seen enough for now.

Record problems even when they aren't the model's fault. If the assistant tells a customer a refund is approved but can't show the reference number, that's a product gap worth fixing, even though no prompt change will solve it.

One person should own the call on quality. Usually that's a domain expert, such as an experienced account manager, adviser or support lead, working closely with the PM. When five people must agree on every judgement, nothing moves. If the feature truly needs five different experts to judge one answer, the scope is probably too broad.

If there's no real data yet, generate test inputs deliberately rather than asking a model for "some test questions." Pick the dimensions that vary in real use, such as customer type, question type and how much context is available, write a few combinations by hand, and only then scale them up. Replace synthetic examples with real ones as soon as you can.

4. Turn "good" into criteria you can check

The failure groups from error analysis become your criteria. Each one turns into a specific check. For a typical knowledge assistant, they might look like this:

Two rules make criteria usable:

• Prefer pass or fail to scores. "Rate helpfulness from 1 to 10" produces numbers nobody agrees on. "Does it state the renewal date correctly?" gets the same answer from everyone who checks it.
• Keep criteria separate. A single overall score hides the problem. An answer can be perfectly grounded and still useless, or fluent and quietly wrong. Separate criteria tell the team what to fix.

Three more habits keep criteria honest:

• Avoid off-the-shelf scores. Many eval tools offer ready-made metrics like "helpfulness" or "coherence," or text-similarity scores. They measure general qualities, not your users' problems. A high helpfulness score won't tell you the assistant quoted a retired price. Use them, at most, to find outputs worth reading.
• Judge the outcome, not the mechanism. Ask "did the customer get the right renewal date?" rather than "did the retrieval step return a document?" Outcomes are what users feel, and what experts can judge without understanding the system.
• Test refusals in both directions. Include roughly as many questions the feature can't answer as ones it can. A feature that refuses too often is as broken as one that makes things up, and you only see both if you test both.

If you want to track gradual progress, break an answer into smaller pass or fail checks instead of a score. "Included four of the five required facts" is measurable and meaningful; "7 out of 10" is neither.

5. Decide how each criterion gets graded

A criterion is only useful if someone, or something, can check it consistently and often. There are three ways to grade, and most features need all three.

The PM doesn't need to build the graders, but should own what they check. Write each criterion clearly enough that an engineer can turn it into a code check or a grading prompt, and an expert can apply it by hand. If two experts grade the same answer differently, the criterion isn't clear yet.

A few rules save a lot of wasted effort here:

• Fix the obvious first. Many failures disappear once the prompt states something nobody wrote down, like "keep answers under 80 words" or "always cite the source." Fix those directly; they don't need an evaluator.
• Use the cheapest grader that works. A code check costs almost nothing to run and maintain. A model-based grader needs labelled examples, tuning and regular upkeep. Save it for failures you'll keep working on.
• One grader per failure. A model-based grader asked to judge "overall quality" is hard to trust and impossible to act on. A grader that only answers "did it escalate when it should have?" can be checked and improved.
• Prove the grader agrees with your expert. Have the expert label a set of outputs as pass or fail, then compare. Track two numbers: how many real failures the grader catches, and how many good answers it wrongly flags. Keep some labelled examples aside that were never used while tuning the grader, and check it against those at the end. If it only does well on the examples it was tuned on, it can't be trusted.

6. Set the bar before you build

The non-negotiables should be agreed before the first prompt is written. The quality targets come after the first round of error analysis, once you know which failures are real, but before the team starts tuning. Set them any later and they drift to whatever the current version happens to score.

A launch bar has three parts:

• Non-negotiables. Zero failures allowed on the harmful tier. If the feature invents a certification even once in the eval set, it doesn't ship.
• Quality targets. For each remaining criterion, the pass rate needed to launch, and a higher one to aim for. Base these on how people perform today. If experienced reps answer a question type correctly most of the time, the AI should match that before it replaces their first draft.
• Operational limits. How fast it must respond and what it can cost per use. A perfect answer that arrives after the customer call has ended is a failed answer.

Also decide what the feature does when it isn't confident. "Escalate to a person" or "show the source and let the user decide" are product decisions, and they often matter more than another point of accuracy.

Stakeholders will want one number. Resist a weighted average of scores; when it moves from 3.2 to 3.7, nobody knows what improved for users. A simpler figure works better: the share of examples that pass every check, reported separately for release-blocking checks and for everything else, with the option to drill down into each check.

And don't aim for 100%. If every example passes, the eval set is too easy and has stopped teaching you anything. Add harder cases until it challenges the feature again.

Write all of this down next to the eval set. When a stakeholder asks "is it good enough to ship?", the answer is a table, not an opinion.

7. Keep the evals alive after launch

The eval set you write before building is your best guess at what users will ask. Production shows you what they actually ask, and it's always different.

The eval set is a loop, not a launch checklist

Real examples

inputs, ideal answers

Criteria

pass or fail

Graders

code, model, human

Launch bar

agreed up front

Ship and monitor

ratings, edits, escalations

Production failures

every miss becomes a case

Ship and monitor → what users actually hit → production failures → every miss becomes a new real example.

After launch, watch the signals that show where the feature falls short: low ratings, answers users heavily edit or ignore, questions escalated to a person, and support complaints. Review a sample every week with the experts who helped write the original examples.

Every real failure goes back into the eval set with its ideal answer. Run the full set on every change to the prompt, model, data or tools, and block releases that make it worse. Over time, the eval set becomes the most accurate description of what your users need, far more accurate than the original requirements.

A rhythm that works: read 10 to 20 production outputs every week, focusing on the unusual ones such as long conversations, retries, escalations and low ratings, plus a few picked at random so you still find what no signal points to. Run a fuller error analysis after any big change: a new model, a new data source, a new user group, or a spike in complaints.

Eval sets go stale as the product and its users change. Retire checks that pass every time, especially expensive ones. Don't worry that scores stop being comparable with last quarter's; for long-term trends, track product metrics such as adoption, task completion and churn alongside the evals.

Two things are worth protecting. First, keep error analysis in the team. Handing it to an outside labelling vendor breaks the link between seeing a failure and understanding the product, and that understanding is the point. Second, if outputs are hard for experts to check, treat it as a product problem. Show the sources, the extracted facts and the reasoning steps so the person can verify the answer in seconds. A feature that is easy to check is easier to evaluate and easier to trust.

The eval set is the spec

Product managers sometimes treat evals as something the ML team does after the product decisions are made. I think that's backwards. Choosing which examples matter, what a good answer contains, which mistakes are unacceptable and what bar justifies launch are product decisions. They're the most important product decisions in an AI feature.

Models will change. Prompts will be rewritten. The team may switch providers twice before the feature is a year old. The eval set is what stays, and it's what lets you make each of those changes with confidence instead of hope.

If you can't say what "good" means for your AI feature, you're not ready to build it. If you can, you've already written the most important part of the spec.

Further reading: several of the practices here, especially error analysis, a single quality owner, pass or fail checks and validating model-based graders, are covered in depth in Hamel Husain and Shreya Shankar's AI Evals FAQ.