Making Every Rep Your Best Rep

Everyone has the documents. Almost nobody reads them.

On a recent Prodaptix client project, I worked with the sales and account management team. Like every company I have seen, they were not short of documentation. Product updates, release notes, pitch decks, FAQs, pricing pages, battlecards. Someone had written all of it, carefully.

When we looked at how much of it the reps actually read, it was a single-digit percentage. Not because they were lazy. They were on calls, in inboxes and chasing renewals. Reading a long release note was never going to win against the next meeting.

So the knowledge existed. It just never reached the person talking to the customer.

The problem isn't the documents. It's too much context, found too late.

My first instinct was the usual one: clean up the docs, write shorter summaries, run another enablement session. Then I watched how reps actually worked, and it was clear that more content would only make it worse.

The problem was not missing information. It was too much of it, spread across too many places, with no way to find the right piece at the moment it mattered. And that moment is always the tough one: a customer asks about a new feature mid-call, or pushes back on pricing, or wants to know if last month's update fixes their issue.

In that moment, nobody opens a shared drive and starts searching. They guess, they stall, or they promise to follow up. Each of those costs a little trust.

Before we built anything

It was tempting to start with a model and a vector database. We started with the reps instead.

• Collect the real questions. We gathered the questions reps and account managers actually ask, from calls, emails and internal chats, and the ones they wish they could ask mid-call.

• Map where the answers live. For each type of question, we found which documents held the answer, who owned them, and how often they changed.

• Decide what counts as the truth. Where two sources disagreed, we agreed which one wins, and that the latest product update beats an older deck.

• Define success up front. We agreed on what we would measure before writing any code, so we could tell whether the assistant was helping or just impressive in a demo.

That groundwork shaped every technical decision that followed.

What we built: the right context, at the moment it's needed

We built a RAG-based assistant on top of the company's existing documents and product updates. The goal fit on one line: make every sales rep and account manager as good as your best one.

Your best rep doesn't know more because they read more. They know where things are, they remember what changed, and they bring it up at the right time. The assistant gives everyone that, in three moments:

• Before the call. Ask what changed for this customer's product, get a briefing, and practise answers to the questions you expect.

• During the call. Ask a quick question and get a short, sourced answer while the customer is still talking.

• After the call. Draft the follow-up email with the right product details already in it.

Keeping it true: real product updates, not last quarter's deck

An assistant like this is only as useful as it is current. If it repeats an old feature list or a retired price, reps stop trusting it after the first mistake, and they don't come back.

So product updates were the heart of the system, not an afterthought. New releases and changes flow into it as they ship, and when two sources disagree, the newer, higher-ranked source wins.

The technical decision that mattered most was search. Semantic search is good at meaning but blurs exact identifiers: a plan name, a feature name, a version number. Reps ask in exactly those terms, and "Growth" and "Growth Plus" are not the same answer. So we combined keyword search (BM25) for exact matches with vector search for meaning, and merged the two rankings. The trade-off is more tuning and one more moving part, in exchange for answers that don't mix up similarly named plans.

Every answer shows where it came from, and who to ask

A rep about to quote a renewal price needs more than an answer. They need to know if they can say it to the customer. So every answer comes with a citation card, not just a link.

Here is what that looks like for a question like "What's the renewal price for this customer's plan?" (an illustrative example; names and figures are placeholders):

• Answer: Renewal price for the plan, per seat per year, with the applicable discount rules.
• Source: Official price list, renewals section.
• Last updated by: Pricing owner, RevOps.
• Last updated: 3 days ago.
• Cited on: Today, at the time of the question.
• Confidence: Medium: the price list and the latest product update differ on one plan feature.

The confidence score combines how strongly the sources matched, whether they agree, and how recent they are. But a model's confidence is not proof of correctness, so the score never decides on its own. We checked the thresholds against the evaluation set and set explicit rules:

• Pricing and contract terms: always confirmed with the owner before they are committed to a customer, whatever the score.

• Sources disagree: the assistant shows both, names the conflict, and caps the answer at medium.

• No reliable source: it says "I don't know" and routes the question to the right person instead of guessing.

• Source past its review date: the answer is flagged as possibly outdated, even if it matches well.

• Everything else: high confidence can be used directly; medium or low means check first.

Checking first used to mean hunting for the right person in a chat channel. Now the assistant does that step too. When an answer needs checking, it drafts a message to the person who last updated the source, quoting the exact answer and citation:

Hi [owner name], I'm about to confirm the renewal price for [customer] on a call. The assistant pulled this from the price list you updated 3 days ago: [quoted answer]. It flagged a difference with the latest product update on [feature]. Can you confirm this is still correct to quote?

The rep reviews it, sends it with one click, and the owner's reply goes back into the source. Instead of a vague question in a busy channel, the rep sends a short, specific message to the one person who knows.

How we measured it: evals and LLM as a judge

The real questions we collected became our evaluation set. For each one, someone who knew the answer wrote the expected answer and the source it should come from. Every change to chunking, search or prompts ran against this set before it reached reps.

We measured quality at three levels:

• Retrieval: Did the right source appear in the top results? Is the assistant even looking in the right place?
• Answer quality: Correctness, faithfulness to the source and completeness. Is the answer right, and does it only say what the source says?
• Correct refusal: Does it say "I don't know" instead of guessing?
• Business: Usage, time to answer, customer outcomes and leads closed. Is it actually making reps better?

Grading every answer by hand after every change doesn't scale, so we used an LLM as a judge. A second model receives the question, the retrieved sources, the assistant's answer and the expected answer, and scores it against a written rubric: is it correct, is every claim supported by the sources, and does it answer what was asked.

We didn't trust the judge blindly. We regularly checked a sample of its scores against human reviewers from the sales team, and tightened the rubric wherever they disagreed. The judge gave us speed; the human spot checks kept it honest.

Running it: documents never stop changing

Launch was the easy part. Products ship, pricing shifts, decks get rewritten. Without continuous operation, an assistant like this quietly starts giving outdated answers.

So the ingestion pipeline runs continuously:

• Detect changes. New and edited documents and product updates are picked up from their source as they change.

• Re-embed only what changed. For an edited document, its old chunks are removed and new ones created. A new document is simply added.

• Retire what's outdated. Superseded content is marked as such, so it no longer competes with the current version.

• Re-run the evals. Every content update and every system change runs against the evaluation set, with the LLM judge, before it reaches reps.

• Close the loop. When a rep flags a wrong answer, it goes to the owner of that document, and the question is added to the evaluation set.

That last step mattered most. The evaluation set grew from the field, so it kept testing what reps were actually asking, not what we had guessed at the start.

How usage changed

Before the assistant, a rep facing an unfamiliar question had three options: guess, stall, or promise to follow up and then ask a colleague. The documents were never really in the loop.

After it, the documents became part of the conversation. Reps began asking questions before calls to prepare and practise, during calls when a customer raised something new, and after calls to write follow-ups with the right product details. Questions that used to go to colleagues went to the assistant first.

We tracked usage the same way we tracked quality: how many reps used it each week, how many questions they asked, which moment they used it in, and how often they still needed a human. Usage was the leading signal. When reps trusted the answers, they came back; when an answer was wrong, we saw it in the flags before we saw it anywhere else.

What changed: knowledge in the conversation

The change showed up where it counts. Customers were happier, because their questions got real answers on the call instead of "let me get back to you". Reps walked in prepared, knew what had shipped, and could connect a new feature to the customer's actual problem.

The docs hadn't changed. The reps hadn't suddenly started reading more. The difference was that the knowledge finally showed up in the conversation, at the moment it could do something.

---

Explore more of my work in Applied AI and Architecture, or get in touch.