Grounding an assistant in your own documents

The model is rarely the bottleneck

When a grounded assistant gives a confidently wrong answer, the instinct is to blame the model and reach for a bigger one. Usually the retrieval step simply did not surface the paragraph that contained the answer, and the model did what it was asked to do with the context it was handed.

Which means the work is in the unglamorous half: how documents are split, how they are searched, and what the system does when nothing relevant is found.

Chunking is a content decision, not a parameter

Splitting every document into 500-token windows is the default and it is wrong for most real corpora. A policy document has clauses. A manual has procedures. A contract has numbered sections that refer to each other. Cutting those at a fixed length separates a rule from its exception, and retrieval then returns one without the other — which is precisely how a confidently wrong answer gets produced.

Split on the document's own structure where it has one. Keep the heading path with each chunk so the model knows a passage came from “Section 4.2 — Termination” and not from nowhere. Overlap enough that a sentence spanning a boundary survives.

Hybrid search, because embeddings miss exact terms

Vector search is good at meaning and bad at identifiers. Ask for part number “HX-4420-B” and a pure embedding search will cheerfully return passages about similar-sounding parts, because the embedding of a serial number is close to the embedding of every other serial number.

Run keyword search alongside vector search and merge the results. In practice this single change fixes more real complaints than any model upgrade, because real internal questions are full of codes, names and acronyms.

• Vector search for “what is our policy on returning damaged goods?”
• Keyword search for “HX-4420-B”, “clause 7(c)”, “Form 16”.
• Rerank the merged set with a cross-encoder before it goes to the model — retrieving twenty candidates and keeping the best five beats retrieving five and hoping.

Citations are not decoration

Every sentence of an answer should be traceable to a retrieved passage the user can open. This is not a trust ornament; it is the mechanism by which the system becomes debuggable. When an answer is wrong, the citation tells you instantly whether retrieval failed or generation failed, and those have completely different fixes.

It also changes user behaviour. People check a cited answer and stop checking an uncited one, which is exactly backwards from what you want.

The refusal path is the feature

An assistant that answers everything is worse than useless in an operational setting, because its failures are indistinguishable from its successes. The system needs an explicit path for “the documents do not contain this” — triggered by retrieval scores below a threshold, and by an instruction the model is actually held to.

A good refusal says what it searched, what it found, and who to ask instead. That turns a dead end into a routing step.

What to measure before you roll it out

• Retrieval recall@k on a set of real questions with known answers. If the right chunk is not in the top k, nothing downstream can save you.
• Groundedness — does every claim in the answer appear in the cited passages? This is checkable automatically.
• Refusal accuracy — how often it refuses when it should, and how often it refuses when the answer was right there.
• Freshness — how long between a document changing and the index reflecting it. An assistant quoting last quarter's policy is a liability.

And the honest part

For a surprising number of requests the right answer is not an assistant at all. If the questions people ask are five variations of the same lookup, build the lookup. A search box over clean data beats a chat interface over messy data, costs less to run, and nobody has to review its output.