Marcio Miike Jr. / Notes

How an AI answers from your company’s documents

A language model has not read your company’s documents. To answer from them, it is handed a few passages at the moment someone asks. The answer can only be as good as those passages.

Once, ahead of time Each time someone asks split stored as numbers finds go to writes search the question itself Documents 2,000, say Passages 40,000 pieces Index all 40,000 A few passages the closest 5 all the model sees of your documents Model a language model Answer with its sources Question in plain words 1 2 3 Once, ahead of time Each time someone asks split stored as numbers finds go to writes search the question itself Documents 2,000, say Passages 40,000 pieces Index all 40,000 A few passages the closest 5 all the model ever sees Model a language model Answer with its sources Question in plain words 1 2 3
The model never reads the archive. For each question, a search picks a few passages, and the model writes its answer from those. This is called retrieval-augmented generation, or RAG.

Reading the picture

The first part happens once, before anyone asks anything. Say a company has 2,000 documents. Each one is cut into passages of a page or less, which makes about 40,000 pieces. Each passage is turned into a list of numbers that stands for what it says, so that passages about similar things get similar numbers. The numbers go into an index.

The rest happens each time someone asks. The question is turned into numbers the same way, and the index returns the five or so passages whose numbers are closest. The model is given those passages and the question, and writes the answer.

The green box is the point. Of 40,000 passages, the model sees five. Whatever is not in them does not exist for the model.

Where it goes wrong

  • 1The splitA passage cut in the middle of a table or a procedure loses its meaning. The search can then only find half an answer.
  • 2The searchThe right passage exists but is not among the few that come back. Part numbers and codes are a common cause, because a search by meaning treats two codes one digit apart as nearly the same.
  • 3The writingThe passages do not contain the answer, and the model writes one anyway.

What follows

When an answer is wrong, look at the passages first. If the right one was not there, the fix is in the split or the search, and a better model will not help. If it was there and the model ignored it, the fix is in the instructions or the model.

That is why work on these systems is mostly work on the documents and the search.