Marcio Miike Jr. / Notes

How to tell whether search got better

After a change to a search system, someone asks whether it got better. Looking at a few results does not answer that. A fixed set of questions and one number does.

Before the change position in the results score Question 1 1 2 3 4 5 1 ÷ 2 = 0.50 Question 2 1 2 3 4 5 1 ÷ 1 = 1.00 Question 3 1 2 3 4 5 not found = 0 Question 4 1 2 3 4 5 1 ÷ 5 = 0.20 Average of the four 0.43 After the change position in the results score Question 1 1 2 3 4 5 1 ÷ 1 = 1.00 Question 2 1 2 3 4 5 1 ÷ 1 = 1.00 Question 3 1 2 3 4 5 1 ÷ 3 = 0.33 Question 4 1 2 3 4 5 1 ÷ 2 = 0.50 Average of the four 0.71 Before the change position in the results score Question 1 1 2 3 4 5 1 ÷ 2 = 0.50 Question 2 1 2 3 4 5 1 ÷ 1 = 1.00 Question 3 1 2 3 4 5 not found = 0 Question 4 1 2 3 4 5 1 ÷ 5 = 0.20 Average of the four 0.43 After the change position in the results score Question 1 1 2 3 4 5 1 ÷ 1 = 1.00 Question 2 1 2 3 4 5 1 ÷ 1 = 1.00 Question 3 1 2 3 4 5 1 ÷ 3 = 0.33 Question 4 1 2 3 4 5 1 ÷ 2 = 0.50 Average of the four 0.71
The same four questions, before and after a change. The green square is where the right document came up. Its score is one divided by its position, and the average of the scores is the number to compare. The four questions here are an example.

Reading the picture

First, collect real questions, and for each one have a person mark the document that answers it. This set stays fixed.

Run every question through the search and look at where the right document comes up. First place scores 1. Second place scores a half, third a third, and so on. If it is not in the results at all, the score is 0.

The average of the scores is called the mean reciprocal rank, or MRR. In the picture it goes from 0.43 to 0.71, about 1.7 times.

Why this number

People rarely look past the first few results. This number pays most for putting the right document first, and almost nothing for putting it tenth.

It also makes arguments short. Two ways of searching, run on the same questions, give two numbers.

What it does not say

It says nothing about the answer a model writes from the document. It also assumes one right document for each question, which is not always true.

Where I used it

I used it on a search across an advertising agency’s internal documents: about 25 terabytes of proposals, research reports and contracts. We compared ways of searching on the same questions, including one that combines search by words with search by meaning.

Over the pilot, the number rose 1.7 times. The time to find a document went from about three hours to as little as ten minutes. It is one of the five systems in the chart.