How the Search Scoring Algorithm Works

Introduction

Have you ever wondered how Elasticsearch scores results? Or why Solr ranked your results in a particular order? Both systems use Apache Lucene internally, which scores using a combination of the Boolean Model and the Vector Space Model.

The Boolean Model (built on the inverted index) shortlists documents that match a query. The Vector Space Model then scores and sorts those documents in decreasing order of relevance. This article explains how that scoring works.

What are Vectors and Vector Space?

A vector is a quantity with both magnitude and direction. A point in 2D geometry, for example, can be represented as 2*x + 3*y — x and y are the dimensions, 2 and 3 are the magnitudes.

Vector space is the imaginary space where all vectors represented using similar dimensions coexist. In this space, we can add, subtract, and multiply vectors.

Term Vectors

In information retrieval, we represent queries and documents as term vectors. Each unique word becomes a dimension, and its frequency in the document or query becomes the magnitude.

For example, the query "Eye for an Eye" becomes:

2*(Eye) + 1*(for) + 1*(an)

The words in parentheses are dimensions. The numbers are magnitudes — in this simple case, just word frequency.

Assigning Weights to Vectors

Raw frequency isn't the best weight. We need smarter weights so that the mathematical operations on vectors produce meaningful relevance scores. The weight for each term is composed of two components: Term Frequency and Inverse Document Frequency.

Term Frequency — Rewarding Repetition

If two documents both contain query words, the one where those words appear more often is probably more relevant. Term Frequency captures this — the more times a term appears in a document, the higher its TF score.

Inverse Document Frequency — Penalising Common Words

Using only TF creates a problem. The word "the" appears in almost every document, so it would have a high TF score everywhere — but it adds no meaning to ranking.

Inverse Document Frequency fixes this. We count how many documents contain a term (document frequency), then invert it. Rare words get high IDF; common words get low IDF.

IDF = 1 / document_frequency

The final weight combines both:

Weight = TF × IDF

This gives more importance to words that appear often in a specific document but rarely across the corpus.

Cosine Similarity — How Similar are Two Vectors?

Once we have weighted term vectors for both the query and each document, we need a way to compare them. The most commonly used method is cosine similarity.

Cosine similarity measures the angle between two vectors in the vector space. If the angle is small (vectors point in the same direction), the documents are similar. If the angle is large, they are different.

Two documents that share many of the same important terms will have vectors pointing in similar directions — and thus a high cosine similarity score.

Once every matching document is scored, we sort them in decreasing order of score. The document most similar to the query comes first. That's your ranked search result.

Conclusion

We started with what vectors are, learned how to represent documents and queries as term vectors, assigned meaningful weights using TF×IDF, and calculated similarity using cosine similarity. This, in a nutshell, is how the Vector Space Model ranks search results in Lucene — and by extension in Solr and Elasticsearch.

To understand how the candidate document list is built before scoring begins, read the article on the inverted index data structure.