Apache Lucene Tutorial

Introduction

Apache Lucene is a full-text search library written in Java. It is a top-level Apache Project and was written in 1999 by Doug Cutting. Lucene is a powerful library that forms the core of many search-based technologies like Elasticsearch and Solr.

Lucene is very fast — it can find a document containing a given word among millions of documents in milliseconds, with indexing speeds as high as 700GB per hour.

This tutorial will first go over the basic concepts of Apache Lucene, then explore the Lucene API, and finally build a simple search application. We will be using Lucene 8.

History of Apache Lucene

Lucene was first published in 1999 by Doug Cutting on SourceForge. In September 2001 it joined the Jakarta family of Apache Software Foundation. In January 2005, it became its own top-level project. Initially, various other projects like Mahout, Tika, and Nutch were part of Lucene before separating into their own Apache projects.

Lucene vs Solr and Elasticsearch

While Lucene is just a Java library, Elasticsearch and Solr are full-fledged search servers. Think of Lucene as the engine, Elasticsearch as a Ferrari, and Solr as a Lamborghini.

With Lucene you get core search capabilities. Elasticsearch and Solr add HTTP interfaces, distributed computing, automatic performance management, and many other production features on top.

Adding Dependencies

We need lucene-core to start. Optionally add lucene-queryparser for query parsing and lucene-analyzers-common for additional analyzers.

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-core</artifactId>
  <version>8.10.1</version>
</dependency>

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-queryparser</artifactId>
  <version>8.10.1</version>
</dependency>

<dependency>
  <groupId>org.apache.lucene</groupId>
  <artifactId>lucene-analyzers-common</artifactId>
  <version>8.10.1</version>
</dependency>

A Simple Indexer

Let's create a simple indexer that adds three movies to a Lucene index:

public class SimpleIndexer {
  public void index(String indexPath) throws IOException {
    Directory directory = FSDirectory.open(Paths.get(indexPath));
    Analyzer analyzer = new StandardAnalyzer();
    IndexWriterConfig indexWriterConfig = new IndexWriterConfig(analyzer);
    IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);

    Document movie1 = new Document();
    movie1.add(new TextField("title", "Harry Potter and the Prisoner of Azkaban", Field.Store.YES));

    Document movie2 = new Document();
    movie2.add(new TextField("title", "Lord of the Rings: The fellowship of the ring.", Field.Store.YES));

    Document movie3 = new Document();
    movie3.add(new TextField("title", "Toy Story 3", Field.Store.YES));

    indexWriter.addDocument(movie1);
    indexWriter.addDocument(movie2);
    indexWriter.addDocument(movie3);
    indexWriter.close();
  }
}

A Simple Searcher

Now let's create a searcher that finds a movie by title:

public class SimpleSearcher {
  public void search(String indexPath, String title) throws IOException, ParseException {
    IndexReader indexReader = DirectoryReader.open(FSDirectory.open(Paths.get(indexPath)));
    IndexSearcher indexSearcher = new IndexSearcher(indexReader);
    Analyzer analyzer = new StandardAnalyzer();
    QueryParser queryParser = new QueryParser("title", analyzer);
    Query query = queryParser.parse(title);

    TopDocs topDocs = indexSearcher.search(query, 10);
    System.out.println(String.format("Found %d hits.", topDocs.totalHits.value));

    for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
      Document movie = indexSearcher.doc(scoreDoc.doc);
      System.out.println(String.format("Found: %s", movie.get("title")));
    }
  }
}

Output for query "Harry Potter":

Found 1 hits.
Found: Harry Potter and the Prisoner of Azkaban

Core Concepts

Index

All documents of similar types are stored in an index. Think of it like a database table. Internally Lucene uses an inverted index — a mapping of words to the list of documents containing them.

Documents

A document is the basic unit of search in Lucene. When you search, Lucene returns document IDs. A document is a collection of fields.

Fields

A field is a subunit of a document — think of it as an attribute or database column. For a blog index, you might have fields for title, body, date, and author. Lucene supports searching by specific fields.

Analysis

Analysis is the process of splitting text into tokens for the inverted index. The tokens generated at index time must match those generated at query time. If they don't match, you won't see correct results.

Indexing API

Directory

The Directory interface abstracts how the index is stored — file system, memory, or network. Use FSDirectory.open() for persistent indexes:

Path indexPath = Files.createTempDirectory("tempIndex");
Directory directory = FSDirectory.open(indexPath);

Field Types

Choose the right field type for your data:

• TextField — for full-text search
• StringField — for exact match
• IntPoint — for integer data

IndexWriter

The IndexWriter adds documents to the index. It needs a Directory and IndexWriterConfig:

Directory dir = FSDirectory.open(Paths.get("index"));
IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer());
IndexWriter writer = new IndexWriter(dir, config);
writer.addDocument(doc);

Analyzer

An Analyzer converts text into tokens. StandardAnalyzer splits on word boundaries, lowercases tokens, and removes stop words.

Search API

Query

The Query class is the base for all Lucene queries. Common types include TermQuery, PhraseQuery, MatchAllDocsQuery, and BooleanQuery.

QueryParser

QueryParser lets you write queries as text rather than constructing Query objects in code:

QueryParser parser = new QueryParser("body", new StandardAnalyzer());
Query query = parser.parse("body:panda");

IndexSearcher

Runs the query and returns results as a TopDocs object:

IndexSearcher searcher = new IndexSearcher(DirectoryReader.open(directory));
TopDocs topDocs = searcher.search(query, 10);

TopDocs and ScoreDoc

TopDocs contains the total hit count and an array of ScoreDoc objects, each holding a document ID and score. Fetch the full document via:

Document doc = searcher.doc(scoreDoc.doc);

Conclusion

In this tutorial we covered what Lucene is, how it differs from Solr and Elasticsearch, built a simple Java application, and walked through the Indexing and Search APIs. You now have the foundation to build your own Lucene-powered search application.