Tuesday, March 22, 2016

Starting Search with Apache Lucene 5.3.x/5.4.x

Before we delve into Apache Lucene, the following are the most important terms that you need to be familiar with. This will also help you clarify a few terms before getting into 'search' or 'information retrieval':

Let us get ahead with Apache Lucene 5.3.x/5.4.y; The most important aspects of Lucene is mentioned under each of the headings.

1. Lucene Introduction (Usage)
  • Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
  • Apache Lucene is an open source project available for free download
  • Scalable, High-Performance Indexing
  • Powerful, Accurate and Efficient Search Algorithms
  • Cross-Platform Solution
2. LuceneTerms (Concepts)
  • Inverted Index
    Inverted Index is used to get traverse from the string or search term to the document id's or locations of these terms. If we were to visualize this in terms of an 'index' - it would be 'inverted', as we would be using the term as a handle to retrieve 'id' or 'locations' - reverse of the popular usage of an index.

  • Index
    Index is an handle (information) that can be used to get further related information from a file, database or any other source of data. Usually, Index is also accompanied by compression, check-sum, hash or location of the remaining data. Index contains multiple Documents.

  • Document
    Document is a collection of Fields and the Values against each of the Fields. It is more like saying "Employee Name" - "Sumith Puri" | "Employee Desingation" - "Software Architect" | "Employee Age" - "33" | "Employee ID" - "067X" forms a document. The Lucene Indexing Process adds multiple documents to an Index. The entire set of Documents is called the Corpus.
  • Field
    contains Terms and are simply 'Sets of Tokens' of information. The Lucene Indexing process take care to Identify (or Process) Fields and Index them. Fields belong to a Document always.

  • Terms
    Terms are nothing but a 'Token' or 'String' of Information. This 'Term' is the smallest piece of Information that will be Indexed to form the Inverted Index. Set of Distince Terms is called the Vocabulary.

  • String
    String is simply a 'Token' or the English Language String.

  • Segment
    Segment is a fragmented or chunked part of the entire Index, for better storage and faster retrieval.

3. Lucene Segment (Indexing)

Each segment index maintains the following:

Field Names: This contains the set of field names used in the index.

Stored Field Values: This contains, for each document, a list of attribute-value pairs, where the attributes are field names. These are used to store auxiliary information about the document, such as its title, url, or an identifier to access a database. The set of stored fields are what is returned for each hit when searching. This is keyed by document number.   

Term Dictionary: A dictionary containing all of the terms used in all of the indexed fields of all of the documents. The dictionary also contains the number of documents which contain the term, and pointers to the term's frequency and proximity data.

Term Frequency Data: For each term in the dictionary, the numbers of all the documents that contain that term, and the frequency of the term in that document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)

Term Proximity Data: For each term in the dictionary, the positions that the term occurs in each document. Note that this will not exist if all fields in all documents omit position data.

Normalization Factors: For each field in each document, a value is stored that is multiplied into the score for hits on that field.

Term Vectors: For each field in each document, the term vector (sometimes called document vector) may be stored. A term vector consists of term text and term frequency. To add Term Vectors to your index see the Field constructors

Deleted Documents: An optional file indicating which documents are deleted. 

4. Lucene Internals (Architecture)

    Fig. 1: Lucene Architectural Layers [Non-Copyrighted Image]