Add field in Lucene document

Question

Hello I have a 32mb file. It is a simple dictionary file encoded 1250 with 2.8 million lines in it. Every line has only one unique word:

cat
dog
god
...

I want to use Lucene to search for every anagram in dictionary of specific word. For example:

I want to search every anagram of the word dog and lucene should search my dictionary and return dog and god. In my webapp I have a Word Entity:

public class Word {
    private Long id;
    private String word;
    private String baseLetters;
    private String definition;
}

and baseLetters is the variable which are sorted letters alphabetically for searching such anagrams [god and dog words will have the same baseLetters: dgo]. I succeeded in searching such anagrams from my database using this baseLetters variable in different service but I have problem to create index of my dictionary file. I know I have to add to fields:

word and baseLetters but I have no idea how to do it :( Could someone show me some directions to achieve this goal?

For now I have only something like that:

public class DictionaryIndexer {

private static final Logger logger = LoggerFactory.getLogger(DictionaryIndexer.class);

@Value("${dictionary.path}")
private String dictionaryPath;

@Value("${lucene.search.indexDir}")
private String indexPath;

public void createIndex() throws CorruptIndexException, LockObtainFailedException {
    try {
        IndexWriter indexWriter = getLuceneIndexer();
        createDocument();           
    } catch (IOException e) {
        logger.error(e.getMessage(), e);
    }       
 }

private IndexWriter getLuceneIndexer() throws CorruptIndexException, LockObtainFailedException, IOException {
    StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
    IndexWriterConfig indexWriterConfig = new IndexWriterConfig(Version.LUCENE_36, analyzer);
    indexWriterConfig.setOpenMode(OpenMode.CREATE_OR_APPEND);
    Directory directory = new SimpleFSDirectory(new File(indexPath));
    return new IndexWriter(directory, indexWriterConfig);
}

private void createDocument() throws FileNotFoundException {
    File sjp = new File(dictionaryPath);
    Reader reader = new FileReader(sjp);

    Document dictionary = new Document();
    dictionary.add(new Field("word", reader));
}

}

PS: One more question. If i register DocumentIndexer as a bean in Spring will the index be creating/appending every time I redeploy my webapp? and the same will be with the future DictionarySearcher?

Answer 1

Lucene isn't the best tool for this, because you aren't doing a search: you are doing a lookup. All the real work occurs in the "indexer" and then you just store the results of all your work. The lookup can be O(1) in any hash-type storage mechanism.

Here's what your indexer should do:

1. Read the entire dictionary into a simple structure like a SortedSet or String[]
2. Create an empty HashMap<String,List<String>> (probably the same size, for performance) for storing the results
3. Iterate through the dictionary alphabetically (really any order will work, just make sure you hit all entries)
   1. Sort the letters in the word
   2. Look-up the sorted-letters in your storage collection
   3. If the lookup succeeds, add the current word to the list ; otherwise, create a new list containing the word and put it into the storage Map
4. If you need this map later, store the map on the disk; otherwise, keep it in memory
5. Discard the dictionary

Here's what your lookup process should do:

1. Sort the letters in the sample word
2. Look-up the sorted-letters in your storage collection
3. Print the List that comes back from the lookup (or null), taking care to omit the sample word from the output

If you want to save heap space, consider using a DAWG. You'll find that you can represent the entire dictionary of English words in a few hundred kilobytes instead of 32MiB. I'll leave that as an exercise for the reader.

Good luck with your homework assignment.

Answer 2

The function createDocument() should be

private void createDocument() throws FileNotFoundException {
    File sjp = new File(dictionaryPath);
    BufferedReader reader = new BufferedReader(new FileReader(sjp));

    String readLine = null;
    while((readLine = reader.readLine() != null)) {
        readLine = readLine.trim();
        Document dictionary = new Document();
        dictionary.add(new Field("word", readLine));
        // toAnagram methods sorts the letters in the word. Also makes it
        // case insensitive.
        dictionary.add(new Field("anagram", toAnagram(readLine)));
        indexWriter.addDocument(dictionary);
    }
}

If you are using Lucene for a lot of functionality, considering using Apache Solr, a search platform built on top of Lucene.

You could also model your index with just one entry per anagram group.

{"anagram" : "scare", "words":["cares", "acres"]}
{"anagram" : "shoes", "words":["hoses"]}
{"anagram" : "spore", "words":["pores", "prose", "ropes"]}

This would require updates to existing documents in the index while processing your dictionary file. Solr would help with higher level API in such cases. For e.g., IndexWriter does not support updating documents. Solr supports updates.

Such an index would give one result document per anagram search.

Hope it helps.