Precision recall in lucene java

Question

I want to use Lucene to calculate Precision and Recall.

I did these steps:

1. Made some index files. To do this I used indexer code and indexed .txt files which exist in this path C:/inn (there are 4 text files in this folder) and take them in "outt" folder by setting the indexpath to C:/outt in the Indexer code.

2. Created a package called lia.benchmark and a class inside it which is called "PrecisionRecall" and add externaljars (rightclick --> Java build path --> add external jars) and added Lucene-benchmark-.3.2.0jar and Lucene-core-3.3.0jar

3. Set the topicsfile path in code to C:/lia2e/src/lia/benchmark/topics.txt and
   qrelsfile to C:/lia2e/src/lia/benchmark/qrels.txt and dir to "C:/outt".

   Here is code:

   package lia.benchmark;        
   import java.io.File;  
   import java.io.PrintWriter;  
   import java.io.BufferedReader;  
   import java.io.FileReader;  
   import org.apache.lucene.search.*;  
   import org.apache.lucene.store.*;  
   import org.apache.lucene.benchmark.quality.*;  
   import org.apache.lucene.benchmark.quality.utils.*;  
   import org.apache.lucene.benchmark.quality.trec.*;  
   
    public class PrecisionRecall {  
   
      public static void main(String[] args) throws Throwable {  
   
     File topicsFile = new File("C:/lia2e/src/lia/benchmark/topics.txt");  
            File qrelsFile = new File("C:/lia2e/src/lia/benchmark/qrels.txt");  
            Directory dir = FSDirectory.open(new File("C:/outt"));  
            IndexSearcher searcher = new IndexSearcher(dir, true);  
   
            String docNameField = "filename";  
   
            PrintWriter logger = new PrintWriter(System.out, true);  
   
            TrecTopicsReader qReader = new TrecTopicsReader();   
            QualityQuery qqs[] = qReader.readQueries(                        
                    new BufferedReader(new FileReader(topicsFile)));  
   
            Judge judge = new TrecJudge(new BufferedReader(          
                   new FileReader(qrelsFile)));                                          
   
            judge.validateData(qqs, logger);                                          
   
            QualityQueryParser qqParser = new SimpleQQParser("title", "contents");  
   
            QualityBenchmark qrun = new QualityBenchmark(qqs, qqParser, searcher, docNameField);  
      SubmissionReport submitLog = null;  
            QualityStats stats[] = qrun.execute(judge,                   
                      submitLog, logger);  
   
           QualityStats avg = QualityStats.average(stats);          
           avg.log("SUMMARY",2,logger, "  ");  
      dir.close();  
     }  
   } 
   

4. Initialized qrels and topics. In documents folder (C:\inn) I have 4 txt files which 2 of them is relevance to my query ( query is apple) so I filled qrels and topics.

   the qrels file like this:

   <top>  
       <num> Number: 0 
       <title> apple
       <desc> Description:  
       <narr> Narrative:  
   </top>  
   

   and topics file like this:

   0    0      789.txt           1
   0    0      101.txt           1
   

   I tried also the Path format namely for example "C:\inn\789.txt" instead of "789.txt" but results are zero:

   0 - contents:apple
   0 Stats:
   Search Seconds: 0.016
   DocName Seconds: 0.000
   Num Points: 2.000
   Num Good Points: 0.000
   Max Good Points: 2.000
   Average Precision: 0.000
   MRR: 0.000
   Recall: 0.000
   Precision At 1: 0.000
   SUMMARY
   Search Seconds: 0.016
   DocName Seconds: 0.000
   Num Points: 2.000
   Num Good Points: 0.000
   Max Good Points: 2.000
   Average Precision: 0.000
   MRR: 0.000
   Recall: 0.000
   Precision At 1: 0.000
   

Can you tell me what is wrong with me?

I really need to know why results are zero.

Answer 1

I'm afraid that the qrels.txt format is wrong: the javadoc suggests the following:

Expected input format:

 qnum  0   doc-name     is-relevant

Two sample lines:

 19    0   doc303       1
 19    0   doc7295      0

(I know it's 2.3.0 javadoc, but the format wasn't changed in 3.0)

So it seems that you've swapped the files: TrecTopicsReader expects what you have in qrels.txt; TrecJudge expects what you have in topics.txt.