Menu
I am started working on resume retrieval(document) component based on lucene.net engine. It works great, and it fetches the document and score it based on the
the idea behind the VSM is the more times a query term appears in a document relative to the number of times the term appears in all the documents in the collection, the more relevant that document is to the query.
Lucene's Practical Scoring Function is derived from the below.
score(q,d)=coord(q,d)·queryNorm(q)· ∑( tf(t in d) ·idf(t)2 · t.getBoost() · norm(t,d) )
t in q
in this
This is very great indeed in most of the situation, but due to the fieldnorm calculation the result is not accurate
fieldnorm aka "field length norm" value represents the length of that field in that doc (so shorter fields are automatically boosted up).
Due to this we didn't get the accurate results. Say for an example i got 10000 documents in which 3000 documents got java and oracle keyword. And the no of times it appears vary on each document.
Due to the nature of the business we need to retrieve the documents got more search keyword occurrence should come first, we don't really care about the length of the document.
Because of this a Guy with a big resume with lot of keywords is been moved below in the result and some small resumes came up.
To avoid that i need to disable length normalization. Can some one help me with this??
I have attached the Luke result image for your reference.
In this image, document with java 50 times and oracle 6 times moved down to 11 th position.
But this document with java 24 times and oracle 5 times is a top scorer due to the fieldnorm.
Hope i conveyed the info clear... If not please ask me, i ll give more info
945
Lucene uses a combination of the Vector Space Model (VSM) and the Boolean model of information Retrieval to determine how relevant a document is to a user's query. It assigns a default score between 0 and 1 to all search results, depending on multiple factors related to document relevancy.
Score Boosting Lucene allows influencing search results by "boosting" in more than one level: Document level boosting - while indexing - by calling document. setBoost() before a document is added to the index.
Lucene is able to achieve fast search responses because, instead of searching the text directly, it searches an index instead. This would be the equivalent of retrieving pages in a book related to a keyword by searching the index at the back of a book, as opposed to searching the words in each page of the book.
You can disable length normalization with Field.setOmitNorms(true)
187
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
