Search Engine Keywords Parser

Question

Here is what I want to do:

I need to create a search engine parser that uses the following operators:

• Apples AND Oranges (AND operator)
• Apples OR Oranges (OR operator)
• Apples AND NOT Oranges (AND NOT operator)
• " Apples " (Quotes operator)
• Apples AND ( Oranges OR Pears ) (Parentheses operator)
• Appl* (Star operator)

With some preg_replace, I manage to convert the string into an array and then I parsed this array to get a MySQL query. But I don't like that way and it's very unstable!

I searched the web for some script that does that and I didn't have any luck!

Can someone please help me implement this??

Thanks

Answer 1

Ok, this is going to be a large answer.

I think what you need is a parser generator. A piece of software that generates code to parse text according to a given grammar. These parsers often have 2 main components: a lexer and a parser. The lexer identify TOKENS (words), the parser check whether the token order is right according to your grammar.

In the lexer, you should declare the following tokens

TOKENS ::= (AND, OR, NOT, WORD, WORDSTAR, LPAREN, RPAREN, QUOTE)
WORD ::= '/w+/'
WORDSTAR ::= '/w+\*/'

The grammar should be defined like this:

QUERY ::= word
QUERY ::= wordstar
QUERY ::= lparen QUERY rparen
QUERY ::= QUERY and QUERY
QUERY ::= QUERY or QUERY
QUERY ::= QUERY and not QUERY
QUERY ::= quote MQUERY quote
MQUERY ::= word MQUERY
MQUERY ::= word

This grammar defines a language with all the features your need. Depending on the software you use, you could define functions to handle each rule. That way, you can transform your text-query into a sql where clause.

I'm not really into php, but i searched the web for a parser generator and PHP_ParserGenerator appeared.

Keep in mind that as long as your database grows these queries may become a problem for a structured storage system.

You may want to try a full-text search engine that allows you to perform this and many other features related to text search. This is how IndexTank works

First, you add (or 'index' in search dialect) all your db records (or documents) to IndexTank.

$api = new ApiClient(...);
$index = $api->get_index('my_index');
foreach ($dbRows as $row) {
  $index->add_document($row->id, array('text' => $row->text));
}

After that, you can search in the index with all the operators you want

$index = $api->get_index('my_index');
$search_result = $index->search('Apples AND Oranges');
$search_result = $index->search('Apples OR Oranges');
$search_result = $index->search('Apples AND NOT Oranges');
$search_result = $index->search('"apples oranges"');
$search_result = $index->search('Apples AND ( Oranges OR Pears )');
$search_result = $index->search('Appl*');

I hope I answered your question.

Answer 2

Also, this is not exactly what you're looking for, but maybe close: MySQL Full-text searching.

• http://devzone.zend.com/article/1304
• http://www.artfulcode.net/articles/full-text-searching-mysql/
• http://jeremy.zawodny.com/blog/archives/000576.html