Best way to parse a text document

Question

I'm trying to parse a plain text document in PHP but have no idea how to do it correctly. I want to separate each word, assign them an ID and save the result in JSON format.

Sample text:

"Hello, how are you (today)"

This is what im doing at the moment:

$document_array  = explode(' ', $document_text);
json_encode($document_array);

The resulting JSON is

[["Hello,"],["how"],["are"],["you"],["(today)"]]

How do I ensure that spaces are kept in-place and that symbols are not included along with the words...

[["Hello"],[", "],["how"],[" "],["are"],[" "],["you"],["  ("],["today"],[")"]]

I’m sure some sort of regex is required... but have no idea what kind of pattern to apply to deal with all cases... Any suggestions guys?

Answer 1

This is actually a really complex problem, and one that's subject to a fair amount of academic reaserch. It sounds so simple (just split on whitespace! with maybe a few rules for punctuation...) but you quickly run into issues. Is "didn't" one word or two? What about hyphenated words? Some might be one word, some might be two. What about multiple successive punctuation characters? Possessives versus quotes? etc etc. Even determining the end of a sentence is non-trivial. (It's just a full stop right?!)

This problem is one of tokenisation and a topic that search engines take very seriously. To be honest you should really look at finding a tokeniser in your language of choice.

Answer 2

Maybe this:?

array_filter(preg_split('/\b/', $document_text))

the 'array_filter', removes the empty values at the first and/or last index of the resulting array, which will appear if your string start or ends with a word boundary (\b see: http://php.net/manual/en/regexp.reference.escape.php)