fixing words with spaces using a dictionary look up in python?

Question

I have extracted the list of sentences from a document. I am pre-processing this list of sentences to make it more sensible. I am faced with the following problem

I have sentences such as "more recen t ly the develop ment, wh ich is a po ten t "

I would like to correct such sentences using a look up dictionary? to remove the unwanted spaces.

The final output should be "more recently the development, which is a potent "

I would assume that this is a straight forward task in preprocessing text? I need help with some pointers to look for such approaches. Thanks.

Answer 1

Take a look at word or text segmentation. The problem is to find the most probable split of a string into a group of words. Example:

 thequickbrownfoxjumpsoverthelazydog

The most probable segmentation should be of course:

 the quick brown fox jumps over the lazy dog

Here's an article including prototypical source code for the problem using Google Ngram corpus:

• http://jeremykun.com/2012/01/15/word-segmentation/

The key for this algorithm to work is access to knowledge about the world, in this case word frequencies in some language. I implemented a version of the algorithm described in the article here:

• https://gist.github.com/miku/7279824

Example usage:

$ python segmentation.py t hequi ckbrownfoxjum ped
thequickbrownfoxjumped
['the', 'quick', 'brown', 'fox', 'jumped']

Using data, even this can be reordered:

$ python segmentation.py lmaoro fll olwt f pwned
lmaorofllolwtfpwned
['lmao', 'rofl', 'lol', 'wtf', 'pwned']

Note that the algorithm is quite slow - it's prototypical.

Another approach using NLTK:

• http://web.archive.org/web/20160123234612/http://www.winwaed.com:80/blog/2012/03/13/segmenting-words-and-sentences/

As for your problem, you could just concatenate all string parts you have to get a single string and the run a segmentation algorithm on it.

Answer 2

Your goal is to improve text, not necessarily to make it perfect; so the approach you outline makes sense in my opinion. I would keep it simple and use a "greedy" approach: Start with the first fragment and stick pieces to it as long as the result is in the dictionary; if the result is not, spit out what you have so far and start over with the next fragment. Yes, occasionally you'll make a mistake with cases like the me thod, so if you'll be using this a lot, you could look for something more sophisticated. However, it's probably good enough.

Mainly what you require is a large dictionary. If you'll be using it a lot, I would encode it as a "prefix tree" (a.k.a. trie), so that you can quickly find out if a fragment is the start of a real word. The nltk provides a Trie implementation.

Since this kind of spurious word breaks are inconsistent, I would also extend my dictionary with words already processed in the current document; you may have seen the complete word earlier, but now it's broken up.

Answer 3

--Solution 1:

Lets think of these chunks in your sentence as beads on an abacus, with each bead consisting of a partial string, the beads can be moved left or right to generate the permutations. The position of each fragment is fixed between two adjacent fragments. In current case, the beads would be :

(more)(recen)(t)(ly)(the)(develop)(ment,)(wh)(ich)(is)(a)(po)(ten)(t)

This solves 2 subproblems:

a) Bead is a single unit,so We do not care about permutations within the bead i.e. permutations of "more" are not possible.

b) The order of the beads is constant, only the spacing between them changes. i.e. "more" will always be before "recen" and so on.

Now, generate all the permutations of these beads , which will give output like :

morerecentlythedevelopment,which is a potent
morerecentlythedevelopment,which is a poten t
morerecentlythedevelop ment, wh ich is a po tent
morerecentlythedevelop ment, wh ich is a po ten t
morerecentlythe development,whichisapotent

Then score these permutations based on how many words from your relevant dictionary they contain, most correct results can be easily filtered out. more recently the development, which is a potent will score higher than morerecentlythedevelop ment, wh ich is a po ten t

Code which does the permutation part of the beads:

import re

def gen_abacus_perms(frags):
    if len(frags) == 0:
        return []
    if len(frags) == 1:
        return [frags[0]]

    prefix_1 = "{0}{1}".format(frags[0],frags[1])
    prefix_2 = "{0} {1}".format(frags[0],frags[1])
    if len(frags) == 2:
        nres = [prefix_1,prefix_2]
        return nres

    rem_perms = gen_abacus_perms(frags[2:])
    res = ["{0}{1}".format(prefix_1, x ) for x in rem_perms] + ["{0} {1}".format(prefix_1, x ) for x in rem_perms] +  \
["{0}{1}".format(prefix_2, x ) for x in rem_perms] + ["{0} {1}".format(prefix_2 , x ) for x in rem_perms]
    return res



broken = "more recen t ly the develop ment, wh ich is a po ten t"
frags = re.split("\s+",broken)
perms = gen_abacus_perms(frags)
print("\n".join(perms))

demo:http://ideone.com/pt4PSt

---

--Solution#2:

I would suggest an alternate approach which makes use of text analysis intelligence already developed by folks working on similar problems and having worked on big corpus of data which depends on dictionary and grammar .e.g. search engines.

I am not well aware of such public/paid apis, so my example is based on google results.

Lets try to use google :

1. You can keep putting your invalid terms to Google, for multiple passes, and keep evaluating the results for some score based on your lookup dictionary. here are two relevant outputs by using 2 passes of your text :

This outout is used for a second pass :

Which gives you the conversion as ""more recently the development, which is a potent".

To verify the conversion, you will have to use some similarity algorithm and scoring to filter out invalid / not so good results.

One raw technique could be using a comparison of normalized strings using difflib.

>>> import difflib
>>> import re
>>> input = "more recen t ly the develop ment, wh ich is a po ten t "
>>> output = "more recently the development, which is a potent "
>>> input_norm = re.sub(r'\W+', '', input).lower()
>>> output_norm = re.sub(r'\W+', '', output).lower()
>>> input_norm
'morerecentlythedevelopmentwhichisapotent'
>>> output_norm
'morerecentlythedevelopmentwhichisapotent'
>>> difflib.SequenceMatcher(None,input_norm,output_norm).ratio()
1.0

Answer 4

I would recommend stripping away the spaces and looking for dictionary words to break it down into. There are a few things you can do to make it more accurate. To make it get the first word in text with no spaces, try taking the entire string, and going through dictionary words from a file (you can download several such files from http://wordlist.sourceforge.net/), the longest ones first, than taking off letters from the end of the string you want to segment. If you want it to work on a big string, you can make it automatically take off letters from the back so that the string you are looking for the first word in is only as long as the longest dictionary word. This should result in you finding the longest words, and making it less likely to do something like classify "asynchronous" as "a synchronous". Here is an example that uses raw input to take in the text to correct and a dictionary file called dictionary.txt:

dict = open("dictionary.txt",'r')                                #loads a file with a list of words to break string up into
words = raw_input("enter text to correct spaces on: ")
words = words.strip()                                            #strips away spaces
spaced = []                                                      #this is the list of newly broken up words
parsing = True                                                   #this represents when the while loop can end
while parsing:
    if len(words) == 0:                                          #checks if all of the text has been broken into words, if it has been it will end the while loop
        parsing = False
    iterating = True
    for iteration in range(45):                                  #goes through each of the possible word lengths, starting from the biggest
        if iterating == False:
            break
        word = words[:45-iteration]                              #each iteration, the word has one letter removed from the back, starting with the longest possible number of letters, 45
        for line in dict:
            line = line[:-1]                                     #this deletes the last character of the dictionary word, which will be a newline. delete this line of code if it is not a newline, or change it to [1:] if the newline character is at the beginning
            if line == word:                                     #this finds if this is the word we are looking for
                spaced.append(word)
                words = words[-(len(word)):]                     #takes away the word from the text list
                iterating = False
                break
print ' '.join(spaced)                                           #prints the output

If you want it to be even more accurate, you could try using a natural language parsing program, there are several available for python free online.

Answer 5

Here's something really basic:

chunks = []
for chunk in my_str.split():
    chunks.append(chunk)
    joined = ''.join(chunks)
    if is_word(joined):
        print joined,
        del chunks[:]

# deal with left overs
if chunks:
    print ''.join(chunks)

I assume you have a set of valid words somewhere that can be used to implement is_word. You also have to make sure it deals with punctuation. Here's one way to do that:

def is_word(wd):
    if not wd:
        return False
    # Strip of trailing punctuation. There might be stuff in front
    # that you want to strip too, such as open parentheses; this is
    # just to give the idea, not a complete solution.
    if wd[-1] in ',.!?;:':
        wd = wd[:-1]
    return wd in valid_words

Answer 6

You can iterate through a dictionary of words to find the best fit. Adding the words together when a match is not found.

def iterate(word,dictionary):
   for word in dictionary:
      if words in possibleWord:
        finished_sentence.append(words)
        added = True
      else:
        added = False
      return [added,finished_sentence]
sentence = "more recen t ly the develop ment, wh ich is a po ten t "
finished_sentence = ""
sentence = sentence.split()
for word in sentence:
  added,new_word = interate(word,dictionary)
  while True:   
    if added == False:
      word += possible[sentence.find(possibleWord)]
      iterate(word,dictionary)
    else:
      break
  finished_sentence.append(word)

This should work. For the variable dictionary, download a txt file of every single english word, then open it in your program.