I have a set of concatenated word and i want to split them into arrays
For example :
split_word("acquirecustomerdata")
=> ['acquire', 'customer', 'data']
I found pyenchant
, but it's not available for 64bit windows.
Then i tried to split each string into sub string and then compare them to wordnet to find a equivalent word. For example :
from nltk import wordnet as wn
def split_word(self, word):
result = list()
while(len(word) > 2):
i = 1
found = True
while(found):
i = i + 1
synsets = wn.synsets(word[:i])
for s in synsets:
if edit_distance(s.name().split('.')[0], word[:i]) == 0:
found = False
break;
result.append(word[:i])
word = word[i:]
print(result)
But this solution is not sure and is too long. So I'm looking for your help.
Thank you
Check - Word Segmentation Task from Norvig's work.
from __future__ import division
from collections import Counter
import re, nltk
WORDS = nltk.corpus.brown.words()
COUNTS = Counter(WORDS)
def pdist(counter):
"Make a probability distribution, given evidence from a Counter."
N = sum(counter.values())
return lambda x: counter[x]/N
P = pdist(COUNTS)
def Pwords(words):
"Probability of words, assuming each word is independent of others."
return product(P(w) for w in words)
def product(nums):
"Multiply the numbers together. (Like `sum`, but with multiplication.)"
result = 1
for x in nums:
result *= x
return result
def splits(text, start=0, L=20):
"Return a list of all (first, rest) pairs; start <= len(first) <= L."
return [(text[:i], text[i:])
for i in range(start, min(len(text), L)+1)]
def segment(text):
"Return a list of words that is the most probable segmentation of text."
if not text:
return []
else:
candidates = ([first] + segment(rest)
for (first, rest) in splits(text, 1))
return max(candidates, key=Pwords)
print segment('acquirecustomerdata')
#['acquire', 'customer', 'data']
For better solution than this you can use bigram/trigram.
More examples at : Word Segmentation Task
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With