Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What is a relatively simple way to determine the probability that a sentence is in English?

I have a number of strings (collections of characters) that represent sentences in different languages, say:

Hello, my name is George.

Das brot ist gut.

... etc.

I want to assign each of them scores (from 0 .. 1) indicating the likelihood that they are English sentences. Is there an accepted algorithm (or Python library) from which to do this?

Note: I don't care if the grammar of the English sentence is perfect.

like image 782
sdasdadas Avatar asked Feb 08 '13 04:02

sdasdadas


1 Answers

A bayesian classifier would be a good choice for this task:

>>> from reverend.thomas import Bayes
>>> g = Bayes()    # guesser
>>> g.train('french','La souris est rentrée dans son trou.')
>>> g.train('english','my tailor is rich.')
>>> g.train('french','Je ne sais pas si je viendrai demain.')
>>> g.train('english','I do not plan to update my website soon.')

>>> print g.guess('Jumping out of cliffs it not a good idea.')
[('english', 0.99990000000000001), ('french', 9.9999999999988987e-005)]

>>> print g.guess('Demain il fera très probablement chaud.')
[('french', 0.99990000000000001), ('english', 9.9999999999988987e-005)]
like image 122
Raymond Hettinger Avatar answered Sep 23 '22 20:09

Raymond Hettinger