Language detection with data in PostgreSQL

Question

I have a table in PostgreSQL where a column is a text. I need a library or tool that can identify the language of each text for a test purpose.

There is no need for a PostgreSQL code because I'm having problems to install languages, but any language that can connect to the database, retrieve the texts and identify it arewelcome.

I used Lingua::Identify suggested in the answers right in the Perl script, it worked, but the results are not precise.

The texts I want to identify comes from the web and most are in portuguese, but Lingua::Identify is classifying much as french, italian and spanish that are similar languages.

I need something more precise.

I added the java and r tags because are the languages I'm using in the system and solution using they will be easy to implement, but solutions in any language are welcome.

Answer 1

You can use PL/Perl (CREATE FUNCTION langof(text) LANGUAGEplperluAS ...) with Lingua::Identify CPAN module.

Perl script:

#!/usr/bin/perl
use Lingua::Identify qw(langof);
undef $/;
my $textstring = <>;  ## warning - slurps whole file to memory
my $a = langof( $textstring );    # gives the most probable language
print "$a\n";

And the function:

create or replace function langof( text ) returns varchar(2)
immutable returns null on null input
language plperlu as $perlcode$
    use Lingua::Identify qw(langof);
    return langof( shift );
$perlcode$;

Works for me:

filip@filip=# select langof('Pójdź, kiń-że tę chmurność w głąb flaszy');
 langof
--------
 pl
(1 row)

Time: 1.801 ms

PL/Perl on Windows

PL/Perl language libary (plperl.dll) comes preinstalled in latest Windows installer of postgres.

But to use PL/Perl, you need Perl interpreter itself. Specifically, Perl 5.14 (at the time of this writing). Most common installer is ActiveState, but it's not free. Free one comes from StrawberryPerl. Make sure you have PERL514.DLL in place.

After installing Perl, login to your postgres database and try to run

CREATE LANGUAGE plperlu;

Language identification library

If quality is your concern, you have some options: You can improve Lingua::Identify yourself (it's open source) or you could try another library. I found this one, which is commercial but looks promising.