Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

iconv any encoding to UTF-8

Tags:

I am trying to point iconv to a directory and all files will be converted UTF-8 regardless of the current encoding

I am using this script but you have to specify what encoding you are going FROM. How can I make it autdetect the current encoding?

dir_iconv.sh

#!/bin/bash  ICONVBIN='/usr/bin/iconv' # path to iconv binary  if [ $# -lt 3 ] then   echo "$0 dir from_charset to_charset"   exit fi  for f in $1/* do   if test -f $f   then     echo -e "\nConverting $f"     /bin/mv $f $f.old     $ICONVBIN -f $2 -t $3 $f.old > $f   else     echo -e "\nSkipping $f - not a regular file";   fi done 

terminal line

sudo convert/dir_iconv.sh convert/books CURRENT_ENCODING utf8 
like image 382
Blainer Avatar asked Mar 22 '12 15:03

Blainer


People also ask

What is iconv function?

The iconv() function converts a sequence of characters in one character encoding to a sequence of characters in another character encoding.

What is UTF-8 and UTF-16?

UTF-8 encodes a character into a binary string of one, two, three, or four bytes. UTF-16 encodes a Unicode character into a string of either two or four bytes. This distinction is evident from their names. In UTF-8, the smallest binary representation of a character is one byte, or eight bits.


2 Answers

Maybe you are looking for enca:

Enca is an Extremely Naive Charset Analyser. It detects character set and encoding of text files and can also convert them to other encodings using either a built-in converter or external libraries and tools like libiconv, librecode, or cstocs.

Currently it supports Belarusian, Bulgarian, Croatian, Czech, Estonian, Hungarian, Latvian, Lithuanian, Polish, Russian, Slovak, Slovene, Ukrainian, Chinese, and some multibyte encodings independently on language.

Note that in general, autodetection of current encoding is a difficult process (the same byte sequence can be correct text in multiple encodings). enca uses heuristics based on the language you tell it to detect (to limit the number of encodings). You can use enconv to convert text files to a single encoding.

like image 60
Michal Kottman Avatar answered Sep 24 '22 21:09

Michal Kottman


You can get what you need using standard gnu utils file and awk. Example:

file -bi .xsession-errors gives me: "text/plain; charset=us-ascii"

so file -bi .xsession-errors |awk -F "=" '{print $2}' gives me "us-ascii"

I use it in scripts like so:

CHARSET="$(file -bi "$i"|awk -F "=" '{print $2}')"  if [ "$CHARSET" != utf-8 ]; then   iconv -f "$CHARSET" -t utf8 "$i" -o outfile fi 
like image 29
Julian Hughes Avatar answered Sep 24 '22 21:09

Julian Hughes