Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to check whether a file is valid UTF-8?

I'm processing some data files that are supposed to be valid UTF-8 but aren't, which causes the parser (not under my control) to fail. I'd like to add a stage of pre-validating the data for UTF-8 well-formedness, but I've not yet found a utility to help do this.

There's a web service at W3C which appears to be dead, and I've found a Windows-only validation tool that reports invalid UTF-8 files but doesn't report which lines/characters to fix.

I'd be happy with either a tool I can drop in and use (ideally cross-platform), or a ruby/perl script I can make part of my data loading process.

like image 801
Ian Dickinson Avatar asked Sep 22 '08 14:09

Ian Dickinson


People also ask

How can I tell if a file is UTF-8?

Open the file in Notepad. Click 'Save As...'. In the 'Encoding:' combo box you will see the current file format. Yes, I opened the file in notepad and selected the UTF-8 format and saved it.

How do I know if a file is UTF-8 or UTF-16?

There are a few options you can use: check the content-type to see if it includes a charset parameter which would indicate the encoding (e.g. Content-Type: text/plain; charset=utf-16 ); check if the uploaded data has a BOM (the first few bytes in the file, which would map to the unicode character U+FEFF - 2 bytes for ...

How do I check if a file is UTF-8 encoded in Python?

Could be simpler by using only one line: codecs. open("path/to/file", encoding="utf-8", errors="strict").

How do I know if a file is UTF-8 or not Linux?

To verify if a file passes an encoding such as ascii, iso-8859-1, utf-8 or whatever then a good solution is to use the 'iconv' command.


2 Answers

You can use GNU iconv:

$ iconv -f UTF-8 your_file -o /dev/null; echo $? 

Or with older versions of iconv, such as on macOS:

$ iconv -f UTF-8 your_file > /dev/null; echo $? 

The command will return 0 if the file could be converted successfully, and 1 if not. Additionally, it will print out the byte offset where the invalid byte sequence occurred.

Edit: The output encoding doesn't have to be specified, it will be assumed to be UTF-8.

like image 182
Torsten Marek Avatar answered Oct 19 '22 18:10

Torsten Marek


Use python and str.encode|decode functions.

>>> a="γεια" >>> a '\xce\xb3\xce\xb5\xce\xb9\xce\xb1' >>> b='\xce\xb3\xce\xb5\xce\xb9\xff\xb1' # note second-to-last char changed >>> print b.decode("utf_8") Traceback (most recent call last):   File "<stdin>", line 1, in <module>   File "/usr/local/lib/python2.5/encodings/utf_8.py", line 16, in decode     return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 6: unexpected code byte 

The exception thrown has the info requested in its .args property.

>>> try: print b.decode("utf_8") ... except UnicodeDecodeError, exc: pass ... >>> exc UnicodeDecodeError('utf8', '\xce\xb3\xce\xb5\xce\xb9\xff\xb1', 6, 7, 'unexpected code byte') >>> exc.args ('utf8', '\xce\xb3\xce\xb5\xce\xb9\xff\xb1', 6, 7, 'unexpected code byte') 
like image 28
tzot Avatar answered Oct 19 '22 18:10

tzot