Find non-ASCII characters in a text file and convert them to their Unicode equivalent

Question

I am importing .txt file from a remote server and saving it to a database. I use a .Net script for this purpose. I sometimes notice a garbled word/characters (Ullerهkersvنgen) inside the files, which makes a problem while saving to the database.

I want to filter all such characters and convert them to unicode before saving to the database.

Note: I have been through many similar posts but had no luck.

Your help in this context will be highly appreciated.

Thanks.

Answer 1

Assuming your script does know the correct encoding of your text snippet than that should be the regular expression to find all Non-ASCII charactres:

[^\x00-\x7F]+

see here: https://stackoverflow.com/a/20890052/1144966 and https://stackoverflow.com/a/8845398/1144966

Also, the base-R tools package provides two functions to detect non-ASCII characters:

tools::showNonASCII()
tools::showNonASCIIfile()