How to read files with different encodings using Awk?

Question

How can I correctly read files in encodings other than UTF8 in Awk?

I have a file in Hebrew/Windows-1255 encoding. A simple {print $0} awk prints stuff like �. how can I make it read correctly?

Answer 1

awk itself doesn't have any support for handling different encodings. It will honor the locale specified in the environment, but your best bet is to transcode the input to the proper encoding before handing it off to awk.

-f is the format you want to convert from, -t is the target format, and -c skips over any invalid characters which prematurely terminate iconv's operation. Of course --help will give more details.

iconv -c -f cp1255 -t utf8 somefile | awk ...