How to detect if a file has a UTF-8 BOM in Bash?

Question

I'm trying to write a script that will automatically remove UTF-8 BOMs from a file. I'm having trouble detecting whether the file has one in the first place or not. Here is my code:

function has-bom {     # Test if the file starts with 0xEF, 0xBB, and 0xBF     head -c 3 "$1" | grep -P '\xef\xbb\xbf'     return $? } 

For some reason, head seems to be ignoring the BOM in front of the file. As an example, running this

printf '\xef\xbb\xbf' > file head -c 3 file 

won't print anything.

I tried looking for an option in head --help that would let me work around this, but no luck. Is there anything I can do to make this work?

Answer 1

First, let's demonstrate that head is actually working correctly:

$ printf '\xef\xbb\xbf' >file $ head -c 3 file  $ head -c 3 file | hexdump -C 00000000  ef bb bf                                          |...| 00000003 

Now, let's create a working function has_bom. If your grep supports -P, then one option is:

$ has_bom() { head -c3 "$1" | LC_ALL=C grep -qP '\xef\xbb\xbf'; } $ has_bom file && echo yes yes 

Currently, only GNU grep supports -P.

Another option is to use bash's $'...':

$ has_bom() { head -c3 "$1" | grep -q $'\xef\xbb\xbf'; } $ has_bom file && echo yes yes 

ksh and zsh also support $'...' but this construct is not POSIX and dash does not support it.

Notes:

1. The use of an explicit return $? is optional. The function will, by default, return with the exit code of the last command run.

2. I have used the POSIX form for defining functions. This is equivalent to the bash form but gives you one less problem to deal with if you ever have to run the function under another shell.

3. bash does accept the use of the character - in a function name but this is a controversial feature. I replaced it with _ which is more widely accepted. (For more on this issue, see this answer.)

4. The -q option to grep makes it quiet, meaning that it still sets a proper exit code but it does not send any characters to stdout.