Python, file(1) - Why are the numbers [7,8,9,10,12,13,27] and range(0x20, 0x100) used for determining text vs binary file

Question

Regarding a solution for determining whether a file is binary or text in python, the answerer uses:

textchars = bytearray([7,8,9,10,12,13,27]) + bytearray(range(0x20, 0x100))

and then uses .translate(None, textchars) to remove (or replace by nothing) all such characters in a file read in as binary.

The answerer also argues that this choice of numbers is "based on file(1) behaviour" (for what's text and what's not). What is so significant about these numbers is determining text files from binary?

Answer 1

They represent the most-common codepoints for printable text, plus newlines, spaces and carriage returns and the like. ASCII is covered up to 0x7F, and standards like Latin-1 or Windows Codepage 1251 use the remaining 128 bytes for accented characters, etc.

You'd expect text to only use those codepoints. Binary data would use all codepoints in the range 0x00-0xFF; e.g. a text file will probably not use \x00 (NUL) or \x1F (Unit Separator in the ASCII standard).

It is a heuristic at best, though. Some text files may still try and use C0 control codes outside those 7 characters explicitly named, and I'm sure binary data exists that happens to not include the 25 byte values not included in the textchars string.

The author of the range probably based it on the text_chars table from the file command. It marks bytes as non-text, ASCII, Latin-1 or non-ISO extended ASCII, and includes documentation on why those codepoints where chosen:

/*
 * This table reflects a particular philosophy about what constitutes
 * "text," and there is room for disagreement about it.
 *
 * [....]
 *
 * The table below considers a file to be ASCII if all of its characters
 * are either ASCII printing characters (again, according to the X3.4
 * standard, not isascii()) or any of the following controls: bell,
 * backspace, tab, line feed, form feed, carriage return, esc, nextline.
 *
 * I include bell because some programs (particularly shell scripts)
 * use it literally, even though it is rare in normal text.  I exclude
 * vertical tab because it never seems to be used in real text.  I also
 * include, with hesitation, the X3.64/ECMA-43 control nextline (0x85),
 * because that's what the dd EBCDIC->ASCII table maps the EBCDIC newline
 * character to.  It might be more appropriate to include it in the 8859
 * set instead of the ASCII set, but it's got to be included in *something*
 * we recognize or EBCDIC files aren't going to be considered textual.
 *
 * [.....]
 */

Interestingly enough, that table excludes 0x7F, which the code you found does not.