Correctly decoding zip entry file names -- CP437, UTF-8 or?

Question

I recently wrote a zip file I/O library called zipzap, but I'm struggling with correctly decoding zip entry file names from arbitrary zip files.

Now, the PKWARE spec states:

D.1 The ZIP format has historically supported only the original IBM PC character encoding set, commonly referred to as IBM Code Page 437...

D.2 If general purpose bit 11 is unset, the file name and comment should conform to the original ZIP character encoding. If general purpose bit 11 is set, the filename and comment must support The Unicode Standard, Version 4.1.0 or greater using the character encoding form defined by the UTF-8 storage specification...

which means that conforming zip files encode file names as CP437, unless the EFS bit is set, in which case the file names are UTF-8.

Unfortunately it seems that a lot of zip tools either don't set the EFS bit correctly (e.g. Mac CLI, GUI zip) or use some other encoding, typically the default system one (e.g. WinZip?). If you know how WinZip, 7-Zip, Info-Zip, PKZIP, Java JAR/Zip, .NET zip, dotnetzip, etc. encode file names and what they set their "version made by" field to when zipping, please tell me.

In particular, Info-Zip tries this when unzipping:

• File system = MS-DOS (0) => CP437
  • except: version = 2.5, 2.6, 4.0 => ISO 8859-1
• File system = HPFS (6) => CP437
• File system = NTFS (10) and version = 5.0 => CP437
• otherwise, ISO 8859-1

If I want to support inspecting or extracting from arbitrary zip files and make a reasonable attempt at the file name encoding without the EFS flag, what can I look for?

Answer 1

At the moment situation is as following:

• most of Windows implementations use DOS (OEM) encoding
• Mac OS zip utility uses utf-8, but it doesn't set utf-8 bit flags
• *nix zip utilities silently uses system encoding

So the only way is to check if filename contains something like utf-8 characters (check description of utf8 encoding - first byte should be 110xxxxx, second - 10xxxxxx for 2-bytes encoded chars). If it is correct utf8 string - use utf8 encoding. If not - fall back to OEM/DOS encoding.

Answer 2

The only way to determine if the filename is encoded as UTF-8 without using the EFS flag is to check to see if the high order bit is set in one of the characters. That could possibly mean that the character is UTF-8 encoded. However, it could still be the other way as there are some characters in CP437 that have the high order bit set and aren't meant to be decoded as UTF-8.

I would stick to the PKWARE app note specification and not hack in a solution that tries to conform to every known zip application in existence.