Dummy's guide to Unicode

Question

Could anyone give me a concise definitions of

• Unicode
• UTF7
• UTF8
• UTF16
• UTF32
• Codepages
• How they differ from Ascii/Ansi/Windows 1252

I'm not after wikipedia links or incredible detail, just some brief information on how and why the huge variations in Unicode have come about and why you should care as a programmer.

Answer 1

If you want a really brief introduction: Unicode in 5 Minutes

Or if you are after one-liners:

• Unicode: a mapping of characters to integers ("code points") in the range 0 through 1,114,111; covers pretty much all written languages in use
• UTF7: an encoding of code points into a byte stream with the high bit clear; in general do not use
• UTF8: an encoding of code points into a byte stream where each character may take one, two, three or four bytes to represent; should be your primary choice of encoding
• UTF16: an encoding of code points into a word stream (16-bit units) where each character may take one or two words (two or four bytes) to represent
• UTF32: an encoding of code points into a stream of 32-bit units where each character takes exactly one unit (four bytes); sometimes used for internal representation
• Codepages: a system in DOS and Windows whereby characters are assigned to integers, and an associated encoding; each covers only a subset of languages. Note that these assignments are generally different than the Unicode assignments
• ASCII: a very common assignment of characters to integers, and the direct encoding into bytes (all high bit clear); the assignment is a subset of Unicode, and the encoding a subset of UTF-8
• ANSI: a standards body
• Windows 1252: A commonly used codepage; it is similar to ISO-8859-1, or Latin-1, but not the same, and the two are often confused

Why do you care? Because without knowing the character set and encoding in use, you don't really know what characters a given byte stream represents. For example, the byte 0xDE could encode

• Þ (LATIN CAPITAL LETTER THORN)
• fi (LATIN SMALL LIGATURE FI)
• ή (GREEK SMALL LETTER ETA WITH TONOS)
• or 13 other characters, depending on the encoding and character set used.