Why are there duplicate characters in Unicode?

Question

I can see some duplicate characters in Unicode. For example, the character 'C' can be represented by the code points U+0043 and U+0421. Why is this so?

Answer 1

As others have noted, your main fallacy here is confusing the Latin and Cyrillic scripts and some glyphs therein (namely C (U+0043 LATIN CAPITAL LETTER C) and С (U+0421 CYRILLIC CAPITAL LETTER ES) ). There are many such character pairs that look alike but are different characters. You will find plenty among Latin, Greek and Cyrillic, for example. Most of the time they only work in either uppercase or lowercase, though.

However, there are in fact duplicates, sometimes intentionally so. For example, the entire (ASCII) Latin alphabet is represented twice again in the 'Halfwidth and Fullwidth Forms' Unicode block between U+FF00 and U+FFEF. There are other such examples, though, most notably in the mathematical alphabet section on Plane 1 where there are three or four more Latin alphabets present.

There are other things that are in fact the same character but at different code points. For example, there is µ (U+00B5 MICRO SIGN) and μ (U+03BC GREEK SMALL LETTER MU). Those are usually linked by decomposition.

Unicode deals with an abstract concept called code point. The code point unambiguously defines a character and its script or group. It says nothing about how the corresponding glyph in a font would be rendered (which may vary wildly for Latin already). It also does not define how this code point is represented in a file or memory (i.e. as a byte sequence). That's a job for one of the Unicode Transformation Formats.

What is reason to have a similar looking character in two languages with different code points?

The main points of Unicode here are:

• Compatibility to every previously existing character encoding. This has to ensure that there are one-to-one mappings for every character that was used in an encoding to a direct equivalent as a Unicode code point.
• Faithfully and accurately represent every script that is used nowadays, later expanded to other scripts that were in use and need to be stored in computer systems.

So there is a very strong incentive to keep scripts separate and not try to map characters according to their appearance. Appearance can be tricky anyway. Take for example the Cyrillic letter 'т', which appears like a smaller upper-case Latin 'T' here. However, the usual way it is rendered when italicized: 'т' looks like a lower-case Latin 'm'. You really don't want to map such characters by appearance.

Answer 2

If you look at the U+0400 to U+04FF code chart you'll find that U+0421 is a Cyrillic capital leter "es". It may look like a Latin C, but it's a different logical character.

Answer 3

The letters LOOK the same, but are very different. U+0043 is the latin letter C, but U+0421 is cyrillic С (which corresponds to an S in the latin alphabet).

Because of their unrelated meanings, separate code points are required to prevent casing and sorting algorithms from being horribly context-sensitive - you would suddenly have to guess what language you were dealing with.

Answer 4

For the same reason that 0 and O look the same (in most mono-spaced fonts), but are encoded differently - they mean different things.