Why were the code points in the range of U+D800 to U+DFFF removed from the Unicode character set?

Question

I am learning about UTF-16 encoding, and I have read that if you want to represent code points in the range of U+10000 to U+10FFFF, then you have to use surrogate pairs, which are in the range of U+D800 to U+DFFF.

So let's say I want to encode the following code point: U+10123 (10000000100100011 in binary):

First I layout this sequence of bits:

110110xxxxxxxxxx 110111xxxxxxxxxx

Then I fill the places with the x with the binary format of the code point:

1101100001000000 1101110100100011 (D840 DD23 in hexadecimal)

---

I have also read that the code points in the range of U+D800 to U+DFFF were removed from the Unicode character set, but I don't understand why this range was removed!

I mean this range can be easily encoded in 4 bytes, for example the following is the UTF-16 encoded format of the U+D812 code point (1101100000010010 in binary):

1101100000110110 1101110000010010 (D836 DC12 in hexadecimal)

Note: I was using UTF-16 Big Endian in my examples.

Answer 1

I don't have an official source to back this up, but I believe it was to prevent confusion, so that you couldn't get a code sequence that could be interpreted as both valid UTF-16 and valid UCS-2. The loss of 2048 code points was nothing compared to the addition of 1048576 new ones.

Answer 2

Codepoints U+D800 - U+DFFF are reserved exclusively¹ for use with UTF-16. Since they are not in the range of U+10000 - U+10FFFF, UTF-16 would not encode them individually using surrogate pairs, so it would be ambiguous (and illegal²) for these individual codepoints to appear un-encoded in a UTF-16 sequence.

---

Per the Unicode.org UTF-16 FAQ:

¹: Q: What are surrogates?

A: Surrogates are code points from two special ranges of Unicode values, reserved for use as the leading, and trailing values of paired code units in UTF-16. Leading, also called high, surrogates are from D800₁₆ to DBFF₁₆, and trailing, or low, surrogates are from DC00₁₆ to DFFF₁₆. They are called surrogates, since they do not represent characters directly, but only as a pair.

²: Q: Are there any 16-bit values that are invalid?

A: Unpaired surrogates are invalid in UTFs. These include any value in the range D800₁₆ to DBFF₁₆ not followed by a value in the range DC00₁₆ to DFFF₁₆, or any value in the range DC00₁₆ to DFFF₁₆ not preceded by a value in the range D800₁₆ to DBFF₁₆.