I'm learning the C language on Linux now and I've came across a little weird situation.
As far as I know, the standard C's char
data type is ASCII, 1 byte (8 bits). It should mean, that it can hold only ASCII characters.
In my program I use char input[]
, which is filled by getchar
function like this pseudocode:
char input[20]; int z, i; for(i = 0; i < 20; i++) { z = getchar(); input[i] = z; }
The weird thing is that it works not only for ASCII characters, but for any character I imagine, such as @&@{čřžŧ¶'`[łĐŧđж←^€~[←^ø{&}čž
on the input.
My question is - how is it possible? It seems to be one of many beautiful exceptions in C, but I would really appreciate explanation. Is it a matter of OS, compiler, hidden language's additional super-feature?
Thanks.
Unicode uses two encoding forms: 8-bit and 16-bit, based on the data type of the data that is being that is being encoded. The default encoding form is 16-bit, where each character is 16 bits (2 bytes) wide. Sixteen-bit encoding form is usually shown as U+hhhh, where hhhh is the hexadecimal code point of the character.
It can represent all 1,114,112 Unicode characters. Most C code that deals with strings on a byte-by-byte basis still works, since UTF-8 is fully compatible with 7-bit ASCII. Characters usually require fewer than four bytes. String sort order is preserved.
Unicode Character “C” (U+0043)
As far as I know, the standard C's char data type is ASCII, 1 byte (8 bits).
There is no magic here - The C language gives you acess to the raw bytes, as they are stored in the computer memory. If your terminal is using utf-8 (which is likely), non-ASCII chars take more than one byte in memory. When you display then again, is our terminal code which converts these sequences into a single displayed character.
Just change your code to print the strlen
of the strings, and you will see what I mean.
To properly handle utf-8 non-ASCII chars in C you have to use some library to handle them for you, like glib, qt, or many others.
ASCII is a 7 bit character set. In C normally represented by an 8 bit char. If highest bit in an 8 bit byte is set, it is not an ASCII character.
Also notice that you are not guaranteed ASCII as base, tho many ignore other scenarios. If you want to check if a "primitive" byte is a alpha character you can in other words not, when taking heed to all systems, say:
is_alpha = (c > 0x40 && c < 0x5b) || (c > 0x60 && c < 0x7b);
Instead you'll have to use ctype.h
and say:
isalpha(c);
Only exception, AFAIK, is for numbers, on most tables at least, they have contiguous values.
Thus this works;
char ninec = '9'; char eightc = '8'; int nine = ninec - '0'; int eight = eightc - '0'; printf("%d\n", nine); printf("%d\n", eight);
But this is not guaranteed to be 'a':
alhpa_a = 0x61;
Systems not based on ASCII, i.e. using EBCDIC; C on such a platform still runs fine but here they (mostly) use 8 bits instead of 7 and i.e. A
can be coded as decimal 193
and not 65
as it is in ASCII.
For ASCII however; bytes having decimal 128 - 255, (8 bits in use), is extended, and not part of the ASCII set. I.e. ISO-8859 uses this range.
What is often done; is also to combine two or more bytes to one character. So if you print two bytes after each other that is defined as say, utf8 0xc3 0x98
== Ø, then you'll get this character.
This again depends on which environment you are in. On many systems/environments printing ASCII values give same result across character sets, systems etc. But printing bytes > 127 or double byted characters gives a different result depending on local configuration.
I.e.:
Mr. A running the program gets
Jasŋ€
While Mr. B gets
Jasπß
This is perhaps especially relevant to the ISO-8859 series and Windows-1252 of single byte representation of extended characters, etc.
10
. That is; the first byte in UTF-8, if not ASCII, tells how many bytes this character has. You could also say ASCII characters say no more bytes follow - because highest bit is 0.
I.e if file interpreted as UTF-8:
fgetc(c); if c < 128, 0x80, then ASCII if c == 194, 0xC2, then one more byte follow, interpret to symbol if c == 226, 0xE2, then two more byte follows, interpret to symbol ...
As an example. If we look at one of the characters you mention. If in an UTF-8 terminal:
$ echo -n "č" | xxd
Should yield:
0000000: c48d ..
In other words "č" is represented by the two bytes 0xc4 and 0x8d. Add -b to the xxd command and we get the binary representation of the bytes. We dissect them as follows:
___ byte 1 ___ ___ byte 2 ___ | | | | 0xc4 : 1100 0100 0x8d : 1000 1101 | | | +-- all "follow" bytes starts with 10, rest: 00 1101 | + 11 -> 2 bits set = two byte symbol, the "bits set" sequence end with 0. (here 3 bits are used 110) : rest 0 0100 Rest bits combined: xxx0 0100 xx00 1101 => 00100001101 \____/ \_____/ | | | +--- From last byte +------------ From first byte
This give us: 00100001101 2 = 26910 = 0x10D => Uncode codepoint U+010D == "č".
This number can also be used in HTML as č
== č
Common for this and lots of other code systems is that an 8 bit byte is the base.
Often it is also a question about context. As an example take GSM SMS, with ETSI GSM 03.38/03.40 (3GPP TS 23.038, 3GPP 23038). There we also find an 7bit character table, 7-bits GSM default alphabet, but instead of storing them as 8 bits they are stored as 7 bits1. This way you can pack more characters into a given number of bytes. Ie standard SMS 160 characters becomes 1280 bits or 160 bytes as ASCII and 1120 or 140 bytes as SMS.
1 Not without exception, (it is more to the story).
I.e. a simple example of bytes saved as septets (7bit) C8329BFD06 in SMS UDP format to ASCII:
_________ 7 bit UDP represented | +--- Alphas has same bits as ASCII as 8 bit hex '0.......' C8329BFDBEBEE56C32 1100100 d * Prev last 6 bits + pp 1 | | | | | | | | +- 00 110010 -> 1101100 l * Prev last 7 bits | | | | | | | +--- 0 1101100 -> 1110010 r * Prev 7 + 0 bits | | | | | | +----- 1110010 1 -> 1101111 o * Last 1 + prev 6 | | | | | +------- 101111 10 -> 1010111 W * Last 2 + prev 5 | | | | +--------- 10111 110 -> 1101111 o * Last 3 + prev 4 | | | +----------- 1111 1101 -> 1101100 l * Last 4 + prev 3 | | +------------- 100 11011 -> 1101100 l * Last 5 + prev 2 | +--------------- 00 110010 -> 1100101 e * Last 6 + prev 1 +----------------- 1 1001000 -> 1001000 H * Last 7 bits '------' | +----- GSM Table as binary
And 9 bytes "unpacked" becomes 10 characters.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With