Detecting multibyte character sequences?

Question

I'm writing a parser which parses UTF-8 strings. Characters outside of the ASCII range can only occur inside of string literals, which begin and end with ' or ". The rest of the language may only contain ASCII characters, so I can simply return an error if I find a byte outside the ASCII range.

The problem I can't seem to figure out is, when I encounter a non-ASCII character inside of a string literal, how can I detect how many bytes to skip for that character? My concern is that if a multi-byte character contains a ' or " as one of the bytes, my parser would end the string literal early.

Perhaps a shorter way to ask this is, if I encounter a byte in the 0x80-0xFF range, how can I detect how many bytes are in that character in a UTF-8 encoded string?

I'm writing this parser in C but I suspect that doesn't matter.

Answer 1

My concern is that if a multi-byte character contains a ' or " as one of the bytes, my parser would end the string literal early.

Ah, this is your misunderstanding. The brilliance of UTF-8 is that this cannot happen. In UTF-8, the byte 0x27 can only mean APOSTROPHE. It can never be part of a multi-byte sequence. This is because continuation bytes begin with the high bit set to 1.

A major design goal of UTF-8 is that existing and naïve ASCII implementations will work identically when parsing UTF-8 streams, even if the stream includes non-ASCII bytes. You can safely parse for " and continue to accumulate bytes until you reach " (and use \ to escape internal "), and never have to worry about whether there are multi-byte characters involved with UTF-8. ASCII parsers do not need to understand UTF-8 or perform any UTF-8 decoding in order to work correctly.

Beyond that, if you decide you really do want to know the answer to your question, the first byte's number of leading 1 bits tells you the length, with the exception that zero 1s is "1 byte" and one 1 is "continuation".

0x00 - 0x7F -> 1 byte
0x80 - 0xBF -> (continuation)
0xC0 - 0xDF -> 2 bytes
0xE0 - 0xEF -> 3 bytes
0xF0 - 0xF7 -> 4 bytes

You can also just keep scanning along until you find something in the range 0x00-0x7F.