UTF-8 encoder and decoder

How it works

UTF-8 stores each Unicode code point in 1 to 4 bytes. Code points below 128 (ASCII) are one byte 0xxxxxxx, so plain English text is the same in ASCII and UTF-8. Larger code points use a lead byte that tells the length, 110xxxxx, 1110xxxx or 11110xxx, followed by continuation bytes 10xxxxxx; the bits of the code point fill the x places from left to right. For é = U+00E9 = 00011 101001 that gives 11000011 10101001, the bytes C3 A9.

The design has useful properties: a continuation byte can never be mistaken for the start of a character, so a decoder can resynchronize after an error, and sorting the bytes sorts the code points. An ASCII byte never appears inside a multibyte character, so searching for / or a quote in UTF-8 is safe.

Not every byte sequence is valid: a continuation byte without a lead, a sequence cut short, an overlong form (like C0 AF for /, which once let attackers sneak paths past filters), the surrogates U+D800 to U+DFFF, or anything above U+10FFFF. This decoder replaces each invalid sequence by U+FFFD, exactly as browsers do.