Unicode, UTF-8 and Strings That Lie

Interviewers ask "reverse a string" or "what is the length of this string?" and then slip in an emoji. They also ask why a database column rejected a name, why a file shows café, or why two strings that look the same are not ===. All of it is one topic: text is numbers, the numbers are stored in several layers, and each layer has its own idea of "one character".

This lesson goes layer by layer, with small programs that you can run. The string-searching algorithms in Lesson 7 assume you already know what a "character" is in your language. For the basic string problems, see DSA Lesson 9 and DSA Lesson 30. For locale-aware numbers and plurals with Intl, see Full-Stack Lesson 27.

Words used in this lesson

• A character set is a list of symbols, each with a number. An encoding is the way those numbers are written as bytes.
• Unicode is the character set that covers almost all writing systems. A code point is one number in it, written like U+1F600. Code points go from U+0000 to U+10FFFF.
• ASCII is the old 7-bit set (128 symbols: English letters, digits, punctuation). The first 128 Unicode code points are the same as ASCII.
• A code unit is the basic piece of an encoding. In UTF-16 it is 16 bits; in UTF-8 it is 8 bits (one byte).
• UTF-8 and UTF-16 are two encodings of Unicode. UTF-8 uses 1 to 4 bytes per code point. UTF-16 uses 1 or 2 units of 16 bits.
• A surrogate pair is two UTF-16 code units that together mean one code point above U+FFFF. A lone surrogate is one half of a pair with no partner; it is not valid text.
• A combining mark is a code point that attaches to the one before it (for example an accent).
• A grapheme cluster is what users see as one symbol ("user-perceived character"). Unicode defines the rules for where cluster boundaries are.
• A ZWJ (zero width joiner, U+200D) is an invisible code point that glues other code points into one symbol.
• Normalisation means converting text to a standard form, so that equal-looking text has equal code points. The forms are NFC, NFD, NFKC and NFKD.
• A locale is a language and region setting, such as sv or de-DE. Collation is the rule set for ordering strings in a locale.
• Mojibake is garbled text that appears when bytes are decoded with the wrong encoding.
• BOM (byte order mark) is the code point U+FEFF placed at the start of a file, sometimes used to signal the encoding.

The four layers

Take the word café. In your head it has four characters. Here it is at each layer:

Most of the time the numbers agree for English text, which is why bugs hide. They start to differ with accents, emoji, and non-Latin scripts. I counted four things for several strings. Each count comes from a different tool:

const enc = new TextEncoder();function report(s) {  const cps = [...s];  console.log(JSON.stringify(s).padEnd(14),    "length(UTF-16 units)=" + s.length,    "code points=" + cps.length,    "UTF-8 bytes=" + enc.encode(s).length,    "graphemes=" + [...new Intl.Segmenter("en", { granularity: "grapheme" }).segment(s)].length);}for (const s of ["A", "é", "€", "😀", "👨‍👩‍👧", "🇮🇳", "é", "हिन्दी", "👍🏽"]) report(s);

"A"            length(UTF-16 units)=1 code points=1 UTF-8 bytes=1 graphemes=1"é"            length(UTF-16 units)=1 code points=1 UTF-8 bytes=2 graphemes=1"€"            length(UTF-16 units)=1 code points=1 UTF-8 bytes=3 graphemes=1"😀"           length(UTF-16 units)=2 code points=1 UTF-8 bytes=4 graphemes=1"👨‍👩‍👧"     length(UTF-16 units)=8 code points=5 UTF-8 bytes=18 graphemes=1"🇮🇳"         length(UTF-16 units)=4 code points=2 UTF-8 bytes=8 graphemes=1"é"           length(UTF-16 units)=2 code points=2 UTF-8 bytes=3 graphemes=1"हिन्दी"       length(UTF-16 units)=6 code points=6 UTF-8 bytes=18 graphemes=2"👍🏽"         length(UTF-16 units)=4 code points=2 UTF-8 bytes=8 graphemes=1

Read the table carefully, because every row teaches something:

• "A": everything is 1. English fits in ASCII.
• "é" as one code point U+00E9: 1 unit, 1 code point, but 2 bytes in UTF-8.
• "€": 1 unit, 3 bytes.
• "😀": length is 2, because the code point U+1F600 is above U+FFFF, so UTF-16 needs a surrogate pair. It is 1 code point and 4 bytes.
• The family emoji: 8 units, 5 code points, 18 bytes, 1 symbol. Three people and two joiners.
• The flag 🇮🇳: two code points (the "regional indicator" letters I and N) that a font draws as one flag.
• The 7th row looks the same as the 2nd row, but it is e plus a combining acute accent (U+0301): 2 code points, 1 symbol. (This is why the two é rows print slightly differently in the padded output.)
• "👍🏽": a thumbs-up plus a skin-tone modifier, 2 code points, 1 symbol.
• The Hindi word has 6 code points and my Node counted 2 grapheme clusters. That depends on the Unicode version (newer versions have rules that keep conjuncts of Indic scripts together). Other engines may print another number. I only claim what this Node printed.

Code points and UTF-16

A code point is just a number. "😀".codePointAt(0) is 128512, which is 0x1F600. Unicode has room for code points up to 0x10FFFF. UTF-16 stores code points up to 0xFFFF as one 16-bit unit. For larger ones it subtracts 0x10000 from the number, giving a 20-bit value, and splits that into two halves of 10 bits. The high half is added to 0xD800 and the low half to 0xDC00. The range U+D800 to U+DFFF is therefore reserved for surrogates and never assigned to real characters.

console.log("😀".codePointAt(0), "😀".codePointAt(1), String.fromCodePoint(0x1F600), "\u{1F600}" === "😀");console.log([...enc.encode("€")].map(b => b.toString(16)), "😀".codePointAt(0).toString(16), "😀".charCodeAt(0).toString(16), "😀".charCodeAt(1).toString(16));

128512 56832 😀 true[ 'e2', '82', 'ac' ] 1f600 d83d de00

(enc is the TextEncoder from the first block.) codePointAt(0) read the whole pair and gave 128512. codePointAt(1) points into the middle of the pair and returns only the second unit, 56832 (0xDE00). charCodeAt(0) and charCodeAt(1) return the two halves: 0xD83D and 0xDE00. Both ways to write the emoji (\u{1F600} and the two-unit escape) are equal.

Iteration differs too: for (const ch of s) and [...s] step by code point, while s[i], charAt and split("") step by code unit.

console.log("😀".split(""), "😀".charAt(0) === "\ud83d");console.log(/^.$/.test("😀"), /^.$/u.test("😀"), /\p{Emoji}/u.test("😀"), /^\p{L}+$/u.test("héllo"), /^\w+$/.test("héllo"), "😀".match(/./g).length, "😀".match(/./gu).length);

[ '\ud83d', '\ude00' ] truefalse true true true false 2 1

Regular expressions have the same trap. Without the u flag, . matches one code unit, so /^.$/ fails on an emoji and "😀".match(/./g) finds 2 matches. With u the pattern works on code points, and you also get \p{...} classes such as \p{L} (any letter). Note that \w means only [A-Za-z0-9_], so /^\w+$/.test("héllo") is false. Use \p{L} for "letters" in real text.

How UTF-8 encodes a code point

UTF-8 uses 1 to 4 bytes. The first bits of the first byte say how long the sequence is. The other bytes always start with 10.

I printed the real bytes in binary with TextEncoder:

const enc = new TextEncoder();const show = s => console.log(JSON.stringify(s), [...enc.encode(s)].map(b => b.toString(2).padStart(8, "0")).join(" "));for (const s of ["A", "é", "€", "😀"]) show(s);

"A" 01000001"é" 11000011 10101001"€" 11100010 10000010 10101100"😀" 11110000 10011111 10011000 10000000

Check é (U+00E9, binary 1110 1001) against the 2-byte pattern: the 11 bits are 00011 101001, so the bytes are 110 00011 and 10 101001, which is 11000011 10101001. It matches.

Three useful design facts follow from the table (they are consequences of the pattern, which I checked in code):

• ASCII text is identical in UTF-8. An old ASCII file is already valid UTF-8.
• You can find the start of a character from anywhere. A byte that starts with 10 is a continuation; skip back until you hit one that does not.
• UTF-8 is self-checking. Random binary data is usually not valid UTF-8, so a decoder can detect errors.

To be sure I understood the table, I wrote the encoder and checked it against TextEncoder for every code point:

function utf8(cp) {  if (cp < 0x80) return [cp];  if (cp < 0x800) return [0xC0 | (cp >> 6), 0x80 | (cp & 0x3F)];  if (cp < 0x10000) return [0xE0 | (cp >> 12), 0x80 | ((cp >> 6) & 0x3F), 0x80 | (cp & 0x3F)];  return [0xF0 | (cp >> 18), 0x80 | ((cp >> 12) & 0x3F), 0x80 | ((cp >> 6) & 0x3F), 0x80 | (cp & 0x3F)];}const enc = new TextEncoder();let ok = true; for (let cp = 0; cp <= 0x10FFFF; cp++) { if (cp >= 0xD800 && cp <= 0xDFFF) continue; const r = [...enc.encode(String.fromCodePoint(cp))]; if (JSON.stringify(r) !== JSON.stringify(utf8(cp))) { ok = false; console.log("mismatch", cp); break; } }console.log("all code points match TextEncoder:", ok);

all code points match TextEncoder: true

The >> shifts and & masks are bit tricks, which Lesson 4 explains. The loop checked every code point except the 2,048 surrogates: my encoder agrees with the built-in one for all of them.

Lone surrogates and well-formed strings

JavaScript lets a string hold a lone surrogate, which is not valid Unicode text. UTF-8 cannot encode it. The encoder replaces it with the replacement character U+FFFD (bytes ef bf bd):

const lone = "\ud83d";console.log([...enc.encode(lone)], lone.isWellFormed(), JSON.stringify(lone), JSON.stringify("😀"));

[ 239, 191, 189 ] false "\ud83d" "😀"

isWellFormed() returns false for such a string, and toWellFormed() replaces lone surrogates with U+FFFD. JSON.stringify writes a lone surrogate as an escape like "\ud83d" and leaves a proper emoji as is. So a lone surrogate usually comes from cutting a string in the middle of a pair.

UTF-16 order is not code-point order

< on strings compares code units, so it can disagree with code-point order for characters above U+FFFF. The fullwidth tilde U+FF5E has a smaller code point than the emoji U+1F600, but a larger first code unit than the emoji's 0xD83D:

console.log("\uFF5E" < "😀", "\uFF5E".codePointAt(0) < "😀".codePointAt(0));

false true

Most code never notices, but a database that sorts by UTF-8 bytes can disagree with JavaScript's <.

Emoji, ZWJ sequences and flags

The family emoji is not a single code point. It is man, ZWJ, woman, ZWJ, girl:

const fam = "👨‍👩‍👧";console.log([...fam].map(c => "U+" + c.codePointAt(0).toString(16).toUpperCase()).join(" "));console.log([...fam].length, fam.length, enc.encode(fam).length);const flag = "🇮🇳"; console.log([...flag].map(c => "U+" + c.codePointAt(0).toString(16).toUpperCase()));const tone = "👍🏽"; console.log([...tone].map(c => "U+" + c.codePointAt(0).toString(16).toUpperCase()));

U+1F468 U+200D U+1F469 U+200D U+1F4675 8 18[ 'U+1F1EE', 'U+1F1F3' ][ 'U+1F44D', 'U+1F3FD' ]

The family is U+1F468 U+200D U+1F469 U+200D U+1F467: three emoji joined with the invisible ZWJ U+200D. If a font has no glyph for the sequence, you see three separate people. The flag is two regional indicator letters. The thumbs-up plus U+1F3FD is a skin-tone modifier. This is how emoji grow: new symbols are built from old code points, not by adding a new number.

A funny consequence of reversing code points: reversing the flag text breaks it.

const flag2 = "🇮🇳!"; console.log([...flag2].reverse().join(""));console.log([..."a😀b"].reverse().join(""));

!🇳🇮b😀a

Reversing by code points kept the plain emoji whole (b😀a), but the flag turned into a different flag: 🇳🇮 is the code for Nicaragua, not India. The reverse of a string that has combined symbols must go by grapheme clusters.

Grapheme clusters with Intl.Segmenter

Intl.Segmenter (a built-in) splits text into graphemes, words or sentences using the Unicode rules and, for words, locale data. I used it for all grapheme counts above.

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });const graphemes = s => Array.from(seg.segment(s), x => x.segment);console.log(graphemes("a👨‍👩‍👧é🇮🇳"));

[ 'a', '👨‍👩‍👧', 'é', '🇮🇳' ]

Four user-visible symbols, although the string has 10 code points: a is 1, the family is 5, e plus the accent is 2, and the flag is 2.

Word and sentence splitting:

const wseg = new Intl.Segmenter("en", { granularity: "word" });console.log([...wseg.segment("Hello, world! It's 3.5 o'clock.")].filter(x => x.isWordLike).map(x => x.segment));console.log([...new Intl.Segmenter("ja", { granularity: "word" }).segment("私は学生です")].map(x => x.segment));

[ 'Hello', 'world', "It's", '3.5', "o'clock" ][ '私', 'は', '学生', 'です' ]

The English word split keeps It's and 3.5 together, which split(" ") or a \W regex would not. Japanese has no spaces between words, and the segmenter found 学生 as one word using dictionary data. Word boundaries come from data, so treat the result as a good approximation and expect small differences between engines.

Normalisation: two spellings of the same letter

The letter é has two spellings in Unicode: one code point U+00E9 (precomposed) or e plus U+0301 (decomposed). They look the same, but they are different strings.

console.log("icu", process.versions.icu, "unicode", process.versions.unicode, "cldr", process.versions.cldr);const a = "é", b = "é";console.log(a, b, a === b, a.length, b.length, a.normalize("NFD") === b, b.normalize("NFC") === a, a.normalize("NFD").length);

icu 77.1 unicode 16.0 cldr 47.0é é false 1 2 true true 2

a === b is false. The lengths are 1 and 2. After normalize, they match. The four forms:

The "K" forms change the meaning of some characters on purpose:

const fi = "fi"; console.log(fi, fi.length, fi.normalize("NFKC"), fi.normalize("NFKC").length, "①".normalize("NFKC"), "Ａ".normalize("NFKC"));

fi 1 fi 2 1 A

The ligature fi (one code point) becomes the two letters fi; the circled digit ① becomes 1; the fullwidth Ａ becomes A. That is useful for search and for stopping fake usernames made of lookalikes, but it loses information, so do not use NFKC on text you must keep as it was.

Rule: normalise text to NFC when it enters your system (forms, files, APIs) and compare in that form.

Comparing and sorting

Plain sort() and < compare code units, which is not alphabetical order for most languages. localeCompare and Intl.Collator use the rules of a locale:

const words = ["z", "a", "Z", "ä", "é", "e", "E", "10", "2", "résumé", "resume"];console.log([...words].sort());console.log([...words].sort((x, y) => x.localeCompare(y)));console.log([...words].sort(new Intl.Collator("sv").compare));

[ '10', '2', 'E', 'Z', 'a', 'e', 'resume', 'résumé', 'z', 'ä', 'é' ][ '10', '2', 'a', 'ä', 'e', 'E', 'é', 'resume', 'résumé', 'z', 'Z' ][ '10', '2', 'a', 'e', 'E', 'é', 'resume', 'résumé', 'z', 'Z', 'ä' ]

(Node printed these over several lines; I joined them.) The default sort puts all capitals before all lower-case letters and puts ä and é after z. localeCompare puts ä next to a and é next to e. Swedish puts ä after z, as the Swedish alphabet does. So "alphabetical" depends on the locale. Notice also that "10" sorts before "2" in all three: they are compared as text. For numbers inside text, ask for numeric collation:

console.log(["file10", "file2", "file1"].sort(), ["file10", "file2", "file1"].sort(new Intl.Collator(undefined, { numeric: true }).compare));console.log("a".localeCompare("A"), "a".localeCompare("A", undefined, {sensitivity:"base"}), "a".localeCompare("á", "en", {sensitivity:"base"}), "a".localeCompare("á", "en", {sensitivity:"accent"}));console.log(a.localeCompare(b));

[ 'file1', 'file10', 'file2' ] [ 'file1', 'file2', 'file10' ]-1 0 0 -10

(a and b are the two spellings of é from the normalisation section.) sensitivity: "base" ignores case and accents (a equals A and á). "accent" ignores case but not accents. The last line shows that localeCompare treats the two spellings of é as equal (0), while === does not. For sorting a long list, build one Intl.Collator and reuse its compare; it is faster than calling localeCompare on every pair (I did not time this).

Mojibake: bytes decoded with the wrong encoding

A file is bytes. The reader must know the encoding. If it guesses wrong, you get mojibake. The classic case is UTF-8 bytes read as Latin-1 (each byte becomes one character):

const bytes = Buffer.from("café", "utf8");console.log(bytes, bytes.toString("utf8"), bytes.toString("latin1"));const garbled = bytes.toString("latin1");console.log(Buffer.from(garbled, "latin1").toString("utf8"));const saved = Buffer.from(garbled, "utf8");           // the garbled text saved again as UTF-8console.log(saved.length, saved.toString("latin1"));

<Buffer 63 61 66 c3 a9> café cafécafé7 café

The two bytes c3 a9 of é became the two characters Ã and ©. If you know this happened, you can reverse it: turn the text back into bytes with Latin-1 and decode them as UTF-8 (second line). But if the garbled text was saved again as UTF-8 (double encoding, last line), the damage is bigger: 7 bytes now, and a hidden control character appears. Fix the cause, not the output. Common causes are a missing charset=utf-8 in the HTTP header or HTML <meta charset>, a database connection or table with a Latin-1 setting, a CSV opened in a tool that assumes a legacy encoding, and a BOM that is not expected.

Invalid bytes and the BOM:

console.log(JSON.stringify(new TextDecoder().decode(new Uint8Array([0x63, 0x61, 0x66, 0xe9]))));try { new TextDecoder("utf-8", { fatal: true }).decode(new Uint8Array([0x63, 0xe9])); } catch (e) { console.log(e.name + ": " + e.message); }const bom = new Uint8Array([0xef, 0xbb, 0xbf, 0x68, 0x69]);console.log(JSON.stringify(new TextDecoder("utf-8", { ignoreBOM: true }).decode(bom)), JSON.stringify(new TextDecoder().decode(bom)));

"caf�"TypeError: The encoded data was not valid for encoding utf-8"hi" "hi"

By default, TextDecoder replaces bad bytes with U+FFFD (the black diamond �), so 0xe9 (Latin-1 é) is not valid UTF-8 and became �. With fatal: true it throws instead. A decoder removes the BOM by default; ignoreBOM: true keeps it, so the second string starts with an invisible U+FEFF and has length 3 (it prints as "hi" but it is not equal to "hi"). That invisible character is a known cause of "my JSON file or first CSV header does not match".

Buffer.byteLength and TextEncoder give the UTF-8 size. For €, Buffer.byteLength gives 3 in UTF-8 and 2 in utf16le. When a database or API says "max 255", ask whether it means characters, UTF-16 units or bytes.

Safe truncation

Cutting a string at a fixed place can break it. Three examples with "Hi 😀 there" and "I love 👨‍👩‍👧 and 🇮🇳 café":

const s = "Hi 😀 there";console.log(JSON.stringify(s.slice(0, 4)), s.slice(0, 4).isWellFormed(), s.slice(0, 5));try { encodeURIComponent(s.slice(0, 4)); } catch (e) { console.log(e.name + ": " + e.message); }console.log(JSON.stringify(s.slice(0,4).toWellFormed()));

"Hi \ud83d" false Hi 😀URIError: URI malformed"Hi �"

slice(0, 4) cut the emoji in half. The result is a lone surrogate: isWellFormed() is false, encodeURIComponent throws URIError, and when printed it shows as �. Even code points are not enough, as the family emoji shows. Cut at grapheme boundaries:

function truncateGraphemes(s, max, ell = "…") {  const g = graphemes(s);  return g.length <= max ? s : g.slice(0, max - 1).join("") + ell;}const text = "I love 👨‍👩‍👧 and 🇮🇳 café";console.log(truncateGraphemes(text, 10), "|", text.slice(0, 10), "|", [...text].slice(0, 10).join(""));

I love 👨‍👩‍👧 … | I love 👨‍ | I love 👨‍👩

(graphemes is the helper from the Segmenter section.) The grapheme version kept the family whole. slice(0, 10) ended with a man and a joiner. The code-point version ended with a man, a joiner and a woman, a two-person family that is not what the text said.

If the limit is in bytes (a database column, a header, a protocol), count bytes of whole clusters:

function truncateUtf8(s, maxBytes) {  const enc = new TextEncoder(); let out = "", used = 0;  for (const { segment } of seg.segment(s)) {    const n = enc.encode(segment).length;    if (used + n > maxBytes) break;    out += segment; used += n;  }  return out;}const t2 = truncateUtf8(text, 12);console.log(JSON.stringify(t2), new TextEncoder().encode(t2).length);const naive = new TextDecoder().decode(new TextEncoder().encode(text).slice(0, 12)); console.log(JSON.stringify(naive), naive.endsWith("�"));

"I love " 7"I love 👨�" true

With a 12-byte limit, the safe function kept "I love " (7 bytes) because the whole family emoji needs 18 bytes. The naive byte cut kept the man emoji and then half of the next character, which decoded to �. The safe version gives up some space to avoid broken text. Note that a cluster might be longer than your limit; then the output is shorter, and that is correct.

Interview questions

Key takeaways

• A "character" has four sizes: grapheme cluster, code point, UTF-16 unit and UTF-8 byte. Know which one your tool counts.
• length, [i], slice and split("") count UTF-16 units. for...of and [...s] go by code points. Intl.Segmenter goes by what the user sees.
• UTF-8 uses 1 to 4 bytes; ASCII is unchanged. TextEncoder and TextDecoder convert between strings and bytes.
• Emoji and many scripts are built from several code points (ZWJ, modifiers, flags, combining marks). Reversing or cutting them wrongly changes the meaning.
• Normalise to NFC before comparing. Use NFKC only for loose matching.
• Sort with Intl.Collator and a locale. Order depends on the language.
• Mojibake is a wrong decode. Fix the declaration of the encoding. Use fatal: true when you need errors, not silent replacement.
• Truncate at grapheme boundaries, and by whole clusters when the limit is in bytes.

What's next

Lesson 4: Binary, Endianness and Bit Tricks. You saw shifts and masks building UTF-8 bytes. Next we cover binary numbers, how bytes are ordered in memory, and the bit tricks that appear in interviews.