Skip to content
Locale Lab

Chapter 14·Strings, Names, and Input·~8 min·

Unicode Normalization, Graphemes, and Case

JavaScript strings index 16-bit UTF-16 code units rather than user-perceived characters. Canonical normalization, grapheme cluster segmentation, and locale-independent case folding prevent string comparison and truncation bugs.

Unicodeencodingnormalizationcase folding

Overview

A single user-perceived character can consist of multiple Unicode code points and UTF-16 code units. String comparison fails when equivalent characters use different normalization forms, and `.slice()` corrupts multi-unit sequences. Normalization, `Intl.Segmenter`, and locale-aware case mapping prevent string corruption.

Core rule

Normalize strings to NFC at system boundaries, segment user-visible characters by grapheme cluster, and pin the locale when changing case.

Specification note: The family emoji 👨‍👩‍👧‍👦 has a JavaScript .length of 11, consists of 7 Unicode code points, and renders as 1 grapheme cluster.

Failure mode

JavaScript strings store sequences of UTF-16 code units. The string "café" occupies four code units when é is precomposed (NFC) and five code units when é is decomposed into e plus a combining acute accent (NFD). Both forms render identically on screen but compare unequal with ===. Unnormalized database keys allow duplicate accounts and fail lookup queries.

Case conversion also depends on locale rules. Turkish maps ASCII I to dotless ı (U+0131). Lowercasing security tokens or field names with a host-default locale on a Turkish server corrupts ASCII identifiers, as seen in Spring Framework CVE-2024-38820.

Stakeholder Teaching Aid & Novice Guide
Cross-functional alignment tool

A visible character on screen does not equal one byte or one string unit in memory. Measuring string length with .length truncates accented names and emoji.

Business stakes & user risk

Valid user names like 'Renée' or compound emoji like family avatars fail input validation or get truncated mid-character, corrupting user profiles.

Rule of thumb

Measure user-visible character length using Intl.Segmenter with granularity: 'grapheme', not string.length.

The Novice Trap vs. Production Standard

Contrast the intuitive assumption with the architecture required for global releases.

Naive mental model

“A string's .length property counts the number of letters a user typed.”

Naive approach: String length check

if (username.length > 10) throw new Error("Too long");

Defect:A user entering 'José' with combining accents or an emoji avatar fails validation despite seeing fewer than 5 characters.

Production standard

JavaScript .length measures 16-bit UTF-16 code units. Accented characters with combining marks count as 2 units, and emoji can count as 11 or more.

Production standard: Grapheme cluster segmentation

const count = [...new Intl.Segmenter(locale, { granularity: "grapheme" }).segment(username)].length;

Benefit:Accurately counts user-perceived letters regardless of Unicode composition, accent marks, or emoji sequences.

Cross-Functional Review Questions

Questions for product managers, designers, engineers, and testers to ask during sprint reviews.

For EngineeringQuestion #1

“Do database character limits and form validation count grapheme clusters rather than UTF-16 units?”

What to look for: Input validation schemas utilizing grapheme segmenters or Unicode normalization (NFC) before length checks.
For DesignQuestion #2

“Do character count indicators in textareas display human-perceived character counts?”

What to look for: Social post composers and bio editors allowing users to type emoji without burning 10+ characters per icon.
For QAQuestion #3

“Are input fields tested with complex Unicode strings: accented letters, Korean Hangul, and family emoji?”

What to look for: Test cases ensuring characters like 'é' (e with combining acute) are not sliced in half upon submission.
For ProductQuestion #4

“Are search queries normalized to Unicode NFC before database matching?”

What to look for: Search infrastructure matching precomposed 'é' with decomposed 'e + acute' seamlessly.

Mechanism

Code units, code points, and grapheme clusters

JavaScript strings expose three distinct length levels. The .length property counts 16-bit UTF-16 code units. Iterating a string with [...str] counts Unicode code points (U+0000 to U+10FFFF), where any supplementary code point above U+FFFF occupies a two-unit surrogate pair. Intl.Segmenter with { granularity: 'grapheme' } counts extended grapheme clusters (UAX #29), which correspond to single user-perceived characters. For example, the family emoji 👨‍👩‍👧‍👦 is 1 grapheme cluster joined by Zero Width Joiners (U+200D), 7 Unicode code points, and 11 UTF-16 code units. Slicing a string by UTF-16 index can split a surrogate pair or a ZWJ sequence and corrupt the text.

Canonical equivalence creates a second mismatch when comparing strings. The word café can be encoded in Canonical Composition (NFC) as 4 code points ending with precomposed é (U+00E9), or in Canonical Decomposition (NFD) as 5 code points ending with e (U+0065) plus a combining acute accent (U+0301). Both forms render identically, but === returns false because their code point sequences differ. Call .normalize('NFC') on external input before comparing, indexing, or storing strings.

Case mapping is also locale-sensitive. English maps I (U+0049) to i (U+0069), whereas Turkish and Azerbaijani map I to dotless ı (U+0131) and dotted İ (U+0130) to i. Use locale-independent case folding for internal identifiers and security blocklists, and pass the active locale to .toLocaleUpperCase(locale) and .toLocaleLowerCase(locale) for display text.

Interactive demo

Guided exercises

  1. 1
    On the Turkish I tab, type ID (plain ASCII) into the input. English lowercases `ID` to `id`, whereas Turkish (`tr`) lowercases `ID` to dotless `ıd` (`U+0131 U+0064`), causing an ASCII equality check against `id` to return `false`.
  2. 2
    Open the NFC vs NFD tab and read the code-point dumps. Precomposed `café` (NFC) has 4 code points and decomposed `café` (NFD) has 5 code points. Strict equality (`===`) returns `false` until both strings are normalized with `.normalize('NFC')`.
  3. 3
    Still on that tab, scroll to “Normalize your own string”. Read the three rows for the seeded text, then replace it with any accented word or a filename pasted from Finder. The seeded decomposed string has `.length = 5` and ends in `U+0301`, whereas its NFC form has `.length = 4`.
  4. 4
    On the Emoji length tab, pick the family emoji 👨‍👩‍👧‍👦. `str.length` reports `11` UTF-16 code units, `[...str].length` reports `7` code points, and `Intl.Segmenter` reports `1` grapheme cluster.
  5. 5
    Pick the Scotland flag 🏴󠁧󠁢󠁳󠁣󠁴󠁿. The flag sequence combines a black flag base with invisible tag characters (`gbsct`) and a cancel tag: `1` visible glyph, `7` code points, and `14` UTF-16 code units.

The same string, lowercased in English and in Turkish. Try İD, STRAİT, or the literal word ID.

toLocaleLowerCase("en-US")

i̇d

toLocaleLowerCase("tr-TR")

id

en === tr ?

false

Security implication

A blocklist rejects the field names "id" and "userid" after lowercasing with a bare toLocaleLowerCase(), which follows the host locale. On a Turkish device, ID becomes ıd (dotless ı) and passes the filter; the default sample İD becomes id and is blocked.

Locale-independent toLowerCase() blocks "İD": no · Bare toLocaleLowerCase() on a Turkish device blocks it: yes

CVE-2024-38820 in Spring has this shape. GitHub's password reset had a related case-mapping bug with the dotless ı.

The fix: Never make security or routing decisions with a bare toLocaleLowerCase(). Use toLowerCase(), which the spec defines as locale-independent (root-locale case mapping), or pin the locale: toLocaleLowerCase("en-US").

Implementation rules

  • Normalize all incoming strings to NFC at system boundaries. Call input.normalize("NFC") before storing, hashing, or comparing text.
  • Keep programmatic case conversion locale-independent. Use locale-invariant case folding (Locale.ROOT in Java, .toLowerCase() in JavaScript) for identifiers and security checks. Use .toLocaleLowerCase(locale) only for human-facing display text.
  • Segment user-visible characters with Intl.Segmenter. Use new Intl.Segmenter(locale, { granularity: "grapheme" }) when counting or truncating user input so multi-code-point emoji and combining marks stay intact.
  • Apply UTS #39 confusable detection to unique identifiers. Check usernames and domain handles against Unicode confusable skeletons to block cross-script homograph spoofing.

Operational context

Roles

EngineeringSecurityQA

Workflow stages

  • ·Designing username and identifier validation
  • ·Reviewing string truncation and character limit logic
  • ·Security audits of authentication and allowlist checks

Production incident

Spring Framework CVE-2024-38820 resulted from locale-sensitive case conversion. DataBinder lowercased disallowed field names using Java's default-locale toLowerCase(). On hosts running a Turkish default locale, uppercase 'I' became dotless 'ı', allowing disallowed parameter names to bypass the security blocklist.

Spring: CVE-2024-38820 ↗

Terms in this chapter

Further reading

Related chapters