Chapter 15·Strings, Names, and Input·~7 min·
Character Encoding and Garbled Text (Mojibake)
Character encodings map integer Unicode code points to byte sequences. Decoding a UTF-8 byte stream with a legacy single-byte encoding table corrupts non-ASCII characters into mojibake.
Overview
Unicode assigns an integer code point to each character, while an encoding scheme maps that integer to bytes in memory or on disk. Decoding a UTF-8 byte stream with a single-byte table such as Windows-1252 produces corrupted text known as mojibake.
Core rule
Use UTF-8 across all storage and transport layers and declare the encoding at every read boundary. Mojibake occurs when a reader decodes bytes with the wrong character table.
Specification note: UTF-8 encodes é (U+00E9) as the two bytes 0xC3 0xA9. Decoding those bytes as Windows-1252 or ISO-8859-1 produces é.
Failure mode
UTF-8 and UTF-16 encode the same Unicode code points into different byte sequences. A raw byte stream does not contain metadata that declares which encoding produced it. When a sender writes UTF-8 bytes and a receiver decodes those bytes as Windows-1252 or ISO-8859-1, every non-ASCII character corrupts. The UTF-8 bytes for café (63 61 66 C3 A9) render as café. This corruption is called mojibake, and it must be fixed by declaring the UTF-8 encoding at the read boundary.
When a system reads a UTF-8 text file using a legacy character encoding, accented letters and symbols transform into garbled symbols known as mojibake.
Business stakes & user risk
Displaying 'Renée' instead of 'Renée' or '€' instead of '€' destroys user trust, ruins customer invoices, and corrupts database records permanently.
Rule of thumb
Enforce UTF-8 everywhere: database tables, HTTP headers (charset=utf-8), API payloads, and CSV exports.
The Novice Trap vs. Production Standard
Contrast the intuitive assumption with the architecture required for global releases.
“Text is just text; computers understand characters automatically without explicit encoding declarations.”
Naive approach: Missing charset declaration in headers and CSV exports
res.setHeader("Content-Type", "text/csv"); // Lacks charset declarationDefect:Microsoft Excel on Windows opens the CSV with legacy ANSI encoding, turning 'François' into 'François'.
Computers only store bytes. If a program assumes Windows-1252 or Latin-1 when reading multi-byte UTF-8 bytes, it misinterprets the bytes as two separate symbols.
Production standard: Explicit UTF-8 with BOM for spreadsheet compatibility
res.setHeader("Content-Type", "text/csv; charset=utf-8"); res.write("\uFEFF" + csvData);Benefit:Guarantees that modern browsers, backend parsers, and legacy spreadsheet software display international characters correctly.
Cross-Functional Review Questions
Questions for product managers, designers, engineers, and testers to ask during sprint reviews.
“Do all HTTP response headers, database connections, and file exporters explicitly declare UTF-8?”
“Are customer CSV and PDF download reports tested with French, German, and Asian text in Microsoft Excel?”
“How do third-party data imports and webhook payloads validate character encodings?”
“Are custom email templates configured with UTF-8 meta tags?”
Mechanism
Unicode code points and byte encodings
Unicode assigns an abstract integer code point to each character, such as U+0041 for A or U+20AC for €. Files, network sockets, and database columns store raw bytes rather than code points. A character encoding defines the mapping between integer code points and byte sequences. UTF-8 and UTF-16 encode the same Unicode repertoire into different byte layouts.
UTF-8 uses variable-length byte sequences from 1 to 4 bytes. ASCII characters (U+0000 to U+007F) occupy 1 byte and match 7-bit ASCII identically. Latin letters with diacritics, Greek, Cyrillic, Hebrew, and Arabic occupy 2 bytes. Most Basic Multilingual Plane characters (including CJK ideographs) occupy 3 bytes, and supplementary characters above U+FFFF (such as emoji) occupy 4 bytes. UTF-16 uses 2 bytes for all Basic Multilingual Plane characters and 4-byte surrogate pairs above U+FFFF.
"A" U+0041 UTF-8: 41 UTF-16: 0041
"€" U+20AC UTF-8: E2 82 AC UTF-16: 20AC
"😀" U+1F600 UTF-8: F0 9F 98 80 UTF-16: D83D DE00 (surrogate pair)Interactive demo
Guided exercises
- 1On the Bytes tab, clear the input field and type Hello. UTF-8 uses 5 bytes and UTF-16 uses 10 bytes. Pure ASCII text requires twice the byte storage in UTF-16 because every code unit is 16 bits.
- 2Click the Aé€😀 preset and inspect the table rows for é, €, and 😀. UTF-8 byte length increases from 2 bytes (é) to 3 bytes (€) to 4 bytes (😀). The emoji row shows a UTF-16 surrogate pair requiring two 16-bit code units.
- 3Inspect the UTF-16 column for 😀 (D83D DE00) on the Bytes tab. JavaScript strings use UTF-16 code units, so "😀".length evaluates to 2 even though the character is 1 Unicode code point and 1 grapheme cluster.
- 4Switch to the Mojibake tab with the default sample text. Step 3 displays café and € when decoded as Windows-1252, while the verification line below decodes the exact same bytes cleanly as UTF-8.
- 5On the Mojibake tab, change 'decode as' from Windows-1252 to ISO-8859-1 (Latin-1). The corrupted output for € changes. Windows-1252 assigns printable punctuation to bytes 0x80–0x9F, whereas ISO-8859-1 assigns invisible C1 control codes (shown as ⟨82⟩).
- 6On the Mojibake tab, replace the input with plain ASCII text such as Receipt total. No mojibake occurs. Bytes 0x00–0x7F map to identical characters in ASCII, UTF-8, Windows-1252, and ISO-8859-1.
Graphemes
4
what a user sees
Code points
4
[...str].length
UTF-8 bytes
10
on the wire / on disk
UTF-16 bytes
10
JS string in memory
| Char | Code point | UTF-8 bytes | UTF-16 units |
|---|---|---|---|
| A | U+0041 | 41(1B) | 0041 |
| é | U+00E9 | C3 A9(2B) | 00E9 |
| € | U+20AC | E2 82 AC(3B) | 20AC |
| 😀 | U+1F600 | F0 9F 98 80(4B) | D83D DE00surrogate pair |
"😀".length is 2 in JavaScript.Implementation rules
- Standardize on UTF-8 across all storage and transport layers. Configure source files, HTTP payloads, message queues, and database columns (using
utf8mb4in MySQL and MariaDB) to store UTF-8. - Declare the character encoding at every boundary. Send
Content-Type: ...; charset=utf-8HTTP headers, place<meta charset="utf-8">at the start of<head>, and pass explicit UTF-8 encoding arguments when reading files. - Never fix mojibake by editing corrupted strings by hand. The underlying UTF-8 bytes are usually intact. Configure the reader to decode the byte stream with the matching encoding.
- Measure storage limits in bytes and UI limits in graphemes. Use
new TextEncoder().encode(str).lengthto check database column byte limits, and useIntl.Segmenterto count user-perceived characters.
Operational context
Roles
Workflow stages
- ·Diagnosing corrupted characters in CSV exports or emails
- ·Configuring database character sets and collations
- ·Auditing file I/O and HTTP serialization pipelines
Production incident
Legacy MySQL tables configured with the `utf8` (alias `utf8mb3`) character set store a maximum of 3 bytes per character and truncate or reject 4-byte UTF-8 sequences such as emoji and supplementary CJK ideographs. Production MySQL and MariaDB databases must set both column definitions and client connection charsets to `utf8mb4`.