Skip to content
Locale Lab

Chapter 15·Strings, Names, and Input·~7 min·

Character Encoding and Garbled Text (Mojibake)

Character encodings map integer Unicode code points to byte sequences. Decoding a UTF-8 byte stream with a legacy single-byte encoding table corrupts non-ASCII characters into mojibake.

UTF-8UTF-16mojibakecharset

Overview

Unicode assigns an integer code point to each character, while an encoding scheme maps that integer to bytes in memory or on disk. Decoding a UTF-8 byte stream with a single-byte table such as Windows-1252 produces corrupted text known as mojibake.

Core rule

Use UTF-8 across all storage and transport layers and declare the encoding at every read boundary. Mojibake occurs when a reader decodes bytes with the wrong character table.

Specification note: UTF-8 encodes é (U+00E9) as the two bytes 0xC3 0xA9. Decoding those bytes as Windows-1252 or ISO-8859-1 produces é.

Failure mode

UTF-8 and UTF-16 encode the same Unicode code points into different byte sequences. A raw byte stream does not contain metadata that declares which encoding produced it. When a sender writes UTF-8 bytes and a receiver decodes those bytes as Windows-1252 or ISO-8859-1, every non-ASCII character corrupts. The UTF-8 bytes for café (63 61 66 C3 A9) render as café. This corruption is called mojibake, and it must be fixed by declaring the UTF-8 encoding at the read boundary.

Stakeholder Teaching Aid & Novice Guide
Cross-functional alignment tool

When a system reads a UTF-8 text file using a legacy character encoding, accented letters and symbols transform into garbled symbols known as mojibake.

Business stakes & user risk

Displaying 'Renée' instead of 'Renée' or '€' instead of '€' destroys user trust, ruins customer invoices, and corrupts database records permanently.

Rule of thumb

Enforce UTF-8 everywhere: database tables, HTTP headers (charset=utf-8), API payloads, and CSV exports.

The Novice Trap vs. Production Standard

Contrast the intuitive assumption with the architecture required for global releases.

Naive mental model

“Text is just text; computers understand characters automatically without explicit encoding declarations.”

Naive approach: Missing charset declaration in headers and CSV exports

res.setHeader("Content-Type", "text/csv"); // Lacks charset declaration

Defect:Microsoft Excel on Windows opens the CSV with legacy ANSI encoding, turning 'François' into 'François'.

Production standard

Computers only store bytes. If a program assumes Windows-1252 or Latin-1 when reading multi-byte UTF-8 bytes, it misinterprets the bytes as two separate symbols.

Production standard: Explicit UTF-8 with BOM for spreadsheet compatibility

res.setHeader("Content-Type", "text/csv; charset=utf-8"); res.write("\uFEFF" + csvData);

Benefit:Guarantees that modern browsers, backend parsers, and legacy spreadsheet software display international characters correctly.

Cross-Functional Review Questions

Questions for product managers, designers, engineers, and testers to ask during sprint reviews.

For EngineeringQuestion #1

“Do all HTTP response headers, database connections, and file exporters explicitly declare UTF-8?”

What to look for: Content-Type: text/html; charset=utf-8 headers and database character sets configured to utf8mb4.
For QAQuestion #2

“Are customer CSV and PDF download reports tested with French, German, and Asian text in Microsoft Excel?”

What to look for: Verification that exported invoices open in desktop Excel without displaying mojibake characters.
For ProductQuestion #3

“How do third-party data imports and webhook payloads validate character encodings?”

What to look for: Ingestion pipelines configured to reject or transcode non-UTF-8 payloads before database writes.
For DesignQuestion #4

“Are custom email templates configured with UTF-8 meta tags?”

What to look for: <meta http-equiv='Content-Type' content='text/html; charset=utf-8' /> present in transactional email templates.

Mechanism

Unicode code points and byte encodings

Unicode assigns an abstract integer code point to each character, such as U+0041 for A or U+20AC for €. Files, network sockets, and database columns store raw bytes rather than code points. A character encoding defines the mapping between integer code points and byte sequences. UTF-8 and UTF-16 encode the same Unicode repertoire into different byte layouts.

UTF-8 uses variable-length byte sequences from 1 to 4 bytes. ASCII characters (U+0000 to U+007F) occupy 1 byte and match 7-bit ASCII identically. Latin letters with diacritics, Greek, Cyrillic, Hebrew, and Arabic occupy 2 bytes. Most Basic Multilingual Plane characters (including CJK ideographs) occupy 3 bytes, and supplementary characters above U+FFFF (such as emoji) occupy 4 bytes. UTF-16 uses 2 bytes for all Basic Multilingual Plane characters and 4-byte surrogate pairs above U+FFFF.

"A"    U+0041    UTF-8: 41             UTF-16: 0041
"€"    U+20AC    UTF-8: E2 82 AC       UTF-16: 20AC
"😀"   U+1F600   UTF-8: F0 9F 98 80    UTF-16: D83D DE00  (surrogate pair)
Code points encoded as UTF-8 and UTF-16 bytes.

Interactive demo

Guided exercises

  1. 1
    On the Bytes tab, clear the input field and type Hello. UTF-8 uses 5 bytes and UTF-16 uses 10 bytes. Pure ASCII text requires twice the byte storage in UTF-16 because every code unit is 16 bits.
  2. 2
    Click the Aé€😀 preset and inspect the table rows for é, €, and 😀. UTF-8 byte length increases from 2 bytes (é) to 3 bytes (€) to 4 bytes (😀). The emoji row shows a UTF-16 surrogate pair requiring two 16-bit code units.
  3. 3
    Inspect the UTF-16 column for 😀 (D83D DE00) on the Bytes tab. JavaScript strings use UTF-16 code units, so "😀".length evaluates to 2 even though the character is 1 Unicode code point and 1 grapheme cluster.
  4. 4
    Switch to the Mojibake tab with the default sample text. Step 3 displays café and € when decoded as Windows-1252, while the verification line below decodes the exact same bytes cleanly as UTF-8.
  5. 5
    On the Mojibake tab, change 'decode as' from Windows-1252 to ISO-8859-1 (Latin-1). The corrupted output for € changes. Windows-1252 assigns printable punctuation to bytes 0x80–0x9F, whereas ISO-8859-1 assigns invisible C1 control codes (shown as ⟨82⟩).
  6. 6
    On the Mojibake tab, replace the input with plain ASCII text such as Receipt total. No mojibake occurs. Bytes 0x00–0x7F map to identical characters in ASCII, UTF-8, Windows-1252, and ISO-8859-1.

Graphemes

4

what a user sees

Code points

4

[...str].length

UTF-8 bytes

10

on the wire / on disk

UTF-16 bytes

10

JS string in memory

CharCode pointUTF-8 bytesUTF-16 units
AU+004141(1B)0041
éU+00E9C3 A9(2B)00E9
€U+20ACE2 82 AC(3B)20AC
😀U+1F600F0 9F 98 80(4B)D83D DE00surrogate pair
An ASCII character is 1 byte in UTF-8 and 2 bytes in UTF-16. A code point above U+FFFF (most emoji) takes two UTF-16 units, a surrogate pair, so "😀".length is 2 in JavaScript.

Implementation rules

  • Standardize on UTF-8 across all storage and transport layers. Configure source files, HTTP payloads, message queues, and database columns (using utf8mb4 in MySQL and MariaDB) to store UTF-8.
  • Declare the character encoding at every boundary. Send Content-Type: ...; charset=utf-8 HTTP headers, place <meta charset="utf-8"> at the start of <head>, and pass explicit UTF-8 encoding arguments when reading files.
  • Never fix mojibake by editing corrupted strings by hand. The underlying UTF-8 bytes are usually intact. Configure the reader to decode the byte stream with the matching encoding.
  • Measure storage limits in bytes and UI limits in graphemes. Use new TextEncoder().encode(str).length to check database column byte limits, and use Intl.Segmenter to count user-perceived characters.

Operational context

Roles

EngineeringData InfrastructureQA

Workflow stages

  • ·Diagnosing corrupted characters in CSV exports or emails
  • ·Configuring database character sets and collations
  • ·Auditing file I/O and HTTP serialization pipelines

Production incident

Legacy MySQL tables configured with the `utf8` (alias `utf8mb3`) character set store a maximum of 3 bytes per character and truncate or reject 4-byte UTF-8 sequences such as emoji and supplementary CJK ideographs. Production MySQL and MariaDB databases must set both column definitions and client connection charsets to `utf8mb4`.

Terms in this chapter

Further reading

Related chapters