Lesson 15·Unit 5 · Strings, names, and input·Bytes, encodings, and charset mismatch·Lv 201·~7 min
Character encoding & mojibake
A string is code points; a file is bytes. The encoding is the map between them, and there is more than one map. Get the map wrong on read and café becomes café, reproduced here in your own browser.
By the end
- ·Explain why one character costs a different number of bytes in UTF-8 than in UTF-16.
- ·Trace how mojibake happens, and why the bytes on disk are usually fine.
- ·Name the fix (declare the encoding where you read the file) and know where that declaration goes.
The problem
Text has to become bytes to be stored or sent, and bytes have to become text to be read. UTF-8 and UTF-16 are two different ways to write the same Unicode code points as bytes, and nothing inside a byte stream says which one was used. When the writer says UTF-8 and the reader assumes Windows-1252, every non-ASCII character is misread: café is stored correctly and displayed as café. This mojibake is the single most common "why are there weird characters" bug in localization.
How it works
A string is code points, a file is bytes
Text lives in your program as a sequence of Unicode code points: abstract numbers like U+0041 for A or U+20AC for the euro sign. A file, a network packet, or a database column holds bytes, not code points. An encoding is the map between the two. Unicode defines more than one: UTF-8 and UTF-16 store the same code points with different byte layouts.
UTF-8 is variable-width: an ASCII character is one byte. A is a single 0x41, byte-for-byte identical to 1960s ASCII. Accented Latin letters take two bytes, most other scripts three, and code points above U+FFFF (most emoji) four. UTF-16 uses a two-byte code unit for everything in the common range, so it spends two bytes even on plain ASCII. That overhead on ASCII-heavy text is a main reason UTF-8 became the web's default.
"A" U+0041 UTF-8: 41 UTF-16: 0041
"€" U+20AC UTF-8: E2 82 AC UTF-16: 20AC
"😀" U+1F600 UTF-8: F0 9F 98 80 UTF-16: D83D DE00 (surrogate pair)How it works
Mojibake is a label bug
Nothing inside a stream of bytes announces which encoding produced it. Something must tell the reader: an HTTP charset, a <meta charset> tag, a database setting, or a convention. When the writer used UTF-8 and the reader assumes an 8-bit legacy encoding like Windows-1252, the reader misreads every multi-byte character one byte at a time.
That is how café becomes café. Its UTF-8 bytes are 63 61 66 C3 A9. Read as Windows-1252, C3 is à and A9 is ©, so the two-byte é splits into two Western-European glyphs. The bytes are intact, only mislabeled. Declare the right encoding on the read side. Do not hand-edit the garbled text.
See it yourself
Try these, in order
Each step triggers a specific failure you should recognize on sight.
- 1On the Bytes tab, clear the field and type Hello. UTF-8 bytes = 5, UTF-16 bytes = 10. Pure ASCII costs twice as much in UTF-16, because every code unit is two bytes.
- 2Click the Aé€😀 preset and read the table rows for é, €, and 😀. UTF-8 uses 2, 3, then 4 bytes as the code point climbs. The emoji row carries the tag 'surrogate pair': two UTF-16 units for one code point.
- 3Still on Bytes, note the emoji's UTF-16 column shows two units (D83D DE00). That pair is why "😀".length is 2 in JavaScript. .length counts UTF-16 code units, and this emoji occupies two.
- 4Switch to the Mojibake tab and leave the default text. Step 3 shows café and €, while the line below confirms the same bytes decode correctly as UTF-8. The bytes are fine. Only the read-side label was wrong.
- 5On the Mojibake tab, change 'decode as' from Windows-1252 to ISO-8859-1 (Latin-1). The euro result changes: the two encodings disagree in the 0x80–0x9F byte range, so the same wrong bytes garble differently.
- 6On the Mojibake tab, replace the text with plain ASCII like Receipt total. No corruption appears. ASCII bytes are identical across UTF-8 and Latin-1. That is why mojibake only hits the non-ASCII characters.
Graphemes
4
what a user sees
Code points
4
[...str].length
UTF-8 bytes
10
on the wire / on disk
UTF-16 bytes
10
JS string in memory
| Char | Code point | UTF-8 bytes | UTF-16 units |
|---|---|---|---|
| A | U+0041 | 41(1B) | 0041 |
| é | U+00E9 | C3 A9(2B) | 00E9 |
| € | U+20AC | E2 82 AC(3B) | 20AC |
| 😀 | U+1F600 | F0 9F 98 80(4B) | D83D DE00surrogate pair |
"😀".length is 2 in JavaScript.If you remember one thing
A string is code points. A file is bytes. The encoding is the map between them, and mojibake is what you get when the reader uses a different map than the writer.
What to do about it
- Use UTF-8 everywhere: files, HTTP responses, database columns and connections (
utf8mb4in MySQL), message queues. One encoding end to end removes the mismatch that causes mojibake. - Declare the encoding, don't guess it. Send
Content-Type: …; charset=utf-8, put<meta charset="utf-8">first in<head>, and open files with an explicit UTF-8 encoding rather than the platform default. - Never "repair" mojibake by hand-editing the garbled glyphs. The bytes are usually fine; fix the label on the read side, or you push the mismatch downstream.
- Remember that
"😀".lengthis 2 in JavaScript: UTF-16 stores astral code points as surrogate pairs, so byte and length budgets computed on.lengthundercount real storage and miscount user-visible characters.
Use this with
Stakeholders
Moments
- ·Debugging 'weird characters' in exports, emails, or CSV imports
- ·Setting database and connection charset before a launch
- ·Reviewing file / HTTP encoding in a code review
Field note
The most common production version of this is not a file. It is a database. A MySQL column declared `utf8` (a three-byte subset) or a connection that negotiates `latin1` stores correct UTF-8 text as mojibake. The damage is invisible until someone reads it back. Set columns and the connection to `utf8mb4` before the first insert. A migration after the fact means decoding the corruption byte by byte.
Quick check
3 questions · pass at 2+
Question 1/3
How many bytes does the letter "A" take in UTF-8 versus UTF-16?
Question 2/3
"café" is saved as UTF-8, then a reader decodes those bytes as Windows-1252. What appears, and whose fault is it?
Question 3/3
Why does "😀".length equal 2 in JavaScript?