Why Your Text Turns Into Question Marks: A Practical Guide to Character Encoding

Mojibake is diagnostic — the shape of the corruption tells you the mistake. Plus why MySQL utf8 is not UTF-8, and what %2520 in a URL means.

Every developer eventually meets a string of text that has turned into garbage — curly apostrophes replaced by ’, accented names rendered as ä, or a name in the database that came back as a row of question marks. These are not random glitches. They are a specific, diagnosable class of bug, and the pattern of corruption tells you exactly what went wrong.

Bytes do not know what they mean

A computer stores text as numbers. An encoding is the agreement about which number means which character. Nothing in a file records that agreement — the bytes are just bytes, and the encoding is an assumption made by whatever reads them.

ASCII, the original agreement, covered 128 characters: English letters, digits, punctuation, control codes. Enough for American English and nothing else. Everyone else needed more, and what followed was decades of incompatible national codepages, each claiming the same byte values for different characters. Byte 233 meant é in one codepage and something else entirely in another. A document was only readable if the reader guessed the same codepage the writer used.

Unicode fixed the disagreement by assigning every character in every writing system a unique number, called a code point. UTF-8 is the way those numbers are written as bytes, and it won for two excellent reasons: it is backwards compatible with ASCII, so plain English text is byte-identical and every old tool still works, and it is variable width, so common characters stay compact while the rarer ones expand to two, three or four bytes as needed.

The practical rule that follows is simple and worth being dogmatic about: use UTF-8 everywhere, and declare it explicitly, in the HTTP header, the HTML meta charset, the database, the connection, and the editor. Most encoding bugs are a mismatch at exactly one of those points.

Reading the corruption

The good news is that mojibake is diagnostic. The shape of the mess tells you which mistake was made.

’ where an apostrophe should be is the signature case. A curly apostrophe in UTF-8 is three bytes. Read those three bytes as if each were a separate Windows-1252 character and you get exactly those three symbols. So this pattern means UTF-8 text was read as Windows-1252 — almost always a missing or wrong charset declaration on the reading side. The text is intact; it is being misinterpreted.

A single replacement character, the black diamond with a question mark, means the opposite kind of failure. The reader was using UTF-8 correctly but hit a byte sequence that is not valid UTF-8, so it substituted the official "I cannot decode this" character. The original byte is gone.

Plain question marks are the worst outcome, because they indicate real data loss. Something converted text into an encoding that had no representation for those characters and substituted a question mark for each. That is destructive and irreversible — the information no longer exists. This is what happens when Unicode text is written into a database column that cannot hold it.

The MySQL trap

That last case has a famous instance worth knowing specifically. In MySQL, the encoding named utf8 is not UTF-8. It is a three-byte-maximum implementation, created before anyone expected four-byte characters to matter. It handles most European and Asian text and cannot store anything needing four bytes.

Emoji need four bytes. So do many less common characters, including a good deal of historical script. Store an emoji into a MySQL utf8 column and, depending on configuration, you get either an error or silent truncation.

The correct encoding is utf8mb4 — the same standard, with the full four-byte range. Any new MySQL schema should use it, and legacy schemas using utf8 are carrying a latent bug that surfaces the first time a user puts an emoji in their display name.

Excel, CSV and the byte order mark

The other encoding problem most people meet is exporting a CSV, opening it in Excel, and finding every accented character mangled — while the same file looks perfect in a text editor.

Excel, on Windows, has historically assumed the system codepage rather than UTF-8 when opening a CSV by double-click. The file is fine; Excel is guessing wrong.

The workaround is a byte order mark: three specific bytes at the start of the file that signal UTF-8. Excel recognises them and switches. This is why CSV export code often deliberately writes a BOM that would otherwise be unnecessary.

The catch is that the BOM is not universally welcome. Many Unix tools treat it as content, so a BOM can appear as stray characters at the start of the first field, break a shell script, or confuse a parser expecting the file to begin with a specific header. If you are generating CSV for Excel users, include it; if you are generating it for a machine, do not.

Encoding in URLs, and double encoding

Percent-encoding is a separate layer that sits on top of all this. A URL can only contain a restricted set of ASCII characters, so anything else is written as a percent sign followed by the hexadecimal value of each byte — and those bytes are the UTF-8 bytes of the character.

The bug to recognise here is double encoding. Encode a space once and you get %20. Encode that result again and the percent sign itself gets encoded, producing %2520. If you see %2520 in a URL or a log, something in the chain encoded an already-encoded value — usually a redirect, or two layers of code each being careful.

The related mistake is choosing the wrong scope. Encoding a whole URL must leave the structural characters alone, because the slashes and the question mark need to keep their meaning. Encoding a single parameter value must escape those same characters, or a value containing an ampersand will be read as the start of a new parameter and quietly split your data. In JavaScript that is the difference between encodeURI and encodeURIComponent, and picking the wrong one is the reason an API works until someone searches for something containing a plus sign.

Why string length lies

One last surprise. In several languages, including JavaScript, strings are sequences of UTF-16 code units rather than characters. Characters outside the basic range — most emoji among them — occupy two units, called a surrogate pair.

The consequence is that the length of a string containing an emoji is larger than the number of visible characters, and slicing a string can cut a surrogate pair in half, producing an invalid fragment that renders as a replacement character. Emoji built from several code points joined together — skin tone modifiers, family sequences, flags — make this worse: what looks like one glyph can be half a dozen code points, and naive reversing or truncating breaks it apart into its components.

If you are validating input lengths, truncating for display, or reversing text, this is the detail that turns a working function into a bug report from the one user whose name contains an accent.