Skip to content
Velloreth
Programming 6 min read

Character Encodings Explained: UTF-8 and Mojibake

A character is not a byte. Learn Unicode, UTF-8, the mojibake bug, and how to fix the encoding problems that produce garbled text.

By

Three different things

A character is what a user sees: A, é, 中, 😀.

A code point is a number Unicode assigns to a character: U+0041 for A, U+00E9 for é, U+1F600 for 😀.

A byte is what is stored or transmitted.

The confusion between the three is the source of every encoding bug.

Unicode

Unicode is a standard that assigns a code point to each character.

It covers over 149,000 characters, including every script, a large set of symbols, and the emoji.

A code point is written as U+ followed by a hexadecimal number, such as U+0041.

Unicode does not say how to store a code point. That is the encoding’s job.

UTF-8

UTF-8 encodes a code point as one to four bytes.

Code point rangeBytesExample
U+0000 – U+007F1A
U+0080 – U+07FF2é
U+0800 – U+FFFF3中
U+10000 – U+10FFFF4😀

It is variable-width, which is the key property.

It is backward-compatible with ASCII. The first 128 code points are encoded as a single byte with the same value, so an ASCII file is a valid UTF-8 file.

An English text is compact, because most characters are one byte.

A non-Latin script is larger, because each character is two or three bytes.

The self-synchronising property: a byte that starts a character has a distinctive pattern, so a decoder can find the start of a character after a corruption. This is why UTF-8 is robust.

Other encodings

ASCII — 7 bits, 128 characters. It cannot represent é or any non-English character.

Latin-1 (ISO-8859-1) — 8 bits, 256 characters. It covers most Western European languages and nothing else.

UTF-16 — 2 or 4 bytes per character. It is used internally by JavaScript and Java, and it is rarely the right choice for a file or a network protocol.

UTF-32 — 4 bytes per character. It is simple and wasteful.

The practical answer: use UTF-8 everywhere, and treat any other encoding as a legacy system to be converted.

Mojibake

Mojibake is garbled text produced by decoding with the wrong encoding.

Original:  café
UTF-8:     63 61 66 C3 A9
Decoded as Latin-1:  café

The é is two bytes in UTF-8, and Latin-1 decodes each byte as a separate character, which produces é.

The reverse also happens:

Original:  café
Latin-1:   63 61 66 E9
Decoded as UTF-8:  caf�  (an invalid byte)

The E9 is not a valid UTF-8 sequence, so the decoder produces a replacement character.

The signature of mojibake is a run of Ã, Â, or â characters, which is the UTF-8 lead byte read as Latin-1.

Where the encoding is declared

The HTTP header:

Content-Type: text/html; charset=utf-8

The HTML meta tag:

<meta charset="utf-8">

The database connection:

postgresql://user:pass@host/db?client_encoding=UTF8

The database column and table:

CREATE DATABASE mydb ENCODING 'UTF8';

The file itself, which is a convention rather than a declaration.

The rule: declare it everywhere, and make them agree. A mismatch between the header and the meta tag is a common cause of a subtle bug.

The meta tag must be in the first 1024 bytes of the document, or the browser may have already started decoding.

The database

A database with a Latin-1 encoding cannot store an emoji, and it may silently truncate or replace a character.

The connection encoding matters too. A UTF-8 database with a Latin-1 connection produces mojibake on the way in or out.

The migration is not trivial, because a change of encoding requires a dump and a restore, or a conversion of every column.

The practical advice: create a new database with UTF-8 from the start, and convert a legacy one deliberately.

Fixing a mojibake string

// UTF-8 bytes were decoded as Latin-1
const fixed = Buffer.from(mojibake, 'latin1').toString('utf8');

This works only if no data was lost. If the wrong decoding produced a replacement character, the original byte is gone and the string cannot be recovered.

The general approach: encode the string with the encoding that was wrongly used, then decode it with the correct one.

A Python equivalent:

fixed = mojibake.encode('latin-1').decode('utf-8')

Test on a sample first, and back up the data, because a wrong conversion makes the problem worse.

Encoding in a pipeline

An encoding problem often appears at a boundary rather than in the code, and the boundaries are worth listing because each one is a place where the declaration can be missing. The first is the HTTP response, where the charset belongs in the Content-Type header. The second is the HTML document, where the meta tag must be in the first 1024 bytes. The third is the database connection, where the client encoding must match the database. The fourth is a file read or written by the application, where the encoding should be explicit rather than left to the platform default. The fifth is a message on a queue, where the encoding is often assumed rather than declared. A test that writes a non-ASCII character through the whole pipeline and reads it back is the fastest way to find the boundary that is missing a declaration.

Practical guidance

  • Use UTF-8 everywhere.
  • Declare the encoding in the header, the meta tag, and the database.
  • Make the declarations agree.
  • Put the meta tag in the first 1024 bytes.
  • Set the database and the connection encoding.
  • Do not use a byte order mark for UTF-8.
  • Count characters, not bytes, for a length limit.
  • Test with a non-ASCII character in a form and a database field.
  • Back up before a conversion.

For the related guides, see json-vs-yaml-vs-toml and http-status-codes-explained. For the broader context, form-validation-patterns and environment-variables-explained.

Quotable summary

A character, a code point, and a byte are three different things, and confusing them causes the encoding bugs: Unicode assigns a code point to a character while UTF-8 is one way to encode a code point as bytes. UTF-8 is variable-width, so an ASCII character is one byte and an emoji is four, and it is backward-compatible with ASCII, which is why an English text is compact and an ASCII file is a valid UTF-8 file. Mojibake happens when bytes encoded in one scheme are decoded as another, and its signature is a run of à or  characters, which is the UTF-8 lead byte read as Latin-1. Declare the encoding everywhere, in the HTTP header, the HTML meta tag, and the database connection, and make them agree, because a mismatch is a common cause of a subtle bug.

Frequently asked questions

What is the difference between Unicode and UTF-8?
Unicode is the standard that assigns a code point to each character. UTF-8 is an encoding that turns a code point into bytes. Unicode is the map; UTF-8 is the format.
Why is UTF-8 the default?
It is backward-compatible with ASCII, it is variable-width so an English text is compact, and it can represent every Unicode character.
What is mojibake?
Garbled text produced when bytes are decoded with the wrong encoding, such as UTF-8 bytes read as Latin-1, which turns a character into two or three strange symbols.
How many bytes is an emoji in UTF-8?
Four bytes, because it is outside the Basic Multilingual Plane. A character in the Latin alphabet is one byte, and a character in most other scripts is two or three.
What is a byte order mark?
A special character at the start of a file that identifies the encoding. It is unnecessary for UTF-8 and it can break a parser, so it is usually omitted.
How do I fix a mojibake string?
Re-encode the string with the wrong encoding and decode it with the right one, such as Buffer.from(s, 'latin1').toString('utf8'). It works only if no data was lost.