Character Encoding:
Unicode, UTF-8, ASCII, Hex & Binary

Why Character Encoding Still Matters

Character encoding sounds like a solved problem. UTF-8 won. Over 98% of websites use it. Yet every developer hits encoding bugs: garbled Chinese characters in a CSV export, a "?" where an emoji should be, a JSON file that breaks because someone saved it as Latin-1. These problems haven't disappeared — they've just gotten more subtle.

This guide covers the encoding formats you'll actually encounter day-to-day: Unicode escape sequences in JavaScript, UTF-8 byte inspection, ASCII code lookups, and hex encoding. You'll learn what each one does, when to reach for it, and how to avoid the mistakes that waste hours of debugging.

Unicode: One Encoding to Rule Them All

The Problem Unicode Solved

Before Unicode, every language had its own encoding. Russian used KOI8-R. Japanese used Shift-JIS. Chinese used GB2312. A document written in one encoding would render as garbage in another — a phenomenon called mojibake (文字化け), literally "transformed characters." If you've ever opened a file and seen strings of � or scrambled symbols, you've encountered an encoding mismatch.

Unicode assigns a unique number — a code point — to every character across all writing systems. "A" is U+0041. "中" is U+4E2D. "😀" is U+1F600. That's over 140,000 characters, from Latin to Cuneiform to Emoji, all in one coherent system.

The code point is the abstract identifier. How it gets stored as bytes depends on the encoding form — UTF-8, UTF-16, or UTF-32. UTF-8 is the one you actually need to know.

Unicode Escape Sequences in Code

JavaScript, JSON, and CSS all support Unicode escape sequences — a way to represent any character using only ASCII characters. In JavaScript, you write \uXXXX for characters in the Basic Multilingual Plane (U+0000 to U+FFFF) and \u{XXXXX} for higher code points:

// BMP characters (4 hex digits)
'\u4e2d\u6587'  →  "中文"

// Emoji with surrogate pairs (older syntax)
'\uD83D\uDE00'  →  "😀"

// ES6+ supports full code points with braces
'\u{1F600}'      →  "😀"

When you see a JSON response with \uXXXX sequences, it's because the server is escaping non-ASCII characters to keep the response ASCII-safe. This is common in REST APIs that serve international content. The data is intact — it just needs to be decoded.

Use the Unicode Converter to encode text to escapes or decode \uXXXX back to readable characters. It handles surrogate pairs and emoji correctly, which many hand-rolled solutions get wrong.

UTF-8: How Characters Become Bytes

The Variable-Length Design

UTF-8 is a stroke of engineering elegance. It's variable-length: ASCII characters use 1 byte, extended Latin and Arabic use 2, CJK characters use 3, and emoji use 4. The brilliance is in how it stays backward-compatible — every ASCII document is automatically valid UTF-8.

Here's how the byte patterns work:

Byte Count	First Byte Pattern	Code Point Range	Example
1	0xxxxxxx	U+0000 – U+007F	A → 41
2	110xxxxx	U+0080 – U+07FF	é → C3 A9
3	1110xxxx	U+0800 – U+FFFF	中 → E4 B8 AD
4	11110xxx	U+10000+	😀 → F0 9F 98 80

Continuation bytes always start with 10. This means you can't mistake the middle of one character for the start of another — if your data stream gets corrupted mid-character, the decoder can find the next valid start byte and recover. That's a deliberate design choice, not an accident.

Why Byte Count ≠ Character Count

A common pitfall: the string "Hello世界" has 7 characters but 13 bytes in UTF-8. If you're storing text in a database with a byte-length limit (like MySQL's old VARCHAR(255)), you might truncate multi-byte characters in the middle, producing invalid UTF-8. Always count characters, not bytes, when dealing with text length limits. Use the UTF-8 Converter to see the exact byte breakdown for any text.

ASCII: The Foundation That Won't Die

The 128-Character Baseline

ASCII (American Standard Code for Information Interchange) was published in 1963. It defines 128 characters: 33 control codes (0-31 and 127) and 95 printable characters (32-126). It's tiny by modern standards — no accents, no non-Latin scripts, not even a euro sign. Yet ASCII's influence persists because UTF-8 is backward-compatible: every valid ASCII byte is also valid UTF-8.

The ASCII table has a logical structure worth memorizing in broad strokes:

• 48-57: Digits 0-9. So '5' - '0' equals 5 in most languages.
• 65-90: Uppercase A-Z. Notice that 'A' (65) and 'a' (97) differ by exactly 32 — that's a single bit flip.
• 97-122: Lowercase a-z. The 32-bit gap between cases means you can convert case by toggling bit 5.
• 32: Space. The most common character in most text.
• 10: Line Feed (\n). The Unix newline. Windows uses 13 followed by 10 (\r\n).

Knowing these ranges helps you read hex dumps at a glance. See 48 in a hex editor and you know it's the digit '0'. See 41 and it's 'A'. The ASCII Converter shows decimal, hex, and binary for every character — useful when you're debugging wire protocols or file formats.

Hex Encoding: Bytes as Human-Readable Text

Why Hex Is Ubiquitous

Hexadecimal (base-16) is the lingua franca of low-level programming because it maps cleanly to bytes. Each byte (8 bits) fits exactly into two hex digits (00-FF). This makes hex dumps readable in a way binary isn't — you can scan a hex dump and spot patterns, like ASCII text ranges (20-7E) or null bytes (00).

Hex encoding is everywhere once you start looking:

• CSS colors: #FF5733 is just three bytes (red, green, blue) in hex.
• Cryptographic hashes: SHA-256 outputs are 32 bytes, displayed as 64 hex characters.
• Memory addresses: 0x7fff5fbff8c0 — the 0x prefix signals hex in C, C++, Rust, and many others.
• Network packets: Wireshark displays payload bytes in hex with an ASCII sidebar.

Converting between text and hex is straightforward: each byte of the UTF-8 representation becomes two hex digits. The letter "A" (byte 0x41) becomes the string "41". The Chinese character "中" (bytes 0xE4 0xB8 0xAD) becomes "E4 B8 AD". Use the Hex Converter to encode or decode hex strings — it automatically strips whitespace and 0x prefixes, so you can paste hex dumps directly.

Binary Encoding: The Raw Representation

Binary (base-2) is the fundamental encoding — everything ultimately reduces to 0s and 1s. While you rarely work with raw binary strings in production code, understanding how text maps to bits helps you grasp why certain bugs happen. A byte is 8 bits. The letter "A" in binary is 01000001. That leading zero isn't decoration — it distinguishes ASCII characters (which all start with 0) from extended character sets.

Binary conversion is primarily useful for learning and teaching. If you're explaining character encoding to a junior developer, starting with binary makes everything else click. It also comes up in hardware contexts (microcontrollers, FPGA programming) and in certain interview questions. The Binary Converter handles the conversion in both directions.

Practical Debugging Workflow

When you hit an encoding problem, here's the systematic approach:

1. Identify the symptom. Garbled characters (mojibake) usually mean UTF-8 bytes are being interpreted as Latin-1 or Windows-1252. Question marks (�) are the Unicode replacement character — it means the decoder encountered bytes that aren't valid in the expected encoding.
2. Check the bytes, not the display. Use the UTF-8 Converter or a hex editor to see the actual bytes. What encoding produced them? What encoding is interpreting them?
3. Trace the pipeline. Encoding bugs happen at boundaries: database → backend → JSON serializer → HTTP response → browser. At each step, verify the encoding declaration (charset, Content-Type header, BOM).
4. Fix at the source. The most robust fix is to always declare UTF-8 everywhere: in your database (utf8mb4 in MySQL), in your HTTP headers (Content-Type: application/json; charset=utf-8), in your HTML meta tags, and in your code editor.

Bookmark the Unicode Converter, UTF-8 Converter, ASCII Converter, Hex Converter, and Binary Converter. Together they cover every encoding conversion you'll need in day-to-day development.