Characters, code points and bytes: why counts disagree

Four units, four answers

Take the family emoji, a single glyph a reader would call one character. Here is its length in four different units:

Every one of those is correct. They answer different questions, and each language picked the one its string type is built on. Nothing is broken; the systems simply do not agree on what they are counting.

The family emoji reaches five code points because it is three people joined by two zero-width joiners. Each person is a code point above the basic plane, which costs two UTF-16 code units and four UTF-8 bytes; each joiner costs one and three. The visible glyph is an instruction to the text renderer, not a character in any storage sense.

Where UTF-16 counts twice

JavaScript, C# and Java all represent strings as sequences of 16-bit code units. Sixteen bits covers the first 65,536 code points; everything above that — emoji, historic scripts, many CJK extension characters, mathematical alphabets — is stored as a surrogate pair of two code units.

"a".length        // 1
"é".length        // 1
"中".length        // 1
"😀".length       // 2   <- one character, two code units

So a string property called length returns 2 for something the user typed as one keystroke. That alone is survivable. The dangerous part is indexing: cut a string between the two halves of a surrogate pair and you produce an unpaired surrogate — a string that is no longer valid Unicode. It renders as a replacement character, it may fail to encode as UTF-8, and it can be rejected by a database or an API at a layer far from where it was created.

This is why slice(0, 100) on user text is unsafe. It is right 99% of the time and produces corrupt output the rest, which is the worst ratio for a bug to have.

What a user calls one character

Counting code points instead of code units fixes emoji from the basic planes but not the general case, because many visible characters are several code points by design:

• Combining marks. é can be one code point, or e followed by a combining acute accent. Both display identically.
• Emoji modifiers. A skin tone is a separate code point appended to a base emoji.
• Zero-width joiner sequences. Family and profession emoji are several emoji joined together.
• Flags. A national flag is two regional indicator symbols.
• Indic and other scripts where a written syllable is routinely several code points.

The unit that matches human intuition is the grapheme cluster, defined by Unicode's text segmentation rules. Several platforms expose it: Intl.Segmenter in JavaScript, StringInfo in .NET, Swift's Character type by default. Where it is not available, a library is the sensible route — the rules change with each Unicode revision, so this is not a good thing to implement yourself.

Grapheme clusters are the right unit for a user-facing limit and for truncation. They are the wrong unit for storage, because a cluster has no fixed size.

Database columns count differently again

A column length is a third kind of limit, and the unit depends on both the engine and the type.

Two traps worth naming. On MySQL, the older utf8 alias is a three-byte encoding that cannot store emoji or any character above the basic plane; inserting one either errors or truncates the value at that point depending on strict mode. Use utf8mb4, always.

On SQL Server, NVARCHAR(50) holds 50 code units, so a user who types 30 emoji overflows a column that promised 50 characters — and the error arrives at the database, after every application-level validation has passed.

Truncating without breaking text

If you must cut text to fit, cut in the unit the limit is expressed in, and cut at a safe boundary.

• For a display limit, segment into grapheme clusters and take the first N. This never splits a glyph and never produces invalid text.
• For a byte limit, encode to UTF-8 and walk backwards from the cut point until you reach the start of a character — a byte that is not a continuation byte. Never cut the byte array blindly.
• For a UTF-16 limit, check whether the character before the cut is a high surrogate and move back one if so.

And the more useful advice: reject rather than truncate. Tell the user their input is too long and let them decide what to remove. Silent truncation is how an address loses its postcode.

Which unit to use where

• Grapheme clusters for anything a person sees: counters, limits you show them, truncation for display.
• UTF-8 bytes for storage, network payloads and anything with a size budget.
• Code points for text processing and validation rules.
• UTF-16 code units only when a platform forces it, and then treat every index as a hazard.

The one habit that prevents most of these bugs: state the unit wherever a limit is written down. "Maximum 280" is ambiguous, and every bug in this guide starts with that ambiguity. The Character Counter reports characters, code points and UTF-8 bytes side by side for exactly this reason.