Tokens aren't characters — a large language model's tokenizer breaks text into chunks that don't map one-to-one with letters, and some kinds of content tokenize far less efficiently than plain prose does. That gap is exactly what makes cleanup worth doing before a long paste, rather than just trimming words to save space.
Why Token Count and Character Count Diverge
Common English words and short sequences typically map to single tokens efficiently. Markup, HTML entities, and unusual punctuation characters often don't — a single numeric HTML entity like “ can end up costing more tokens than the one visible character it represents once decoded.
A Measured Example
Running a short HTML snippet — two paragraphs, a heading, some curly quotes and an em dash — through a cleanup pass:
- Character count drops from 488 to 405 — a 17.0% reduction
- Token count drops from 116 to 77 — a 33.6% reduction
The token reduction is nearly double the character reduction. That gap is the markup and entity overhead being removed — content that took up disproportionate token space relative to how much it actually added, once it's gone the remaining text tokenizes far more efficiently.
Why This Matters for Long Context
On a short prompt, saving 39 tokens doesn't register. On a long document being pasted into a limited context window, the same percentage savings can be the difference between content fitting in one message or needing to be split across several — and every split adds its own overhead of repeated instructions or lost continuity between parts.
What Actually Gets Cleaned
Four categories of cleanup account for most of the savings: stripping HTML structure (while preserving paragraph breaks so sentences don't run together), normalizing smart punctuation to plain characters, removing invisible characters like zero-width spaces, and collapsing irregular whitespace. None of them touch the actual wording or meaning of the content — only the formatting overhead around it.
See your own token savings instantly
Open the Token Size Reducer