Stripping HTML for LLM Context: What "Clean" Actually Requires

The naive way to strip HTML is a single find-and-replace: delete anything between angle brackets. That approach technically removes the tags, but it breaks something more important — the structure the tags were providing in the first place.

Why Deleting Tags Alone Isn't Enough

HTML tags like <p>, <h2>, and <li> aren't just markup — they're the only thing separating a heading from the paragraph that follows it, or one list item from the next. Strip them with a blind regex and a heading like "Why Clean Text Matters" followed by a paragraph starting with "When you copy" can collapse into Why Clean Text MattersWhen you copy — one run-on string with no space, no sentence boundary, nothing for the model to parse as two separate ideas.

A proper strip needs to insert a line break before removing each block-level tag, not just delete the tag and move on. That single detail is the difference between HTML-to-text conversion that preserves meaning and one that quietly scrambles it.

What Else Needs Handling Alongside the Tags

• HTML entities — “, —, and similar numeric or named codes need decoding back into actual characters, or they show up as literal escape sequences in your prompt.
• Script and style blocks — a naive strip that only removes tags but not their content will leave raw CSS or JavaScript text sitting in your output, since textContent-style extraction includes it by default.
• Leftover whitespace — once tags are gone, what's left often has doubled-up line breaks and spacing that came from the original markup's indentation.

The Actual Payoff: Token Efficiency

Beyond readability, there's a concrete token cost to skipping this step. Markup and HTML entities tend to tokenize inefficiently — a single — can cost more tokens than the em dash character it represents. In one worked example, cleaning a short HTML snippet dropped its token count from 116 to 77 — a 33.6% reduction — noticeably more than the 17% drop in raw character count, because the removed markup was disproportionately token-expensive relative to its length.