LLM Optimization: Getting Into the Context Window

The unit of retrieval is a fragment, not a page

A page is split into passages before it is ever scored, and the splitter has no understanding of the argument. It cuts on length and on structural markers — headings, paragraph breaks, list boundaries. Whatever falls inside one cut is what the model sees. A page can be authoritative overall and still contribute nothing, because none of its individual fragments answered anything on their own.

Headings are load-bearing

Because splitters lean on structural markers, headings tend to become chunk boundaries. That makes a descriptive heading followed by a complete, self-contained answer the most reliably retrievable shape available. It also means a wall of undifferentiated prose gets split arbitrarily, mid-argument, wherever the length limit happens to land.

Only a handful of passages make it in

Context budgets are finite and shared across the whole answer, so a pipeline typically admits a small number of passages per query. The competition is not with every page on the topic — it is with the few fragments that answer the specific question most directly. Broad coverage loses to precise coverage at this stage, which inverts the usual instinct to make a page cover more.

What this changes in practice

The edits that matter are structural rather than lexical: name the subject at the top of each section instead of relying on a pronoun that refers back to a prior paragraph, put the answer before the reasoning, keep tables and lists near the claim they support, and split a page that answers four questions into four pages that each answer one properly.

LLMs don't browse the open web in real time during most answer generation — they consume sources that retrieval ranking surfaces first. Optimising for the retrieval layer determines whether your content ever enters the generation context window.