How Scraping API Output Format Determines Token Efficiency and LLM Accuracy

LLM-ready structured data extraction from web pages starts with a scraping API that returns boilerplate-stripped markdown for prose or schema-defined JSON for known fields. Output format directly shapes token cost and factual accuracy because it determines what the model must read before it can answer. The right choice strips out page furniture and keeps the facts.

Why the scraping API's output format is not a cosmetic choice

Teams spend months tuning prompts, embeddings, and retrieval strategies, then treat the scraper’s output format like a timezone setting. That leaves a large source of inefficiency untouched. Every token entering an LLM consumes context space and adds to inference cost, whether it contains the answer or a cookie banner. A better prompt does not make the banner disappear. The model still receives it.

Pages are built for browsers and people, so their HTML includes navigation menus, ad scripts, footer links, and other interface elements that have no useful role in answering many questions. Sending all of that into a model makes the pipeline pay for the packaging as well as the contents.

AlterLab’s pipeline analysis describes how raw HTML forces a RAG pipeline to process tag names, CSS classes, JavaScript snippets, and whitespace with no meaning for the task. That extra material inflates context size and can push relevant content outside the context window. Format therefore belongs in the design of the pipeline. It determines what the other optimizations have to work with.

What raw HTML actually looks like inside an LLM's context window

A Wikipedia article test makes the imbalance visible. The article content was roughly 15KB, while the total output was roughly 373KB. The rest included navigation menus, language selector links, table-of-contents controls, sidebar widgets, and edit buttons. The article arrived with an entourage considerably larger than itself.

Those byte counts show how much material surrounds the useful text before tokenization even begins. Inside the context window, that scaffolding is part of what the model has to process. A hidden CSS class takes up space even though it adds nothing to the answer. The browser’s needs have become the model’s workload.

The problem goes beyond a longer bill. The DOM describes page structure and presentation. AlterLab’s breakdown of agent data ingestion identifies embedded SVGs, tracking scripts, and deep div hierarchies as sources of thousands of wasted tokens. It also describes how this noise degrades reasoning by flooding attention with irrelevant structure. The useful fact may be present, but it is competing with the machinery used to display it.

The format hierarchy — how token count and accuracy change from HTML through markdown to JSON

The progression runs from raw HTML through standard markdown and boilerplate-stripped markdown to schema-defined JSON. Each step makes the output more selective. Raw HTML sends the full DOM into the prompt, so it has the most tokens and the weakest signal-to-noise ratio. Standard markdown removes the tag soup, but it can still preserve navigation, footers, and sidebars as perfectly valid links and headings.

Boilerplate-stripped markdown adds a more demanding parsing step. The scraper finds and removes non-content regions before turning the text that is left into markdown. This raises semantic density without requiring the developer to specify every fact that might be useful.

Schema-defined JSON goes further when the target fields are known. Instead of sending an article and asking the model to find a value, it returns the requested keys and typed values. This can be the most token-efficient option for field retrieval, but it requires schema definition, extraction infrastructure, and maintenance.

The reduction helps only when it removes distractions while preserving the evidence the task needs. “LLM-ready” should therefore mean output that enters a retrieval or agent pipeline without a hand-built cleanup stage.

How do you get clean markdown your LLM can use without cleaning it first?

Markdown is the right default when a pipeline needs to reason over prose. It preserves headings, lists, emphasis, and tables while dropping presentation markup, and it supports predictable chunking for embedding. Markdown syntax says nothing about whether a paragraph belongs to the article or the footer.

Standard conversion can turn a navigation menu into a long bulleted list of links. A sidebar can turn up as a stray heading, while footer text becomes a plain paragraph. None of this is a formatting error. It is junk wearing a tidy shirt.

Removing that material requires the scraping layer to classify DOM regions before serialization. The signals include text density, link density, element position, and semantic HTML landmarks. Only the content regions should become the output. This is a different operation from translating HTML elements into their markdown equivalents.

For a developer already requesting markdown, the practical adjustment is to choose the boilerplate-stripped variant. Where supported, that can be a one-line change from format=markdown to format=markdown-llm. The parameter name is less important than the behavior behind it. For agents visiting dozens of URLs per task, the savings accumulate across pages and then across thousands of calls.

When to move from markdown to structured JSON — and what that requires from the scraping layer

JSON earns its extra setup when the target is a known set of fields. An agent answering “what is the current price of this product?” needs a price. With a correctly populated JSON response, the reasoning step becomes a lookup. With markdown, the model has to find the value and pull it out of the text around it.

That does not make JSON a truth guarantee. A price field still has to contain the right price. The advantage is that the response has an explicit contract, so the developer can check the requested keys and types rather than depend on another open-ended reading step.

AlterLab’s documentation describes formats=["json"] paired with a Cortex AI extractor that accepts a natural language or JSON-Schema description of the target fields. The extractor populates the schema and returns those keys. This requires more infrastructure than markdown conversion, and extraction models add latency and per-call cost. Schemas also need maintenance when source pages change structure.

Isn’t this just moving complexity from the prompt to the schema? Yes, but the move changes how that complexity scales. The work of defining a schema grows with the number of fields, while the work of reading noisy pages repeats with every page. At high volume, the upfront work can be spread across many requests.

A hybrid approach uses an LLM once to generate selectors or a schema from a sample page, then applies that definition at scale without per-page LLM extraction costs. The trade-off is lower recurring cost in exchange for the work of generating and maintaining the extraction rules.

How format choice plays out differently in RAG pipelines versus live agent loops

The first branch is whether the downstream task needs prose reasoning or field lookup. For index-time RAG over articles and documents, boilerplate-stripped markdown usually fits. Content arrives asynchronously, so scraping latency is not the binding constraint. The embedding stage can split the text at heading and paragraph boundaries, while stripping out boilerplate keeps that noise out of the vector store.

Product catalogs, job listings, and event data often need filtering on metadata as much as semantic retrieval. JSON is a better fit when those fields define how the records will be searched. A prose-first index and a record-first index need different inputs.

Live agent loops add a timing constraint. The agent blocks while the scraping API responds, so extraction latency and the number of returned tokens both affect the task run. With JSON, the agent can get a price, availability status, or date without searching through prose. For summarizing, comparing, or synthesizing content, markdown preserves the broader evidence.

That distinction prevents a misleading universal ranking. The smallest response is not automatically the most useful one, especially if a narrow schema excludes context needed for comparison. Conversely, a well-written page is still excess input when the agent only needs one field.

Why format decisions are now embedded in protocol-level tool schemas, not just API parameters

Anthropic’s Model Context Protocol, open-sourced in November 2024, sets out a standard client-server interface that lets agents use tools outside the model, including search and scraping. This moves the integration beyond individually written API calls. Agents can discover tools through their declared definitions. The agent uses the tool’s description as part of what it reads when planning.

In this setting, the declared output format shapes what the agent expects before it fetches a page. It informs both the choice of tool and the handling of the response. A noisy default can therefore impose a recurring token tax on every agent using that tool.

Developers should choose an API whose MCP server or tool definition exposes clean markdown and schema-defined JSON. Agents can then inherit useful format defaults without separate configuration for every integration. The same tool contract can serve many consumers, making the result of that choice travel further.

Format and freshness together — what changes when web data must be current, not just clean

A compact response from a stale page is still wrong for a task about current conditions. Competitor pricing, breaking news, and regulatory changes all depend on when the data was obtained. Clean output cannot compensate for old evidence.

Monitoring changes the shape of the interaction. A developer defines a natural language query or a set of URLs, chooses a cadence, and receives a webhook when relevant new information appears. That webhook is another input to the agent. If it contains noisy HTML, the pipeline has reopened the same cleanup problem it solved at scrape time.

Schema-defined JSON makes the alert small and actionable by design. Rather than sending a full-page dump for the agent to interpret, the monitoring layer can report structured changes. The distinction is between an event payload and another document to read.

A platform with one API for search, scraping, and monitoring can follow the same output rules at each stage. Separate tools can introduce format mismatches that require additional cleanup between them. With the same rules, the agent can use the same extraction routine however new information reaches it.

A practical format-selection framework for developers building AI pipelines today

If the target is open-ended prose for RAG, use boilerplate-stripped markdown and prioritize server-side DOM classification over basic conversion. If it is a known set of fields for an agent or structured dataset, use schema-defined JSON. Reuse the schema across pages so each downstream prompt can go without repeated extraction instructions.

Where the API supports it, request markdown and JSON in the same call. Another option is to use markdown for retrieval and trigger JSON extraction only when a field lookup is needed. For monitoring, require structured JSON alerts so the receiving agent can act without a separate extraction step.

Evaluate the API against the work each format promises. For markdown, look for removal of non-content regions and predictable chunking at heading boundaries. For JSON, look for natural language or JSON-Schema definitions, validation before the response returns, and extraction latency suitable for synchronous agents. For monitoring, look for change deltas rather than full-page dumps, plus cadence control and event filtering.

Then test preservation alongside compression. Recall on labeled data shows whether the extractor kept the content that should have been in the output. A smaller response that silently drops evidence is no bargain. Request the same page in each available format, measure token count, check that the required content survives, and run a sample of downstream LLM queries against each version to compare factual accuracy.