A pattern we keep seeing: someone wires a crawler, dumps every page into a bucket, and asks a model to “extract the products.” It demos well on ten URLs. At ten thousand it gets slow, pricey, and quietly wrong.
LLMs are bad scrapers. They’re decent cleanup and classification tools when the hard fetch-and-locate work is already done.
What should never go through a prompt
Prices, SKUs, stock flags, canonical product URLs, ratings counts — anything that lives in a stable DOM node or a JSON blob in the page. Parse those with Cheerio, lxml, or a headless render + selector. Deterministic extractors are cheap to rerun and easy to test when the site shifts one class name.
If you can write a unit test against a saved HTML fixture, don’t spend tokens on it.
Where models earn their keep
- Long, messy descriptions you need normalized into short attributes (“materials”, “fit”, “warranty”).
- Category mapping into your taxonomy when the site’s nav is chaos.
- PDFs and scans where layout varies too much for a single table parser.
- Deduping near-identical titles across merchants without maintaining a thousand regexes.
A pipeline shape that scales
- Fetch with normal scraping discipline (session, proxy, rate limits).
- Extract structured fields into a typed row (nulls allowed, types enforced).
- Gate — required columns present? row counts sane? quarantine failures.
- Enrich — send only the messy text fields (or PDF chunks) to a model, with a strict JSON schema out.
- Merge — model output is additive columns, never the source of truth for price or identity.
// Pseudo-flow: deterministic core, optional model enrich
const row = extractProduct(html); // title, price, sku, url
assertRequired(row, ['title', 'price', 'url']);
if (row.description) {
row.attributes = await llmJson({
system: 'Return only JSON matching the schema.',
schema: { materials: 'string[]', audience: 'string' },
input: row.description.slice(0, 4000),
});
}
Cost and quality are the same conversation
Prompting over full HTML pays for nav, scripts, and cookie banners. Strip to the text you care about. Cache model results by content hash so unchanged descriptions don’t get re-billed every nightly crawl. Log prompt/response pairs for a sample — “the model said so” is not a data dictionary.
What we tell clients
If your goal is RAG over a site, you still want clean chunks with stable IDs and metadata (URL, scraped_at, product_id). Unstructured soup in a vector DB just moves the mess downstream. Structure first; generate second.
AI doesn’t replace the scraper. It sits on top of one that already knows where the truth is on the page.