Full re-scrapes are expensive — start with change detection

Default instinct for a daily scrape: hit every URL again, parse everything again, overwrite the warehouse again. It works. It also burns proxy budget, CPU, and patience on pages that look identical to yesterday.

For a lot of catalog and listing work, a better default is change detection first, full extract second.

What we fingerprint

Not the whole HTML blob — that flips when a marketing banner rotates. Fingerprint the region that holds the data you sell: the product block, the price node, the availability string, the JSON-LD Product object. Stable selector → normalize whitespace → hash. If the hash matches yesterday, you can skip deep parsing (and sometimes skip the render entirely if a cheap HTTP fetch still produces the same fingerprint).

import crypto from 'crypto';
import * as cheerio from 'cheerio';

function regionHash(html, selector) {
  const $ = cheerio.load(html);
  const text = $(selector).text().replace(/\s+/g, ' ').trim();
  return crypto.createHash('sha256').update(text).digest('hex');
}

const today = regionHash(html, '[data-product-detail]');
if (today === storedHash) {
  // unchanged — log a heartbeat, move on
} else {
  // run the full field extractor + write a new row version
}

When this pays off

• Large catalogs where <10% of SKUs change price or stock on a given day.
• Jobs billed by proxy bandwidth or headless minutes.
• Downstream teams that care about deltas (what changed) more than nightly full dumps.

When to still re-scrape everything

First backfill. Schema redesigns. Legal/compliance snapshots that must prove “as of date D we saw X.” Targets that shuffle DOM without real content changes so aggressively that fingerprints become noise — then you move the hash up a layer (API responses, embedded JSON) or accept fuller crawls.

Operational habits that keep it honest

• Store raw HTML (or the JSON payload) for changed URLs only — enough to debug, not a museum of duplicates.
• Force a full pass on a random 1% sample daily so silent selector rot doesn’t hide behind “unchanged.”
• Emit an explicit unchanged event. Silence is not a heartbeat; monitoring should know the job ran.
• Version rows on change so analysts can ask “what was the price last Tuesday?” without you replaying the whole internet.

Change detection isn’t laziness — it’s admitting most pages are boring most days. Spend the scrape budget on the ones that aren’t.