Notes from building production data pipelines.
August 12, 2026
One product link can show three prices, three stock states, and three assortments depending on where your exit IP sits. If you collect without a market column, you’re inventing a fake global catalog.
August 7, 2026
Feeding an LLM a zip of raw HTML feels modern and usually wastes money. Acquire the stable fields first — then use models where the markup actually fails you.
August 7, 2026
Most catalogs barely move day to day. Hashing the parts that matter lets you skip the 95% that didn’t change — and catch the 5% that did.
July 26, 2024
Most collection failures aren't selector bugs — they're session problems. Here's how we keep a Puppeteer browser looking like a real visit long enough to finish the job.
December 17, 2023
Public proxy lists look cheap until half of them are already burned. Here's when private proxies are worth paying for — and when they're not.
October 12, 2023
The best CAPTCHA strategy is not solving them — it's avoiding the behavior that triggers them. When you do hit one, here's the order we try things.
July 3, 2023
Some sites don't just issue a cookie — they expect that cookie to keep coming from the same IP. Rotate carelessly and you'll look logged out, banned, or both.
April 24, 2023
Fetching the page is the easy half. Turning messy HTML into rows you can trust is where pipelines usually get fragile — and where a few habits save you later.
