August 7, 2026
Scraping for AI teams: structure the data before you prompt
Feeding an LLM a zip of raw HTML feels modern and usually wastes money. Extract the stable fields first — then use models where the markup actually fails you.
Blog
Notes from the scraping trenches.
August 7, 2026
Full re-scrapes are expensive — start with change detection
Most catalogs barely move day to day. Hashing the parts that matter lets you skip the 95% that didn’t change — and catch the 5% that did.
July 26, 2024
Building a trusted browser session with Puppeteer
Most scraping failures aren't selector bugs — they're session problems. Here's how we keep a Puppeteer browser looking like a real visit long enough to finish the job.
December 17, 2023
Why we use private proxies for scraping
Public proxy lists look cheap until half of them are already burned. Here's when private proxies are worth paying for — and when they're not.
October 12, 2023
Dealing with CAPTCHAs when you scrape
The best CAPTCHA strategy is not solving them — it's avoiding the behavior that triggers them. When you do hit one, here's the order we try things.
July 3, 2023
When the site ties your session to an IP
Some sites don't just issue a cookie — they expect that cookie to keep coming from the same IP. Rotate carelessly and you'll look logged out, banned, or both.
April 24, 2023
Parsing scraped HTML without making a mess
Fetching the page is the easy half. Turning messy HTML into rows you can trust is where scrapers usually get fragile — and where a few habits save you later.