Dealing with CAPTCHAs when you scrape
CAPTCHAs show up when a site thinks you aren't a person. Solving them is expensive and brittle. Avoiding the trigger is cheaper.
1. Slow down before you reach for a solver
Most CAPTCHAs we see are self-inflicted: too many requests from one IP, no referer chain, jumping straight to deep URLs. Add delays, warm up from the homepage, and keep concurrency low. A surprising number of challenges disappear after that.
2. Look like one browser, not a script farm
Reuse cookies. Use a current user-agent. Prefer a real browser (Puppeteer / Playwright) when the page is JS-heavy. Bare HTTP clients are fine for simple HTML sites; for everything else they fail loudly.
3. Keep the session sticky
If you already solved a challenge (or logged in), don't rotate the IP on the next request. Sites often bind the pass token to that session and address. Break the pair and you'll solve CAPTCHAs in a loop.
4. Use solvers as a last resort
Third-party CAPTCHA services work, and they cost money per solve. We wire them in when avoidance isn't enough — login walls, occasional interstitial checks — not as the primary crawl strategy. Budget for failure retries; solvers aren't 100%.
5. Know when to stop
If a site's terms forbid automated access, or the only path is constant CAPTCHA grinding, that's a product decision — not a scraping puzzle. We tell clients when a target isn't worth the operational cost.
Order of operations for us: reduce request rate → stabilize session + IP → render in a real browser → then, if needed, a solver. Jumping straight to bypass services usually means you're fighting the wrong problem.
