Blog

Dealing with CAPTCHAs when you scrape

Author: ish1301 · Posted: October 12, 2023

CAPTCHAs show up when a site thinks you aren't a person. Solving them is expensive and brittle. Avoiding the trigger is cheaper.

1. Slow down before you reach for a solver

Most CAPTCHAs we see are self-inflicted: too many requests from one IP, no referer chain, jumping straight to deep URLs. Add delays, warm up from the homepage, and keep concurrency low. A surprising number of challenges disappear after that.

2. Look like one browser, not a script farm

Reuse cookies. Use a current user-agent. Prefer a real browser (Puppeteer / Playwright) when the page is JS-heavy. Bare HTTP clients are fine for simple HTML sites; for everything else they fail loudly.

3. Keep the session sticky

If you already solved a challenge (or logged in), don't rotate the IP on the next request. Sites often bind the pass token to that session and address. Break the pair and you'll solve CAPTCHAs in a loop.

4. Use solvers as a last resort

Third-party CAPTCHA services work, and they cost money per solve. We wire them in when avoidance isn't enough — login walls, occasional interstitial checks — not as the primary crawl strategy. Budget for failure retries; solvers aren't 100%.

5. Know when to stop

If a site's terms forbid automated access, or the only path is constant CAPTCHA grinding, that's a product decision — not a scraping puzzle. We tell clients when a target isn't worth the operational cost.

Order of operations for us: reduce request rate → stabilize session + IP → render in a real browser → then, if needed, a solver. Jumping straight to bypass services usually means you're fighting the wrong problem.

Need data scraped? Tell us the source — we’ll reply with a plan.