Blog

If you've scraped for more than a week, you've hit this: the first request works, the next ten return empty HTML or a challenge page. Selectors are fine. The site decided your session looks wrong.

Puppeteer helps because you're driving a real Chrome, not a bare HTTP client. That alone is not enough. You still have to keep user-agent, cookies, and timing consistent across the run.

Start with a stable browser context

Launch once, reuse the same page (or browser context) for the whole job. Recreating the browser on every URL throws away cookies and makes you look like a new visitor every few seconds.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });
  const page = await browser.newPage();
  await page.setUserAgent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
    '(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  );
  await page.setViewport({ width: 1366, height: 768 });
  // scrape here, then close once
  await browser.close();
})();

Cookies are the session

After a login or an initial warm-up visit, dump cookies and reload them on the next run. Don't rotate the IP mid-session if the site ties the cookie jar to that IP — you'll invalidate yourself.

const cookies = await page.cookies();
// persist cookies somewhere durable
await page.setCookie(...savedCookies);

Wait for what you need, not for a fixed sleep

Fixed sleep(3000) calls waste time and still race. Prefer waiting on a selector or network idle for the piece of the page you actually parse.

await page.goto('https://example.com/listings', {
  waitUntil: 'domcontentloaded',
});
await page.waitForSelector('table.data tbody tr');
const rows = await page.$$eval('table.data tbody tr', (trs) =>
  trs.map((tr) =>
    [...tr.querySelectorAll('td')].map((td) => td.innerText.trim())
  )
);

What we actually do in production

• Keep one residential IP per authenticated session; rotate between sessions, not inside them.
• Cap concurrency. Parallel tabs from one IP is a fast way to get challenged.
• Match viewport and user-agent to something current — stale Chrome strings are an easy fingerprint.
• Debug with headed mode once. Headless is fine after you've confirmed the flow.

Trusted sessions are boring infrastructure: same browser, same cookies, same IP, human-ish pacing. Get that right and Puppeteer stops feeling fragile.