Blog

Once you have HTML, the job is to pull out fields that stay correct when the site tweaks markup next month. Fancy tooling doesn't matter as much as boring structure.

Prefer a real parser over regex

Regex is fine for a single ID in a predictable string. For documents, use an HTML parser — BeautifulSoup or lxml in Python, Cheerio in Node. You'll spend less time chasing nested tags and escaped entities.

Select the smallest stable hook

CSS selectors are usually enough. Reach for XPath when you need parent/sibling logic the CSS API makes awkward. Avoid long chains of div:nth-child — those are the first thing a redesign breaks. Prefer semantic hooks: data-* attributes, stable class names on the content itself, itemprop, etc.

Render JS only when you must

If the data is in the initial HTML or a JSON blob in a script tag, parse that. Headless Chrome is slower and heavier; save it for pages that truly won't give you the payload without executing JS.

Clean at the edge

Trim whitespace, normalize prices and dates as you extract, and drop empty rows early. Downstream “we'll clean it in the warehouse” turns into five slightly different price formats.

Fail loudly on shape changes

If a required field is missing, don't silently write nulls for a week. Assert on count and required keys, alert, and keep the raw HTML for that URL so you can fix the selector against reality.

Good parsing is mostly discipline: stable selectors, early validation, and not over-automating pages that already gave you the data in plain HTML.