This is part 7 of our series, and this time we focus on edge cases readers asked about. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines.
Step by step
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
What to watch for
Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Key takeaway
Locate the table by a stable attribute, read the header row once, and convert every cell to a typed value before you store it.