For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines.

Putting it into practice

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

What to watch for

Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Key takeaway

Locate the table by a stable attribute, read the header row once, and convert every cell to a typed value before you store it.