In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines.
Putting it into practice
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
What to watch for
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
Key takeaway
Locate the table by a stable attribute, read the header row once, and convert every cell to a typed value before you store it.