In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option.

Common pitfalls

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

What to watch for

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Key takeaway

Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way.