In part 4 we take the ideas from earlier instalments and apply them to a larger, messier site. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.

Going further

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.

A practical approach

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Key takeaway

Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.