Part 5 looks at what changes when the same technique has to run every day in production. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.

Putting it into practice

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.

A practical approach

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Key takeaway

Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.