Part 9 looks at what changes when the same technique has to run every day in production. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.
Putting it into practice
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Common pitfalls
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Key takeaway
Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.