Part 7 looks at what changes when the same technique has to run every day in production. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
Going further
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
What to watch for
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Key takeaway
Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies.