For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.

Going further

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Common pitfalls

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Key takeaway

Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies.