robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
Putting it into practice
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
What to watch for
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Key takeaway
Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies.