For part 4, we revisit the basics with a fresh example and a few lessons learned the hard way. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.

Going further

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.

Putting it into practice

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Key takeaway

Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies.