For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.

Going further

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Step by step

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Key takeaway

Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.