This is part 3 of our series, and this time we focus on edge cases readers asked about. Before you crawl a site link by link, check whether it already publishes a list of every page it wants found. That list is usually called sitemap.xml.

Going further

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

What to watch for

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Key takeaway

Look for the Sitemap line in robots.txt, follow sitemap index files to their children, and use lastmod to skip pages that have not changed.