Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.

Putting it into practice

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

What to watch for

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Key takeaway

Retry only on errors that can succeed later (timeouts, 429, 5xx), use exponential backoff with jitter, and cap the number of attempts.