This is part 5 of our series, and this time we focus on edge cases readers asked about. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.

A practical approach

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Step by step

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Key takeaway

Retry only on errors that can succeed later (timeouts, 429, 5xx), use exponential backoff with jitter, and cap the number of attempts.