Part 7 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile; one that retries instantly and forever is a nuisance.

A practical approach

Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Common pitfalls

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Key takeaway

Retry only on errors that can succeed later (timeouts, 429, 5xx), use exponential backoff with jitter, and cap the number of attempts.