In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers turns rate limiting from a surprise into a plan.

Common pitfalls

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.

Step by step

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Key takeaway

Track X-RateLimit-Remaining, slow down as it approaches zero, and when you receive a 429, wait for the number of seconds in Retry-After before trying again.