A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers turns rate limiting from a surprise into a plan.

Common pitfalls

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

What to watch for

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Key takeaway

Track X-RateLimit-Remaining, slow down as it approaches zero, and when you receive a 429, wait for the number of seconds in Retry-After before trying again.