For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers turns rate limiting from a surprise into a plan.
Putting it into practice
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Common pitfalls
Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Key takeaway
Track X-RateLimit-Remaining, slow down as it approaches zero, and when you receive a 429, wait for the number of seconds in Retry-After before trying again.