{"data":{"id":29,"slug":"rate-limit-headers-explained-part-2-29","title":"Rate limit headers explained, part 2","excerpt":"For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those…","seriesName":"Rate limit headers explained","seriesPart":2,"sections":[{"heading":"Introduction","paragraphs":["For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers turns rate limiting from a surprise into a plan."]},{"heading":"What to watch for","paragraphs":["Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.","Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would."]},{"heading":"Going further","paragraphs":["Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it."]},{"heading":"Key takeaway","paragraphs":["Track X-RateLimit-Remaining, slow down as it approaches zero, and when you receive a 429, wait for the number of seconds in Retry-After before trying again."]}],"authorId":10,"authorName":"Aisha Rossi","authorSlug":"aisha-rossi","category":"APIs","tags":["python","playwright"],"publishedAt":"2024-04-28T00:00:00.000Z","wordCount":171,"readingTimeMinutes":2}}