{"data":{"id":49,"slug":"rate-limit-headers-explained-part-3-49","title":"Rate limit headers explained, part 3","excerpt":"This is part 3 of our series, and this time we focus on edge cases readers asked about. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…","seriesName":"Rate limit headers explained","seriesPart":3,"sections":[{"heading":"Introduction","paragraphs":["This is part 3 of our series, and this time we focus on edge cases readers asked about. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers turns rate limiting from a surprise into a plan."]},{"heading":"What to watch for","paragraphs":["Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.","Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.","Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy."]},{"heading":"Common pitfalls","paragraphs":["Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.","Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.","Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again."]},{"heading":"Key takeaway","paragraphs":["Track X-RateLimit-Remaining, slow down as it approaches zero, and when you receive a 429, wait for the number of seconds in Retry-After before trying again."]}],"authorId":12,"authorName":"Yuki Andersson","authorSlug":"yuki-andersson","category":"APIs","tags":["python","http","scrapy","testing"],"publishedAt":"2024-03-31T00:00:00.000Z","wordCount":240,"readingTimeMinutes":2}}