{"data":{"id":172,"slug":"session-cookies-for-scrapers-part-9-172","title":"Session cookies for scrapers, part 9","excerpt":"In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and…","seriesName":"Session cookies for scrapers","seriesPart":9,"sections":[{"heading":"Introduction","paragraphs":["In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them back on every later request."]},{"heading":"Putting it into practice","paragraphs":["Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages."]},{"heading":"What to watch for","paragraphs":["Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.","Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again."]},{"heading":"Key takeaway","paragraphs":["Use a session object (requests.Session, a Playwright context) for the whole run, and check for redirects to the login page as a sign the session has expired."]}],"authorId":2,"authorName":"Noah Bennett","authorSlug":"noah-bennett","category":"Scraping Basics","tags":["playwright"],"publishedAt":"2025-02-28T00:00:00.000Z","wordCount":176,"readingTimeMinutes":2}}