{"data":{"id":92,"slug":"session-cookies-for-scrapers-part-5-92","title":"Session cookies for scrapers, part 5","excerpt":"In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and…","seriesName":"Session cookies for scrapers","seriesPart":5,"sections":[{"heading":"Introduction","paragraphs":["In part 5 we take the ideas from earlier instalments and apply them to a larger, messier site. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them back on every later request."]},{"heading":"Common pitfalls","paragraphs":["Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.","Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist."]},{"heading":"Going further","paragraphs":["Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.","Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would."]},{"heading":"Key takeaway","paragraphs":["Use a session object (requests.Session, a Playwright context) for the whole run, and check for redirects to the login page as a sign the session has expired."]}],"authorId":12,"authorName":"Yuki Andersson","authorSlug":"yuki-andersson","category":"Scraping Basics","tags":["ethics","javascript","scrapy"],"publishedAt":"2023-11-30T00:00:00.000Z","wordCount":171,"readingTimeMinutes":2}}