{"data":{"id":192,"slug":"session-cookies-for-scrapers-part-10-192","title":"Session cookies for scrapers, part 10","excerpt":"Part 10 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending…","seriesName":"Session cookies for scrapers","seriesPart":10,"sections":[{"heading":"Introduction","paragraphs":["Part 10 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them back on every later request."]},{"heading":"Putting it into practice","paragraphs":["Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.","Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.","Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it."]},{"heading":"Common pitfalls","paragraphs":["Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.","Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead."]},{"heading":"Key takeaway","paragraphs":["Use a session object (requests.Session, a Playwright context) for the whole run, and check for redirects to the login page as a sign the session has expired."]}],"authorId":5,"authorName":"Chen Patel","authorSlug":"chen-patel","category":"Scraping Basics","tags":["selenium","python","javascript"],"publishedAt":"2026-02-09T00:00:00.000Z","wordCount":206,"readingTimeMinutes":2}}