{"data":{"id":112,"slug":"session-cookies-for-scrapers-part-6-112","title":"Session cookies for scrapers, part 6","excerpt":"Part 6 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them…","seriesName":"Session cookies for scrapers","seriesPart":6,"sections":[{"heading":"Introduction","paragraphs":["Part 6 looks at what changes when the same technique has to run every day in production. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them back on every later request."]},{"heading":"Putting it into practice","paragraphs":["Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.","Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.","Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot."]},{"heading":"Common pitfalls","paragraphs":["Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.","Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator."]},{"heading":"Key takeaway","paragraphs":["Use a session object (requests.Session, a Playwright context) for the whole run, and check for redirects to the login page as a sign the session has expired."]}],"authorId":3,"authorName":"Priya Nakamura","authorSlug":"priya-nakamura","category":"Scraping Basics","tags":["playwright","testing","ethics","json"],"publishedAt":"2025-03-25T00:00:00.000Z","wordCount":237,"readingTimeMinutes":2}}