This is part 4 of our series, and this time we focus on edge cases readers asked about. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them back on every later request.

Putting it into practice

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.

Common pitfalls

Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Key takeaway

Use a session object (requests.Session, a Playwright context) for the whole run, and check for redirects to the login page as a sign the session has expired.