This is part 6 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep is either too short on a slow day or wastes time on a fast one.
Putting it into practice
Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Step by step
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Key takeaway
Replace sleeps with explicit waits for a specific condition: element visible, text present, or network idle, with a sensible timeout.