This is part 10 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep is either too short on a slow day or wastes time on a fast one.

Putting it into practice

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.

Step by step

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Key takeaway

Replace sleeps with explicit waits for a specific condition: element visible, text present, or network idle, with a sensible timeout.