In part 3 we take the ideas from earlier instalments and apply them to a larger, messier site. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep is either too short on a slow day or wastes time on a fast one.

A practical approach

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Step by step

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Key takeaway

Replace sleeps with explicit waits for a specific condition: element visible, text present, or network idle, with a sensible timeout.