Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep is either too short on a slow day or wastes time on a fast one.
Going further
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
What to watch for
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Key takeaway
Replace sleeps with explicit waits for a specific condition: element visible, text present, or network idle, with a sensible timeout.