<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>146</id><slug>waiting-strategies-in-selenium-part-8-146</slug><title>Waiting strategies in Selenium, part 8</title><excerpt>Part 8 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…</excerpt><seriesName>Waiting strategies in Selenium</seriesName><seriesPart>8</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>Part 8 looks at what changes when the same technique has to run every day in production. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep is either too short on a slow day or wastes time on a fast one.</item></paragraphs></item><item><heading>Going further</heading><paragraphs><item>Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.</item><item>Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.</item><item>Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.</item></paragraphs></item><item><heading>Step by step</heading><paragraphs><item>Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.</item><item>Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.</item><item>Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Replace sleeps with explicit waits for a specific condition: element visible, text present, or network idle, with a sensible timeout.</item></paragraphs></item></sections><authorId>8</authorId><authorName>Hannah Novak</authorName><authorSlug>hannah-novak</authorSlug><category>Browser Automation</category><tags><item>selenium</item><item>scrapy</item><item>http</item></tags><publishedAt>2024-02-28T00:00:00.000Z</publishedAt><wordCount>234</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
