This is part 4 of our series, and this time we focus on edge cases readers asked about. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page immediately sees placeholders instead of content.
Common pitfalls
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Going further
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Key takeaway
Scroll the element into view and wait for its real attributes to appear, or read the data-src style attributes that hold the eventual values.