This is part 8 of our series, and this time we focus on edge cases readers asked about. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page immediately sees placeholders instead of content.
Putting it into practice
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Common pitfalls
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Key takeaway
Scroll the element into view and wait for its real attributes to appear, or read the data-src style attributes that hold the eventual values.