In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page immediately sees placeholders instead of content.
What to watch for
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Going further
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
Key takeaway
Scroll the element into view and wait for its real attributes to appear, or read the data-src style attributes that hold the eventual values.