This is part 10 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which means your script has to decide when there is nothing left to load.
Putting it into practice
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.
What to watch for
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Key takeaway
Scroll, wait for the item count to change, and stop when two consecutive scrolls add nothing new. Better still, find the JSON request the page makes and call it directly.