This is part 4 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option.
Step by step
Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.
A practical approach
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Key takeaway
Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way.