In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure early on saves a rewrite later.
Going further
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Putting it into practice
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Key takeaway
Separate fetching, parsing and storing into stages, keep raw responses, make each stage re-runnable, and validate records before they reach storage.