In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure early on saves a rewrite later.
What to watch for
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.
Putting it into practice
Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.
Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.
Key takeaway
Separate fetching, parsing and storing into stages, keep raw responses, make each stage re-runnable, and validate records before they reach storage.