In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Running a browser without a window is faster and works on servers, but some problems are much easier to understand when you can watch the browser do its work.
What to watch for
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Putting it into practice
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.
Key takeaway
Develop headed with slow motion turned on, run headless in CI, and capture screenshots or traces on failure so headless runs are still debuggable.