This is part 2 of our series, and this time we focus on edge cases readers asked about. Running a browser without a window is faster and works on servers, but some problems are much easier to understand when you can watch the browser do its work.

Putting it into practice

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

Step by step

Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Key takeaway

Develop headed with slow motion turned on, run headless in CI, and capture screenshots or traces on failure so headless runs are still debuggable.