Part 8 looks at what changes when the same technique has to run every day in production. Running a browser without a window is faster and works on servers, but some problems are much easier to understand when you can watch the browser do its work.

A practical approach

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Going further

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Key takeaway

Develop headed with slow motion turned on, run headless in CI, and capture screenshots or traces on failure so headless runs are still debuggable.