Part 5 looks at what changes when the same technique has to run every day in production. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.
Putting it into practice
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Step by step
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Key takeaway
Use a proper CSV writer rather than joining strings, quote fields that contain commas or newlines, write UTF-8, and keep dates in ISO 8601.