This is part 7 of our series, and this time we focus on edge cases readers asked about. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.

Common pitfalls

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

What to watch for

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Key takeaway

Use a proper CSV writer rather than joining strings, quote fields that contain commas or newlines, write UTF-8, and keep dates in ISO 8601.