CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.

Common pitfalls

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.

Step by step

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.

Key takeaway

Use a proper CSV writer rather than joining strings, quote fields that contain commas or newlines, write UTF-8, and keep dates in ISO 8601.