Exporting scraped data to CSV
CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.
Developer Advocate
Amelia writes about practical data extraction and test automation.
14 articles
CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
This is part 6 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
This is part 6 of our series, and this time we focus on edge cases readers asked about. Running a browser without a window is faster and works on servers, but some problems are much easier to…
This is part 2 of our series, and this time we focus on edge cases readers asked about. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password,…
Logging in once and staying logged in is a matter of keeping the cookies the server gives you and sending them back on every later request.
For part 8, we revisit the basics with a fresh example and a few lessons learned the hard way. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
Part 7 looks at what changes when the same technique has to run every day in production. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is fragile;…
This is part 8 of our series, and this time we focus on edge cases readers asked about. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
Part 7 looks at what changes when the same technique has to run every day in production. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers turns rate limiting from a surprise into a plan.