Exporting scraped data to CSV
CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.
Showing 1–12 of 56 articles
CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is also surprisingly easy to produce a CSV that opens incorrectly.
robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.
Part 3 looks at what changes when the same technique has to run every day in production. When the data you want is several levels deep and the markup has few useful attributes, XPath's ability to…
This is part 6 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…
In part 7 we take the ideas from earlier instalments and apply them to a larger, messier site. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. A login form is a small piece of UI with a surprising number of paths through it: success, wrong…
In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator…
This is part 3 of our series, and this time we focus on edge cases readers asked about. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this marker'. The…
For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. Before you crawl a site link by link, check whether it already publishes a list of every page it wants…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Logging in once and staying logged in is a matter of keeping the cookies the server gives you and…
A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure early on saves a rewrite later.