Working with CSRF tokens, part 8
This is part 8 of our series, and this time we focus on edge cases readers asked about. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site:…
Showing 37–48 of 67 articles
This is part 8 of our series, and this time we focus on edge cases readers asked about. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site:…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little…
This is part 2 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option.
Part 2 looks at what changes when the same technique has to run every day in production. Lazy loading defers images and sections until they are about to scroll into view. A script that reads the page…
For part 2, we revisit the basics with a fresh example and a few lessons learned the hard way. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
For part 10, we revisit the basics with a fresh example and a few lessons learned the hard way. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those…
Part 5 looks at what changes when the same technique has to run every day in production. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those headers…
In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…
This is part 8 of our series, and this time we focus on edge cases readers asked about. Scrapy makes it very easy to send thousands of requests a minute. Making a spider that a site operator would be…
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. Networks fail, servers restart and rate limits kick in. A scraper that gives up at the first error is…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Infinite scroll replaces the familiar Next button with a page that keeps growing as you move down it, which…