Parsing HTML tables with BeautifulSoup, part 6
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…
Showing 73–84 of 200 articles
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…
Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines.
In part 10 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…
In part 8 we take the ideas from earlier instalments and apply them to a larger, messier site. Offset pagination asks for 'items 40 to 60'. Cursor pagination asks for 'the next page after this…
For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to…
This is part 5 of our series, and this time we focus on edge cases readers asked about. A one-off script grows into a pipeline the moment someone asks for the data again next week. A little structure…
Part 2 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. CSV is the lingua franca of spreadsheets and the most common way scraped data leaves a project. It is…
In part 2 we take the ideas from earlier instalments and apply them to a larger, messier site. Many product pages already contain a clean, structured description of themselves in a script tag meant…
For part 5, we revisit the basics with a fresh example and a few lessons learned the hard way. A selector that worked yesterday and fails today is usually a selector that depended on something nobody…
This is part 10 of our series, and this time we focus on edge cases readers asked about. Most flaky Selenium tests are timing problems in disguise. The element exists, just not yet, and a fixed sleep…
For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. A well-behaved API tells you how many requests you have left before it starts saying no. Reading those…