{"data":{"id":101,"slug":"parsing-html-tables-with-beautifulsoup-part-6-101","title":"Parsing HTML tables with BeautifulSoup, part 6","excerpt":"For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that…","seriesName":"Parsing HTML tables with BeautifulSoup","seriesPart":6,"sections":[{"heading":"Introduction","paragraphs":["For part 6, we revisit the basics with a fresh example and a few lessons learned the hard way. Tables look like the easiest thing on a page to scrape, until you meet merged cells, header rows that repeat every twenty lines and numbers formatted for humans rather than machines."]},{"heading":"Going further","paragraphs":["Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.","Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.","Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream."]},{"heading":"Common pitfalls","paragraphs":["Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.","Log every request with its URL, status code and duration. When something goes wrong at two in the morning, those logs are the difference between a quick fix and a long investigation."]},{"heading":"Key takeaway","paragraphs":["Locate the table by a stable attribute, read the header row once, and convert every cell to a typed value before you store it."]}],"authorId":7,"authorName":"Kwame Mensah","authorSlug":"kwame-mensah","category":"Scraping Basics","tags":["http"],"publishedAt":"2025-08-23T00:00:00.000Z","wordCount":228,"readingTimeMinutes":2}}