This is part 8 of our series, and this time we focus on edge cases readers asked about. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option.
What to watch for
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.
Going further
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Key takeaway
Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way.