{"data":{"id":60,"slug":"when-to-use-an-api-instead-of-scraping-part-3-60","title":"When to use an API instead of scraping, part 3","excerpt":"For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and…","seriesName":"When to use an API instead of scraping","seriesPart":3,"sections":[{"heading":"Introduction","paragraphs":["For part 3, we revisit the basics with a fresh example and a few lessons learned the hard way. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option."]},{"heading":"Going further","paragraphs":["Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.","Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.","Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would."]},{"heading":"A practical approach","paragraphs":["Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it."]},{"heading":"Key takeaway","paragraphs":["Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way."]}],"authorId":12,"authorName":"Yuki Andersson","authorSlug":"yuki-andersson","category":"Ethics & Law","tags":["javascript","json","http","scrapy"],"publishedAt":"2025-05-30T00:00:00.000Z","wordCount":208,"readingTimeMinutes":2}}