{"data":{"id":120,"slug":"when-to-use-an-api-instead-of-scraping-part-6-120","title":"When to use an API instead of scraping, part 6","excerpt":"Part 6 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…","seriesName":"When to use an API instead of scraping","seriesPart":6,"sections":[{"heading":"Introduction","paragraphs":["Part 6 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option."]},{"heading":"Common pitfalls","paragraphs":["Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.","Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.","Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot."]},{"heading":"Step by step","paragraphs":["Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.","Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would."]},{"heading":"Key takeaway","paragraphs":["Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way."]}],"authorId":12,"authorName":"Yuki Andersson","authorSlug":"yuki-andersson","category":"Ethics & Law","tags":["testing","playwright","http","python"],"publishedAt":"2024-10-03T00:00:00.000Z","wordCount":200,"readingTimeMinutes":2}}