{"data":{"id":40,"slug":"when-to-use-an-api-instead-of-scraping-part-2-40","title":"When to use an API instead of scraping, part 2","excerpt":"Part 2 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less…","seriesName":"When to use an API instead of scraping","seriesPart":2,"sections":[{"heading":"Introduction","paragraphs":["Part 2 looks at what changes when the same technique has to run every day in production. If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option."]},{"heading":"A practical approach","paragraphs":["Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.","Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages."]},{"heading":"Putting it into practice","paragraphs":["Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.","Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would."]},{"heading":"Key takeaway","paragraphs":["Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way."]}],"authorId":10,"authorName":"Aisha Rossi","authorSlug":"aisha-rossi","category":"Ethics & Law","tags":["http","python","playwright","scrapy"],"publishedAt":"2025-07-13T00:00:00.000Z","wordCount":183,"readingTimeMinutes":2}}