{"data":{"id":20,"slug":"when-to-use-an-api-instead-of-scraping-20","title":"When to use an API instead of scraping","excerpt":"If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option.","seriesName":"When to use an API instead of scraping","seriesPart":1,"sections":[{"heading":"Introduction","paragraphs":["If a site offers an API for the data you need, scraping its HTML is usually the harder, more fragile and less polite option."]},{"heading":"Putting it into practice","paragraphs":["Be careful with relative URLs. Resolve every link against the page it came from, not against the site root, or you will request pages that do not exist.","Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it."]},{"heading":"Common pitfalls","paragraphs":["Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.","Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace."]},{"heading":"Key takeaway","paragraphs":["Check for a documented API, a data export or a feed first. Scrape only when none exists, and read the terms of service either way."]}],"authorId":11,"authorName":"Oliver Kim","authorSlug":"oliver-kim","category":"Ethics & Law","tags":["scrapy","ethics","javascript"],"publishedAt":"2024-10-26T00:00:00.000Z","wordCount":146,"readingTimeMinutes":2}}