<?xml version="1.0" encoding="UTF-8"?>
<response><data><post><id>103</id><slug>respecting-robots-txt-in-practice-part-6-103</slug><title>Respecting robots.txt in practice, part 6</title><excerpt>In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site…</excerpt><seriesName>Respecting robots.txt in practice</seriesName><seriesPart>6</seriesPart><sections><item><heading>Introduction</heading><paragraphs><item>In part 6 we take the ideas from earlier instalments and apply them to a larger, messier site. robots.txt is a short text file with a long history. It tells automated clients which parts of a site the operator would prefer they left alone, and how quickly they should crawl the rest.</item></paragraphs></item><item><heading>What to watch for</heading><paragraphs><item>Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.</item><item>Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.</item></paragraphs></item><item><heading>Putting it into practice</heading><paragraphs><item>Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.</item><item>Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.</item></paragraphs></item><item><heading>Key takeaway</heading><paragraphs><item>Fetch robots.txt before you crawl, honour Disallow rules and Crawl-delay values, and re-check it periodically because sites change their policies.</item></paragraphs></item></sections><authorId>3</authorId><authorName>Priya Nakamura</authorName><authorSlug>priya-nakamura</authorSlug><category>Ethics &amp; Law</category><tags><item>http</item><item>ethics</item><item>javascript</item><item>python</item></tags><publishedAt>2025-02-26T00:00:00.000Z</publishedAt><wordCount>208</wordCount><readingTimeMinutes>2</readingTimeMinutes></post></data></response>
