Part 3 looks at what changes when the same technique has to run every day in production. Many product pages already contain a clean, structured description of themselves in a script tag meant for search engines. Reading it is often more reliable than parsing the visible HTML.
What to watch for
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Step by step
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Key takeaway
Select script[type='application/ld+json'], parse every block, and pick the one whose @type matches what you need. Fall back to HTML only for missing fields.