In part 9 we take the ideas from earlier instalments and apply them to a larger, messier site. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site: the form posts fine in a browser but fails with a 403 from code.
Putting it into practice
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Time zones matter more than they seem. Store timestamps in UTC and convert only when displaying them.
Before writing a single line of code, look at how the page actually loads its data. Open the network panel, reload, and note which requests return the content you care about. Very often the data you want arrives as JSON from an endpoint you can call directly.
Going further
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
Deduplicate as you go. The same item often appears on several listing pages, and a stable identifier such as a SKU or ID makes duplicates easy to spot.
Key takeaway
Load the form page with a session that keeps cookies, read the hidden token field, and send it back with the same session in the same request.