For part 7, we revisit the basics with a fresh example and a few lessons learned the hard way. Cross-site request forgery tokens are one of the first things that trip up a script trying to log in to a site: the form posts fine in a browser but fails with a 403 from code.

Common pitfalls

Selectors are the most fragile part of any scraper. Prefer stable attributes such as data-testid or semantic structure over generated class names, which can change on every deploy.

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

What to watch for

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Key takeaway

Load the form page with a session that keeps cookies, read the hidden token field, and send it back with the same session in the same request.