For part 9, we revisit the basics with a fresh example and a few lessons learned the hard way. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password, locked account, slow responses and expired forms.

Step by step

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.

Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.

Putting it into practice

Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.

Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.

Key takeaway

Cover each account state with its own test, assert on visible messages and on the resulting URL, and give slow paths an explicit, generous timeout.