This is part 6 of our series, and this time we focus on edge cases readers asked about. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password, locked account, slow responses and expired forms.
Common pitfalls
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
A practical approach
Read the site's terms of service before you start. Many sites welcome automated access within limits and say so explicitly; others ask you to use an API instead.
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Key takeaway
Cover each account state with its own test, assert on visible messages and on the resulting URL, and give slow paths an explicit, generous timeout.