This is part 2 of our series, and this time we focus on edge cases readers asked about. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password, locked account, slow responses and expired forms.
Step by step
Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.
Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.
A practical approach
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Pagination comes in several shapes: numbered pages, next links, offsets and opaque cursors. Each has trade-offs, and a robust scraper should detect the end of the result set rather than assuming a fixed number of pages.
Key takeaway
Cover each account state with its own test, assert on visible messages and on the resulting URL, and give slow paths an explicit, generous timeout.