Part 4 looks at what changes when the same technique has to run every day in production. A login form is a small piece of UI with a surprising number of paths through it: success, wrong password, locked account, slow responses and expired forms.

Putting it into practice

Dynamic pages render content after the initial HTML arrives. A browser automation tool can wait for a specific element to appear, which is far more reliable than sleeping for a fixed amount of time.

Validate the data you extract. Prices without currencies, dates without time zones and empty fields are common sources of silent errors downstream.

A practical approach

Character encodings still cause trouble. Decode responses using the charset the server declares, and normalise text before comparing or storing it.

Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.

Key takeaway

Cover each account state with its own test, assert on visible messages and on the resulting URL, and give slow paths an explicit, generous timeout.