A selector that worked yesterday and fails today is usually a selector that depended on something nobody promised to keep the same, such as a generated class name or the position of a div.
What to watch for
Treat HTTP 429 responses as a signal, not an obstacle. Read the Retry-After header, back off, and resume at a slower pace.
Keep configuration such as base URLs, delays and selectors out of your code. Changing a delay should not require a redeploy.
Write small, focused tests for your parsers using saved HTML fixtures. They run in milliseconds and catch layout changes long before a full crawl would.
Step by step
Store raw responses alongside parsed output while you are developing. When a parser breaks, you can replay the saved HTML instead of hitting the site again.
Always identify your client honestly with a descriptive User-Agent, and keep your request rate modest. A crawler that behaves like a considerate visitor is far less likely to cause problems for the site operator.
Key takeaway
Prefer data-testid attributes, ARIA roles and visible labels. Avoid long chains of nth-child and anything that looks like a hash.