An ounce of prevention is worth a pound of cure.” -Benjamin Franklin
This article is for everyone that asks, “why would you want to keep testing the features that are already live? Shouldn’t we focus on testing new features?” Those of you who already know the importance of testing regressions for both can probably focus on other content instead.
Testing new features that go live is certainly critical to making sure your application works. Brand new features should be tested extensively, not only for functionality, but also for user experience, conversion (as appropriate), performance, etc.
Your current feature set needs to be tested for regressions. A regression is what it sounds like if you took statistics in college: your application regresses from its current state to something worse. It is deviation from expected state. This happens in one of two ways:
- You’re maintaining, altering, fixing, or improving a feature (rather than introducing a new one) and break it.
- You’re changing just about anything and end up breaking something completely different in the application.
The first form of regression is fairly obvious; the second can be a head-scratcher. The short version of why this can happen: almost any application is deeply interconnected. There’s a concept called DRY – “Don’t Repeat Yourself.” Good developers don’t copy code; rather, they make that code accessible to all features that touch it. Any area of an application depends on many others to function properly. If you break something while working on inventory management, you might wreck checkout. Updating infrastructure might break login. It’s not that every change can truly affect every part of the application, but any change might impact multiple parts of the application if a bug is introduced.
Regression testing, therefore, tests to make sure any of these regressions are caught before they make it to production. Generally, you run a suite of regression tests by testing every unit of code, every API, and core user pathway across the application at the browser level, with every build. Some teams can’t run their regression suite fast enough (or are using manual or crowdsourced testing, which has incremental cost per test run because you’re throwing extra bodies at the problem) that they run their browser regression testing suite on a less-frequent schedule. The obvious downside of this is that you’re more likely to let regressions into production before they are caught.
Automation vs. Manual Testing
If you’re considering automating your browser regression suite, start with features that are least likely to be changing rapidly: building automation takes time to pay off, so you want to make sure that these tests are going to be run more than a few times before they need to be re-written. For brand new or rapidly-evolving features, manual testing may be your most efficient approach.