What is Regression Testing and Why is It Important?

An ounce of prevention is worth a pound of cure.”  -Benjamin Franklin

This article is for everyone that asks, “why would you want to keep testing the features that are already live? Shouldn’t we focus on testing new features?” Those of you who already know the importance of testing both can probably focus on other content instead.

Testing new features that go live is certainly critical to making sure your application works. Brand new features should be tested extensively, not only for functionality, but also for user experience, conversion (as appropriate), performance, etc.

Your current feature set needs to be tested for regressions. regression is what it sounds like if you took statistics in college: your application regresses from its current state to something worse. It is deviation from expected state. This happens in one of two ways:

  1. You’re maintaining, altering, fixing, or improving a feature (rather than introducing a new one) and break it.
  2. You’re changing just about anything and end up breaking something completely different in the application.

The first form of regression is fairly obvious; the second can be a head-scratcher. The short version of why this can happen: almost any application is deeply interconnected. There’s a concept called DRY – “Don’t Repeat Yourself.” Good developers don’t copy code; rather, they make that code accessible to all features that touch it. Any area of an application depends on many others to function properly. If you break something while working on inventory management, you might wreck checkout. Updating infrastructure might break login. It’s not that every change can truly affect every part of the application, but any change might impact multiple parts of the application if a bug is introduced.

Regression testing, therefore, tests to make sure any of these regressions are caught before they make it to production. Generally, you run a suite of regression tests by testing every unit of code, every API, and core user pathway across the application at the browser level, with every build. Some teams can’t run their regression suite fast enough (or are using manual or crowdsourced testing, which has incremental cost per test run because you’re throwing extra bodies at the problem) that they run their browser regression testing suite on a less-frequent schedule. The obvious downside of this is that you’re more likely to let regressions into production before they are caught.

Automation vs. Manual Testing

If you’re considering automating your browser regression suite, start with features that are least likely to be changing rapidly: building automation takes time to pay off, so you want to make sure that these tests are going to be run more than a few times before they need to be re-written. For brand new or rapidly-evolving features, manual testing may be your most efficient approach.

When you do decide to automate: automated browser regression testing suites are built in a number of ways. The most basic is scripting them in SeleniumCypressCapybara, with Javascript in TestCafe, or using other such frameworks. They can also be built using record-and-play tools such as Selenium / TestCafe IDE. Machine Learning is making record-and-play stronger and less time-intensive, and will eventually allow record-and-play to drive itself using web traffic data.

What is Continuous Testing? (And Why Should I Care?)

“Move Fast and Break Things” – Facebook’s Old and Very Catchy Developer Motto

“Move Fast with Stable Infra” – Facebook’s New, Less Catchy Developer motto

Continuous testing is the holy grail of software quality assurance. Get it right, and you get to both move fast and also not break things. But it’s not only the holy grail because it will give your application everlasting life quality, but because it’s elusive, requires a lot of work—work which might end up killing you (think: Indiana Jones and the Last Crusade).

Continuous Testing Complements Development

More specifically, it’s a practice that pairs with continuous development (whether or not your team is deploying continuously). Briefly, continuous testing means that you’re developing continuously and testing every commit as you develop. Slightly more specifically, in continuous development, every developer should be committing fairly small changes, very frequently. Every time a developer commits work to remote, a regression testing suite (at the Unit, API, and E2E levels) runs to make sure nothing in the application has broken. This testing can either occur on a feature branch (perhaps during a pull request) or after the feature branch has been merged on a pre-production environment. If the developer is modifying or introducing workflows, each workflow also gets tested end-to-end. That’s continuous testing.

Here’s why (beyond the obvious benefits of testing generally) it’s so great: your developer has just committed code when it gets tested. Not only have they just committed it, but what they committed is necessarily a fairly small amount of code. Your developers, therefore, get instantaneous feedback while they are still contextually aware of what they just wrote—it’s fresh. And because it’s a small amount of code, there are only so many places to look to find out what went wrong. Your developers can thereby rapidly find, understand, and fix the bug. It doesn’t get to production, it doesn’t make it to a JIRA list, and it doesn’t languish for weeks. It gets fixed in minutes, before production, and generally becomes a non-event.

“Dan, how do I drink the soul-nourishing waters from this mythical cup?”

First, you need to have the infrastructure and culture to support continuous deployment. This is hard and we don’t want to trivialize it, but we won’t focus on it here. If you have the infrastructure and culture to support continuous deployment, then you’re ready for continuous testing.

The biggest challenge to continuous testing specifically is that your tests need to be comprehensive in scope yet also run quickly. These tests ideally need to cover the entire application, and they also need to finish running in minutes. It would conventionally seem like you need to choose between runtime and comprehensiveness, but focus can get you both.

Improving Browser Testing Runtime

By far the slowest part of your testing suite will be your browser tests, so this is where you need to do the most work to limit the number and parallelize. We discuss how to assess which user flows to focus on for browser (E2E) testing here, but in brief: focus on the core workflows your users are actually following frequently in your application. This maximizes bang-per-buck and lets you cover everything important without extending runtime so much that it gets in the way of continuous testing. After you’ve built the E2E tests you want, parallelize the ones you can, using a parallel-friendly framework such as TestCafe.

What if my Needs Are Complex?

If you want to test for edge cases, test cross-browser and cross-device, or otherwise have a very large browser testing suite, you may need to prioritize which tests get run with every build and which tests get run out of band, perhaps every few hours. This will allow you to make sure no regressions get to production that would affect core user flows, and any edge cases or low-impact regressions are caught a few hours later at worst.

All of this requires a lot of work even when you have “made it.” It can feel like a truly gargantuan task to get there if you’re not close. Read on, and we’ll be able to help you break the process into discrete steps.