Regression testing is likely nobody’s favorite type of testing, but it is a necessity for developers to ensure quality as code evolves over time. This is particularly important in agile development, with clearly defined sprints and development cycles that do not leave much room for dealing with unexpected occurrences, such as bugs being introduced in otherwise unrelated areas of code. When regression testing is done correctly, it helps to prevent the introduction of new bugs into existing code, which in turn helps maintain the smooth, and therefore rapid, continuation of development within each sprint.
Correctly implementing regression testing into the agile methodology requires forethought and advance planning, as contemporary regression testing relies heavily on automation and continuous execution, which work best with a streamlined test suite. This can go against the testing philosophy of many agile development projects, where the test cycle is simply incorporated into the sprint phase, whether it takes hours or days. Having a regression testing strategy can help teams smoothly incorporate regression testing into their agile development methodology, without sacrificing speed or quality.
Automated Regression Testing
Automation helps cut down on a lot of the tedium and repetition associated with regression testing. This is very important because regression tests should be run against every deployment (or, ideally, every commit). This is the best way to ensure that new bugs are not introduced into existing code, but it also creates the problem of long test suite runtimes and the bloating of test suites over time.
Test suite bloat and long runtimes are not usually considered high priority concerns within agile teams. When a testing cycle is simply part of each sprint process, it can be assigned a length of time equal to what it takes to fully run a test suite, with little pressure on reducing overall test runtime. In some cases, this can mean test suites take hours or even days to run. This creates several problems–one pertinent example is that it creates a gap in test feedback turnaround time. Long test suites can only be run at the end of a sprint, meaning it could be nearly 2 weeks between when a bug is introduced and when it is caught. The time it takes to identify an issue and get that feedback back to a developer can have a significant influence on how quickly an issue gets solved.
Another issue teams may run into is when tests are bundled together into a monolithic test suite that is run once during the testing phase of a sprint. This means missing opportunities to catch bugs and defects earlier on and can obfuscate the source of an issue, making it more difficult to isolate any one particular issue from another, not to mention find the developer who introduced the bug.
While introducing automation to regression testing can and often does reduce test suite runtime and help catch bugs sooner, the tests contained within need to be integrated with careful consideration for factors such as runtime and bloat. When tests are run often, a small amount of bloat can very easily create a large amount of cumulative debt over the long run. Avoiding this bloat is the key to successfully integrating automated regression testing into agile development.
This bloat can be cut down in one of several ways. Rather than adding every single test case into a monolithic test suite, instead, take time to break up your test runs either by test case or by test priority. Test cases can be split several ways, such as by functionality or speed. Test prioritization involves identifying which test cases are highly important–are providing quality value to a project–and which are not. Rather than testing everything, streamline test suites to only include valuable tests– those that cover core functionality or highly-trafficked components. Superfluous tests or those that cover rarely-used use cases can be de-prioritized or even discarded: unit testing and good feature flagging should suffice for edge cases
Regression Testing for Agile Teams
Regression testing helps teams to verify that new features or modifications introduced during a sprint do not break functionality or introduce bugs into an existing codebase. One of the biggest challenges agile teams face with regression testing is maintaining a reasonable test runtime. Shorter test runtimes mean more frequent testing, so there are a number of strategies teams may choose to follow in order to maintain a reasonable test runtime.
The first is to determine the overall approach that will be taken. Teams may choose to either run E2E testing at the sprint level, where only test cases added since the most recent release are run. Or, they may choose full end-to-end regression testing, where all core functionality and critical components are tested. Which one teams choose will depend on numerous factors, but the former method is generally advisable only when a product is already known to be highly stable, the features are architecturally highly isolated, or a project is so new that full end-to-end regression testing is not yet possible.
Traditionally in agile development, regression testing would be run after the development phase of a sprint, towards the end of the testing phase of every sprint. The drawbacks of this approach have already been noted, but there are some alternatives. One is to implement a delayed sprint approach, whereby regression testing is not done at the end of every sprint, but rather done once all functionality is complete and ready for testing.
Another is to simply stagger regression testing by a defined period, such as a week, to allow adequate time to ensure code is stable and ready for testing before regression testing is run. Both of these approaches have their advantages and they can help reduce overall time spent testing, but they also come with their own disadvantages, such as increasing the delay between running tests and getting feedback to developers, and failing to identify what commit contributed to the bug.
Ultimately, reducing test runtime without sacrificing test quality, increasing feedback delay, or impacting developer velocity requires careful planning and strategizing. Adopting a successful test strategy will be project dependent, but generally means choosing one that minimizes the drawbacks these approaches offer while maximizing the benefits.
Building Your Agile Regression Test Strategy
Developing a successful regression testing strategy that works in an agile development environment means developing an approach to testing that is faster to run and more granular than traditional monolithic test suites. Automating regression testing is the key to maintaining both development velocity and test quality. Introducing automation into regression testing requires much more careful consideration of what actually needs to be tested and what tests should be discarded.
Removing test cases may seem antithetical to regression testing, the purpose of which is to maintain consistency in code over time by ensuring old functionality continues to work after the introduction of new code. However, a “just keep adding tests” approach will simply take too long when integrated into modern, automated continuous integration systems. Test suite runtimes should be measurable in minutes, not hours or days. Cutting down on test suites to reach these figures without sacrificing test quality means taking a very different approach to testing than most agile teams are used to.
Determining which tests are relevant can be tricky. This process is often seen as subjective, but there are some objective measurements that you can use as part of a data-driven approach to quantifying test value. Tests should be monitored and analyzed for their use and value alongside metrics such as how frequently test cases are used and the value of the functionality the tests protect against. Using hard data such as user behavior or app traffic can help determine where your users are actually spending most of their time and where it makes the most sense to spend your testing runtime budget.
QA leaders should start strategizing before the testing phase begins. Focus on these high-risk, high-value areas and develop additional sanity testing checks for the most critical application components. These tests should be quick to run but they should ensure that the most vital components are always functioning as expected. Testers should work with developers each cycle to understand the changes being made to the product or requirements and focus on high-risk areas within these first, such as the disruption or alteration of core functionality. Generally speaking, the earlier you can prepare before the testing phase officially begins, the longer you have to actually execute regression testing and respond to known issues during the testing phase itself.
Regression Testing and The Agile Methodology: the Big Picture
The agile methodology is a battle-tested approach to software development that works, but it’s slow and the segmented nature of phases during sprints can lead to problems with developing contemporary regression testing strategies that can keep pace with modern-day software development life cycles. Treating the testing phase of a sprint as a monolithic entity that is only worked on during its defined phase and takes as long as it needs to, will leave teams behind compared to those that utilize more efficient, modern approaches to regression testing.
Regression testing is important and it does not have to clash with the agile approach to software development. This article has covered some of the most important considerations that need to be taken into account when incorporating regression testing into an agile environment. Break down monolithic test suites into smaller components, analyze your tests, identify those that provide value from those that don’t and categorize tests into suites that can be run at different stages of a sprint depending on their use case and priority.
Quantifying test value can be difficult for many teams, so if you are struggling with identifying your most critical tests from less vital ones, then get in touch with a member of ProdPerfect. We can help you analyze and understand your test suite and to help quantify the value of your test cases based on where your testing is most beneficial and where it is least necessary.