Continuous Delivery & Testing

continuous delivery testing

Continuous delivery is the future, and your QA testing system needs to be designed exquisitely to keep up. It’s not enough to just focus on your code–you also need a strong test plan in place so that you can easily and effectively automate your deployments, without bugs–or the test suite itself–getting in the way.

Why Continuous Delivery is Important

Continuous delivery is an approach to software development that emphasizes short software development life cycles and typically uses a highly automated deployment process. Rather than an agile methodology that works in sprints of two weeks or so and batches new code into larger and less frequent patches, continuous delivery integrates new code as and when it passes testing. This approach to testing and delivery gets feedback to developers faster and allows teams to respond to user feedback much more quickly than with agile development and testing practices.

From a business perspective, agile is no longer the fastest or most efficient approach to software development and testing. While releasing every two weeks was once considered fast, in contemporary software, engineering teams may commit dozens of times in a single day. This faster pace often requires some retooling and rethinking of how teams conduct their testing, integration, and deployment process.

The continuous delivery approach is not without its own challenges and caveats that need to be anticipated when looking to make the switch. This article will cover some of the most common pitfalls teams make when looking to switch to continuous delivery and some best practices you can implement to help make the transition as smooth as possible.

The Challenges of Continuous Delivery Testing

Teams looking to transition from an agile testing environment to a CI/CD testing environment will likely face a number of challenges along the way. Anticipating these challenges ahead of time and developing solutions for them is important, because the agile approach to testing will face a number of issues when simply ported to a CI/CD pipeline.

Testing gets in the way because it takes too long to run

One of the most immediate stumbling blocks teams face when making the transition from an agile development and delivery process to continuous delivery is the mismatch in terms of test suite run times. Agile-ready test suites may take multiple hours to run, which is not a problem in a fully agile environment where testing is only run once every two weeks. However, it immediately becomes a pressing issue in a continuous delivery environment, where test suites are expected to take around ten minutes to complete, instead of several hours.

Agile testing simply does not emphasize speed when it comes to test suite run times. This lack of concern for how long tests take to run often leads teams to favor creating as many tests as possible due to the assumption that more tests = better coverage = better software. This approach to testing simply doesn’t work with continuous delivery. Deployments may be run multiple times a day in a continuous delivery environment and each of these deployments may require the full test suite being run against it. There is simply not enough time in the day for agile test suites to handle the pace of deployment expected in continuous delivery environments.

E2E Testing is disruptive and flaky

End-to-end (E2E) testing can be particularly problematic for teams moving to continuous delivery. E2E tests are typically the most disruptive tests in the testing process, as they are often flaky and require constant maintenance. E2E tests deal with the end-user experience and are made to test the entire user journey, either partially or fully. This usually involves checking several different components at once or in tandem as part of a workflow, such as with testing the entire process for adding a product to the cart then logging in and checking out on an eCommerce site. The creation and maintenance of these tests is hard given the constant software updates and UI/UX changes that are made.

It can often result in failed tests that require manual intervention to fix. In an agile environment, the testing team can take the time necessary to fix the issues and update the tests, but in a continuous delivery environment, having to constantly fix failed tests completely defeats the purpose of having a testing and deployment process capable of handling dozens of deployments per day.

You can’t test early in the SDLC

There is another benefit to having a rapid testing and deployment process that often goes unnoticed by non-developers. Having a continuous testing process that can run in under ten minutes, it enables developers to get virtually instant feedback on their code. This is extremely valuable to developers, as it means that when they get the feedback, they may still be in the same headspace as when they wrote the code. This makes comprehending any problems and fixing them a lot easier.

In agile testing, a developer may not receive test feedback on code written until up to two weeks after they wrote it. Gaps as short as overnight can impact developer code comprehension and require them to spend time refamiliarizing themselves with particular code segments and testing issues. Requiring developers to refamiliarize themselves with old code every time they need to fix a problem impacts developer velocity and slows down the bug fixing side of the testing process.

Developers themselves are often eager to test early in the software development life cycle. In fact, some development methodologies such as test-driven development even emphasize writing tests for code before writing the code itself. Tests that developers can run to catch issues early on in development in order to avoid them growing into more difficult-to manage problems down the line, which is simply not possible with agile testing.

Fixing the Continuous Delivery Pipeline

Test suites developed in agile environments do not work well when transitioned to a continuous delivery pipeline. Agile test suites are typically bloated, attempting to cover as many possible test case scenarios as possible, and created with little concern for run time. Continuous delivery emphasizes speed and, as a rule of thumb, should take no more than ten minutes to run in full. There is a simple solution to this issue, though it may seem like a radical idea to some: run fewer tests.

This is not to say that teams should care less about QA in continuous delivery environments. What it means is that teams need to be more selective and judicious about the tests that they run. Running tests for their own sake is time-consuming, and though innocuous as any one test case may seem, in aggregate many unnecessary tests add up to create test suites that take far longer to run than necessary.

It is simply the case that, in an agile test suite that takes several hours to run, not every test is equally valuable. Time is very precious and software development life cycles are short in modern-day software development. Getting your test suite run time down from several hours to ten minutes or less requires being both ruthless and precise when selecting which tests to remove and which to keep.

Key Elements of Continuous Delivery Testing

Getting down to ten minutes or less without sacrificing your testing quality requires being judicious and analytical with how you select your tests. You can cut down on a large number of unnecessary tests by applying the following principles to each test case.

  1. Make sure that test cases only cover important processes and likely failure conditions. Testing for each and every possible event and condition, especially highly improbable scenarios, is not worth the trade-off protection it provides for the time it takes to run them. Remove test cases for scenarios that are extremely unlikely to ever actually occur in production.
  2. Classify your tests appropriately. Many teams run E2E tests that should be unit tests. Things such as testing that a particular value gets returned to the screen after a function is run. That is an ideal case for a unit test rather than an E2E test. E2E tests are slow and expensive, so only use them when you absolutely have to. If a test is better suited to being a unit test, then make it so. You don’t need to spin up a browser to run a unit test.
  3. Recognize the tests that are delivering value. These are the tests that are covering features and functions that are important to the business process and frequently used by your users. They cover scenarios that are likely to occur and help to prevent major issues from ever reaching production. These are the tests that are worth keeping.

Finding out what features your users are using and in which ways they’re being used can be difficult. Many marketing teams spend a lot of time and effort in order to understand their users and gather data from users in real-time in order to understand their behavior and journey through their software.

At ProdPerfect our solution to testing works very similarly. By analyzing your users in real-time, you can gather hard data about user behavior. This removes the guesswork from the test development process. Instead of creating tests around scenarios you think your users are likely to face, you can develop tests around scenarios you know your users face. This is used to guide the testing process, creating tests that cover real scenarios and use cases.

Get Your CI/CD Testing Locked Down

Teams need to be ruthless when making the switch from agile to CI/CD testing. You need to anticipate the likely pain points ahead of time in order to mitigate them without everything grinding to a halt. The most immediate issue for teams will likely be test run times.

You need to be aggressive in your approach to reduce the number of tests in your test suite. Although it is necessary to reduce test run times to 10-20 minutes (or less), it shouldn’t be done without precise knowledge of the value each test case provides. This can be difficult to determine; data lights the way.

If your team is looking to make the transition from agile to CI/CD testing and you think you might need help, contact us. Our approach to test development involves using real user data to create tests based on the way your users actually use your software in production. This provides a data-driven approach to testing that gives you the confidence to know that the tests you keep are the ones that provide real value to your software. This cuts down on your test suite run times and lets your team take a CI/CD approach to development, keeping your team competitive in the modern software development marketplace.