Erik Fogg

Continuous Integration Testing and Agile Development

continuous integration testing

Continuous Integration testing, or testing your code in a CI/CD footing, is one of the most common roadblocks to making a full transition from conventional agile development. The differences in both the testing processes and their cadence can trip up inexperienced teams looking to make the transition. Here’s what we know about how you should approach it, and how to make it work in your DevOps framework.

What is Continuous Integration? How Does it Impact Testing?

Continuous integration (CI) refers to a software development and testing process whereby new code commits are tested and integrated into a codebase as they are committed on a continuous basis, as opposed to having discrete development and testing stages. Very often, new code commits are pushed multiple times per day, each of which often require their own quality testing. Since code is developed and pushed continuously, it needs testing that can handle both the load, speed, and volume of commits typical of continuous integration.

Agile Development & CI/CD

Many teams choose to utilize agile development practices because they are considered battle-tested and allow teams to adapt to changes more quickly than more traditional development practices like waterfall. As you know, agile development is a big leap forward from waterfall – instead of planning out your entire development process, you’re releasing every few weeks, or planning to release certain updates in smaller bites, seeing what happens, and then pivoting based on results.

Continuous integration should ideally be paired with continuous delivery, the other half of CI/CD. As continuous integration leads to new commits being added to the code base on a daily or even hourly basis, this should be paired with a continuous delivery pipeline that ships these new updates to end users on a regular basis, rather than at the end of every month or quarter. This brings critical improvements, new features, bug fixes, etc, to market much faster, and with less disruption to the codebase and user experience, leading to much faster learning, iteration, and ultimately improvement of the product.

There are advantages to agile development. It captures user feedback, which can be very valuable data that allows teams to pivot in response to their target audience. However, agile development doesn’t play nice with CI/CD. It’s simply too slow to keep up with the pace of development in modern-day software development environments, where the software development life cycle is becoming ever shorter. Teams that take weeks between software releases will soon find themselves outpaced and uncompetitive in an industry where software updates may be pushed multiple times a day.

The Role of a CI Toolset in Agile

The aim of continuous integration and automated deployment is to optimize the testing and delivery process by creating an automated pipeline that takes new commits, automatically tests them and pushes commits that successfully pass testing to live. This delivery system requires the right toolset in place to make this a smooth and automatic process, but for teams that take the time to implement it, it can significantly streamline the delivery process.

Teams utilizing agile development practices should find continuous integration somewhat familiar territory. Both encourage flexibility and emphasize being able to respond to change and user feedback. However, continuous integration encourages a much more rapid pace of development and testing, which means that the right tooling needs to be in place in order to accommodate this faster pace of development and shorter software development lifecycles.

Continuous Integration Testing Bottlenecks

One of the biggest stumbling blocks many teams come across when looking into continuous delivery is testing. This is for several reasons. Agile methodology does not prioritize speed and efficiency in testing, so often testing times are too slow for the rapid pace of development in CI/CD pipelines. In a purely agile environment, test suites that take multiple hours to run are acceptable when testing is earmarked as a multi-day/week event run every couple of weeks. However, CI/CD environments can see teams commit multiple times in a day, which makes such lengthy test suites completely untenable.

In addition to this, testing (and particularly end-to-end testing) can be very flaky. Tests may fail and require manual intervention in order to fix them. This disrupts the development and deployment process. These sorts of issues will only be compounded by teams wishing to run these test suites multiple times per day. Slow test suites combined with unstable tests makes it difficult to return test results and feedback to developers in short timeframes. This makes it more likely that they will be removed from the same development mind frame they were in when they wrote the code and requires them to spend time refamiliarizing themselves with particular issues, which impacts developer velocity.

Bottlenecks in testing that go unnoticed or unprioritized in agile environments therefore often become magnified when switching to a CI/CD environment. Teams need to be aware of these issues when considering automating the deployment process. However, in doing so, the benefits are numerous.

Once an adequate deployment process is in place, it drastically reduces the amount of DevOps work that needs to be done in order to do the testing, staging, release, and post-release work to get the code out the door for each release. This has the added effect of shortening the time it takes to get feedback to developers, which helps improve developer velocity.

A CI/CD process does not solve all of the above problems and is not without its own issues to take into consideration. Once you’ve deployed your CI well, you’re going to find new bottlenecks – either in the form of your tests, discovering that your project has many bugs, or even that the bottleneck might be the deployment cadence itself.

Overcoming Testing Obstacles to Continuous Integration

Switching from agile to CI/CD is not as simple as flipping a switch. You need to anticipate issues in your test suite that will become exacerbated by continuous deployment. One of the most common issues that immediately becomes a problem if it is not properly anticipated is test suite runtime.

Quite simply, a test suite that takes hours to run in an agile environment will still take hours to run as part of a continuous deployment workflow. The issue here is that continuous deployment workflows are often built with the expectation that code will be committed and deployed multiple times a day. Teams then become limited by the realities of time itself – a test suite that takes 4 hours to run can only be linearly run a maximum of 6 times in a day (and unless you have a distributed team whose work spans the full 24 hours in a day, in practice this will mean even fewer test suite runs per day.).

Why not simply run these long test suites in independent pipelines and environments for independent deployments? A long-running test suite still means that developers sit largely idle waiting for test results, must pick through test results after the test suite finishes (taking a few hours on its own), and must fix bugs after losing context. If this is the case, pivoting to CI/CD actually slows rather than accelerates your developers, because there are now many long QA cycles per fortnight, rather than one!

There is a far more effective solution: get rid of (most of) your tests. Agile is an old methodology from when biweekly releases were considered fast. There’s no real incentive in agile testing to reduce test suite runtime and every incentive to cover as much of the code base with tests as possible because test suite runtime is not a factor in an agile testing process.

This makes teams relying on traditional testing dogmas less competitive than teams that embrace more modern testing techniques and solutions. Teams that can commit, test and deploy code multiple times a day will have a significant advantage in responding to feedback, reacting to the market, and capitalizing on trends over agile rivals who deploy once every two weeks.

There is of course much more to removing your tests without negatively impacting your test suite and end-product quality than simply chucking a grenade in the whole thing. Doing so safely and efficiently requires analysis of your test scenarios and test data in order to understand how useful each of your tests actually is. Valuable tests should be kept and maintained, while tests that provide little-to-no value should be discarded.

This prompts the question: how do you determine value in tests? The answer is that it involves multiple variables, such as frequency of use, importance to the business process, and severity of the issue, to name a few.

Tests should be separated into four categories:

  1. The first category is for tests that should be outright removed. Examples of these are tests whose conditions are so unlikely to ever actually happen in production that it is simply not worth the time it takes to test for. While individually each test might seem insignificant, cumulatively all of these improbable edge case tests can add valuable minutes onto test suite run times, which becomes a significant detriment in continuous deployment environments.
  2. The second is recognizing end-to-end tests that should properly be unit tests. End-to-end tests are typically the most flaky and prone to error because they often rely on interacting with the UI in ways that are vulnerable to being broken as the UI gets changed independently of the tests. They are expensive to maintain and should not be used for tests that should really be unit tests. Examples of this include testing for expected output values after running a particular function, such as whether the correct value is being returned from a particular event. These should be unit tests because they are testing the functionality of the software and not the end-user experience. Unit tests are much faster and cheaper to run and maintain than end-to-end tests, so it simply doesn’t make sense to run something as an end-to-end test unnecessarily.
  3. The third category is for tests that aren’t actually testing real user behavior. It’s very difficult to determine how your users are actually using your software in the wild and many teams spend significant time and resources conducting market research to try to understand their user base. If you are simply guessing at user behavior, then you may be running tests for situations that real users simply don’t run into.
  4. The fourth and final category is those tests which cover core functionality and follow journeys as users actually traverse them. These are your golden tests, and should remain part of your E2E test suite.

Understanding your tests and product in this way is a challenge and it requires a deep understanding of how your users actually use your software. It’s not enough to estimate which areas of your software are used the most, it’s important that you have solid data to help guide you and determine in which areas you should prioritize testing. Teams that remove tests in the third category should be confident that they are not removing valuable tests. Core tests that don’t follow user journeys should simply be converted, based on product analytics data, to be more accurate.

Get Your CI/CD Testing Discipline Right

Getting your testing and deployment process right in a CI/CD environment requires a lot of thought and effort, but it can pay dividends for the teams that take the time to do it right. The end result is a highly automated testing and deployment process that can handle multiple deployments in a day and gives timely feedback to developers so that they can maintain pace in modern software development environments with short software development life cycles.

At ProdPerfect we believe that the best source of data for user behavior is your users themselves. They should be the oracle that determines which of your tests are actually useful and which aren’t. By deploying a script similar to Google Analytics on your site, we gather data about how your users are actually using your product in real-time. From this data, tests can be developed that are built around actual use cases derived from authentic user behavior. This helps you efficiently cull unnecessary tests in a way that also helps maintain and even improve overall test quality.

If your team is still fully agile and you’re looking to make the leap towards CI/CD, then make sure that a thorough audit of your test suite is part of your pre-launch checklist. Feel free to get in touch with a member of ProdPerfect today: we may be the right partner to help you make the move. Making the transition by yourself is not easy, but our team has the experience and expertise to help you every step of the way, from initially anticipating problems to optimizing your testing process and workflow.