Software testing has gone through several paradigm shifts as the software industry has grown and matured. As software has grown in complexity, and as crowded venture-backed verticals compete ever more fiercely, software testing must evolve to keep up with greater demands. Agile testing, once favored by developers, is now falling out of favor. Releasing software in 2-week sprints is simply too slow in contemporary software development, where updates may be released multiple times in a single week. Agile testing is not so agile, anymore.
Instead, the software development and testing industry is moving towards continuous integration/continuous development (CI/CD), where individual (or small bundles of) commits are pushed to live as soon as they are ready. The partner to CI/CD in software testing is continuous testing, whereby testing becomes part of the CI/CD pipeline. CI/CD and continuous testing (CT) blurs the lines between release cycles and testing phases, opting instead to test software in tandem with its development and release.
E2E Testing and CI/CD
The “CI/CD” posture requires teams to develop a pipeline whereby developers can push commits as they are ready and these get automatically tested and pushed to live should they pass all applicable tests. Most developers rely on CI tools or services, such as TravisCI or Jenkins, to construct the pipelines and deploy commits automatically. This requires good development practices based around rapid workflows and short software development lifecycles.
A big stumbling block for many developers is end-to-end (E2E) testing. This is the process whereby your entire app or website is tested from start to finish, exactly as a real end-user would experience your app. Extensive E2E test suites can take hours, or even days, to run. This is not such an issue in agile development, where there is time allotted at the end of each sprint for testing. However, the time required to run conventional E2E testing can be a big problem for teams using CI/CD to move quickly and push new code regularly.
The Limitations of Agile
Testing software the agile way involves batching changes together towards the end of a sprint and then running E2E regression test suites against the pending release. This works for catching bugs, but there are several problems with this approach. First, it adds a delay between the time a developer commits code and when they get feedback on that code. Depending on when the code was submitted and when the tests were run, this could be days or even a week or more. The developer will need to spend time re-familiarizing themselves with old code, which can significantly slow developer velocity.
Second, this approach makes tracking the source of bugs more difficult, as it can become tricky to untangle exactly which commit caused a bug when they are bundled together. Isolating commits can help determine who committed them and makes it much easier to identify which commit is the source of a particular issue. Both of these challenges dramatically increase the time required to fix bugs once found, costing developer hours and product roadmap time.
Third, the longer release cycle means simply a longer time to market and lower stability of the product. Short release cycles not only get features out faster, but support feature flagging and thus rapid rollbacks if there are any issues–whether or not those issues are functional bugs.
Switching to Continuous Testing
In contrast to agile testing, continuous testing runs many or all of your tests on every commit. This solves many of the issues of agile testing, which crucially includes getting feedback on code back to developers immediately, and directly to the developer who produced the bug in the first place, so they can fix their bug immediately, without context switching.
The obvious issue here is that an E2E test suite that takes hours or days to run as part of an agile development workflow is not practical for use as part of a continuous testing process, where many commits may be made in a single day. The solution here is quite simple, if unintuitive: Get rid of most of your E2E tests.
Test bloat is a real issue and there is little incentive to minimize unnecessary tests with agile testing. However, this practice is inefficient and can cause test suites to take unnecessarily long to run without providing due benefit. Removing tests that don’t provide benefits helps reduce test runtimes, while actually optimizing your test suite. The question then becomes, how do you know which tests to drop and which tests to keep?
You need a hard metric based on real data to identify which tests matter and which don’t. What tests matter to you will vary significantly depending on the specifics of your particular software, but in general, tests for the sake of tests that focus more on code coverage than practical usage are great candidates for deletion.
Identifying Value in Tests for Continuous Testing
Ultimately, E2E testing is intended to mimic how a real user will use a particular piece of software, following the user journey from start to finish and testing actions that users are likely to perform. There are several approaches to identifying how real users will use an app or website, but the very best source is users themselves.
Existing apps and websites should take full advantage of the wealth of data already available to them in the form of real user usage data. This can help to pinpoint the most highly-used areas in your app and therefore where tests will have the most impact. Corner cases and rarely-used sections of the app should not receive the same level of attention, as the payoff between time spent testing and the impact on the real user experience is lesser. Such corner cases can and should be primarily tested by unit testing.
By trimming E2E test suites based on hard data, like real user usage data, you can keep the tests that matter and provide value, while getting rid of the tests that don’t matter or don’t provide sufficient value to justify the time it takes to include them as part of a test suite. While each individual test may seem innocuous enough, these lesser tests can cumulatively cost hours of testing time in sufficiently large projects. Shaving hours off your test run times while increasing test efficiency means that test suites can more easily be run as part of a continuous testing process.
Continuous Testing is the Next Step in Development
Testing needs to move at a rapid pace in order to keep up with development. A development pipeline that includes CI/CD but lags behind in testing will slow down the entire release schedule and impact developer velocity. Waiting around for hours to days for test suites to finish running and return results is inefficient and will leave teams who don’t adapt behind in a very competitive environment.
Contemporary consumers are used to living products and software-as-a-service, where updates are constantly released on a regular cycle and bugs are fixed quickly. In order to meet these expectations, many teams turn to continuous testing to help match the rapid pace of software development.
Continuous testing does not simply offer a way to quickly test new code, but it also offers a way to continually test code. This means that teams can feel confident that critical functionality does not break. For many engineers on-call over weekends or holidays, this in itself is likely to be reason enough to adopt continuous testing.
Whether for speed or security, it pays to adopt continuous testing. Knowing what to test and where is the key to maintaining high test quality while reducing the overall volume of tests and therefore lowering test runtime. At ProdPerfect, we believe that your users are the best source of data for making the decision of which tests to drop and which to keep. We use a traffic analyzer script, much like Google Analytics, to understand how your users are using your app or website and then construct tests based around that data, targeting where they will be most effective.
If you are not yet using continuous testing as part of your development process and you are interested in converting, get in touch with a member of our staff today. We will walk you through the process and work with you to create a continuous testing pipeline that speeds up test runtime, improves test efficiency and creates a faster, smoother development pipeline.