DevOps is a relatively new development paradigm that advocates for a more holistic approach to both software development and testing, by encouraging smoother, continuous collaboration between software development (Dev) and deployment/infrastructure operations (Ops) teams. This impacts all areas of software development, including QA and testing. Whereas developers and QA engineers may have once been siloed, DevOps testing encourages both teams to play a role in the testing of software. It is often said, “DevOps is everyone’s job.”
What is DevOps Testing?
DevOps testing is complex and requires many considerations, but these are the most important: Since DevOps testing is more of a holistic approach to testing than traditional testing paradigms, teams are often unclear about their role in testing throughout the DevOps cycle. Adapting to DevOps testing involves breaking down traditional walls between developers and QA engineers and understanding the roles and responsibilities each team plays in testing.
There’s good reason to want to break down these barriers and unify teams. The end product and end-user experience is a culmination of the work of all teams. Put simply, an end-user does not know or care about which team is responsible for a particular bug, they only care that their experience has been negatively impacted by one.
Teams working in isolation can develop pieces of software that work perfectly well in isolation, but these pieces do not run in isolation. An end product relies on many different fields coming together, from the developers who wrote the code to the environment it runs in and the infrastructure it relies upon; software is the sum of its parts. If each part is not working together, it becomes much more difficult to track and stop bugs before they reach the end-user.
In short, DevOps testing is at its core about testing constantly across the entire software development life cycle, in discrete chunks of time on small pieces of code..
Here are some of the key things you need to consider with your testing approach:
Unit tests: a robust foundation
Unit tests are an integral part of any test suite, but it’s not always clear exactly what should be a unit test. For example, many organizations will require QA engineers to create E2E tests for conditions such as, “does the password reset function work?”, “does the password field reject invalid characters”, and “does the login function reject an empty password field?”. These are all examples of what should rather be unit tests.
The difference between unit tests and E2E tests lies in their scope. Unit tests are intended to test code modules – for example, individual methods inside a class, or individual functions in a library. E2E tests are meant to test the overall app experience like an end-user would when all the individual pieces have been put together.
Getting teams to understand when and where unit tests are more suitable can be difficult, as it may require a culture change in teams used to simply handing all testing processes off to QA. Fundamentally, the developers who write the code understand it the best and therefore are in the best position to test the code in isolation. This frees up QA engineers to focus on E2E testing.
Because unit tests run so quickly and are so stable, and because they provide precise feedback about the location and nature of a bug, they can be run at very high frequency and early in the software development lifecylce, making them the perfect foundation for DevOps testing.
E2E tests need to be focused and fast
If unit tests are meant to test code modules, then E2E tests are meant to test workflows through the application. E2E testing involves testing software as a whole, just as a real user would experience it. These tests should be fiercely focused because of their direct relation to the end-user experience. A bug caught in E2E testing is one that your users are likely to find.
As software becomes more complex and the multitude of possible pathways and user workflows through an app increases, so too must your E2E tests adapt to this increase in complexity. However, E2E tests need to be fast on the scale of minutes, not hours, in order to support a DevOps posture. In practical terms, this means prioritizing testing by focusing on the major processes and most likely pathways that will cover the greatest number of users.
This is where most teams go wrong. QA leaders used to agile deployments often want to write hundreds of tests which can take hours or even days to run, and lead to many false positives — having the discipline to limit the E2E testing cycle to under 20 minutes is critical for supporting the speed required for true DevOps.
Feature Flags: the basis for flexibility
Not only is it virtually impossible to guarantee bug-free code, but you shouldn’t even aim for this. This is a common mistake among some teams, but not only is it a virtually unattainable goal, it will also significantly slow down your rate of development and deployment. Some bugs are acceptable to ship (such as bugs with low frequency or low severity) in order to keep pace with rapid development and short software development life cycles. Ultimately, quality assurance confidence scales asymptotically with the size of the quality assurance cycle: total confidence would require infinite testing time. In DevOps testing, leaders must accept that buggy code will be shipped.
A great way to release software quickly while minimizing bug severity is through feature flags. These are ways to segregate new features in a way that they can be enabled or disabled for all users (or just a subset of users) with a simple switch. This allows for the gradual rollout of new features and, when combined with logging/monitoring, provides a great way to measure the impact of a new feature on users.
Feature flags provide a way to release and monitor the impact of a new feature without impacting everyone or even a substantial portion of users. This is a much less disruptive way of releasing new features, that allows you to measure the positive or negative impact of a feature before rolling it out universally. It also allows you to catch meaningful bugs without exposing them to your entire user base. This provides a more holistic approach to QA, where features can be measured not just for the presence of bugs, but also the impact of a feature on users, whether positive or negative.
Prioritize Speed for DevOps Testing
Speed is the name of the game for both DevOps testing and general contemporary software development. Development and testing must work in tandem with each other, as lag in one area will also impact the other. This need for speed cannot come at the expense of the end-user experience, so teams must adapt in ways that allow them to move fast without breaking things.
Adapting to DevOps testing is hard and can be a challenge for any team. It involves changing how developers and QA engineers view their own roles and the project as a whole. If you are looking to adopt DevOps testing into your project, but you’re not sure how, then get in touch with a member of our team at ProdPerfect today. We’ll help guide you through the process and show you how you can use DevOps testing to prioritize testing and increase the pace of development without negatively impacting the end-user experience.