Software development is an ever-moving target of changing product requirements, market conditions, development methodologies and evolving codebases. Every aspect of software development requires teams to remain on the move in order to stay competitive. When undergoing a digital transformation in your company, whether that be a change in software development methodology, framework, or programming language, testing remains an important consideration at all times. Here’s how to manage testing while undergoing a digital transformation.
“Shift-Left” Testing
To ‘shift left’ in testing is to introduce testing earlier into the development process. Instead of considering testing to be an end-stage process conducted after all development has finished, shift-left testing proposes testing during development, or even before it begins.
There are good reasons for introducing testing earlier into the development process. It helps catch bugs sooner and provides more opportunities for testing a codebase. Test cases can be developed in tandem with development (so, for example, writing unit tests for a class method directly before or after writing the class method) though for many teams this will require a change in business practices.
Some teams may have a clear division between development and testing teams. Often, these teams are kept separate and, in some cases, made to outright compete with each other. Industry best practices, however, are trending towards DevOps. This was originally the merging of developers and IT operations, though nowadays it often includes testers and QA engineers as well. These teams are brought together partially as a response to changing industry practices, such as moving to cloud-based hosting environments, but also to better integrate practices such as shift-left testing, which requires both developers and testers to work closely together.
The Impact of Digital Transformation
These changes can be highly disruptive, particularly to teams who already have clearly defined workflows for development, testing and deployment. Some of these workflows may conflict with modern development patterns. For example, many teams are still committed to agile development practices. Teams that are used to working in 2-week sprints, with discrete, well-defined development, testing. and deployment stages are likely to find the digital transformation to more modern development and testing practices, such as continuous integration/continuous development (CI/CD), highly disruptive.
There are several reasons for this, but two major factors are the difference in expected timescales and the blurring of departmental responsibilities. If your team has not yet started using DevOps methods and processes, it’s likely that testers and developers are on separate teams. Testing is morphing from a discrete stage at the end of development to something which is done before, during, and after development. This will require testers and developers to work much more closely together to develop and run tests that are fast, performant, and do not break during the development life cycle.
This also has a large influence on testing timescales. When testing is a discrete, end-stage process such as in agile, there is ample time set aside specifically for testing. Testers are generally more encouraged to test as much as possible than to test as efficiently as possible, leading to scenarios where test suites may take hours to run. Furthermore, some types of testing are notorious for being unreliable and require constant maintenance, particularly those that address the UI or end-to-end testing. This is a major paradigm shift for many organizations, since moving test suites with these issues to more modern CI/CD development processes is very likely to result in disaster and become a concern for the whole business rather than just an individual team. So how can you avoid or mitigate these issues?
Transforming Testing: Moving Away from the QA Sprint
The software industry is a dynamic one, with best practices and processes changing all the time. Teams must be flexible and receptive to change in order to stay ahead of the competition. The traditional QA sprint limits teams to deployments on a bi-weekly basis. By comparison, teams utilizing modern CI/CD workflows may deploy new, fully-tested code changes multiple times a day.
Moving away from the QA sprint is as much an organizational challenge as it is a technical one. It requires buy-in from developers, testers and IT operations. Depending on your company structure, there may be 3 separate departments used to working with a high level of independence from one another. Management must overcome these challenges in order to align all three departments, facilitate communication between them, and integrate CI/CD practices.
On a technical level, testing transformation will be one of the areas where digital transformation is likely to be the most disruptive. Test suite runtimes need to be cut from hours to minutes – ideally, an entire test suite should run from start to finish in under 10 minutes. This gives developers and IT ops the confidence to deploy code changes virtually any time – something unthinkable in agile, but very common with CI/CD.
Reducing existing test suite runtimes by such large factors requires culling a lot of existing test cases to cut down on test runtimes. This culling should be data-driven with a lot of careful planning and consideration over what to cut and what to keep. This involves understanding the value each test case provides. How likely is a particular test scenario to occur in production? How many users would be impacted by it? What is the severity of the issue? These are all factors that can be used to help determine the value of any particular test case. Those with high value – ones that protect against critical issues, cover common cases experienced by many users, etc. – should be kept. Those with low value – covering edge cases or those with minimal impact to end-users – may be discarded.
This seems simple and logical enough, but actually knowing which test cases are high-value and which are low-value can be difficult to determine, especially without proper data monitoring and analysis tools. For many teams, this is the major stumbling block when looking to undergo a digital transformation in testing. How do you differentiate the high-value test cases from the low-value test cases?
Determining Test Case Value
There’s no simple solution to determining test case value, other than to look at the data rather than relying on assumptions. Testers need to work closely with IT ops in order to understand real-user behavior and gain insights into how your software is actually being used in production. Who better to determine real-user behavior than your real users? Many apps and websites already track user behavior as part of traffic analysis and market research. This data can (and should) also be used to drive testing. Your high traffic hotspots are where you should be focusing testing. Your tests should focus on covering real use cases with coverage aimed at core modules and critical parts of the application.
Without data driving these decisions, it is virtually impossible to know which tests are important and which are not ahead of time. Users can respond and react in surprising and unexpected ways. Using data allows you to adapt and respond to end-user behavior much faster and more precisely than with manual or agile testing practices. This is a benefit not only for testing, but it can also be used to guide future development, highlighting new potential features and avenues of development based on how popular they are with your users.
Take Your Organization Beyond Testing
The process of implementing a digital transformation can be highly disruptive and requires changes across numerous departments and teams. These changes can range from organizational to technical challenges. Making changes to the way tests are run is often one of the most disruptive and challenging of all the changes for many companies. Those used to thinking somewhat uncritically about each test case and instead opting to throw the kitchen sink at every code change will find themselves quickly unstuck.
If your team is moving from something like agile to CI/CD development, testing and deployment practices, it may be worthwhile to reach out for assistance. At ProdPerfect, we have experience helping clients deliver fast, performant test suites that run in minutes and can be integrated into modern development workflows. If your team needs help making the transition, get in touch with an expert on our team.