The Quality Assurance (QA) industry is currently undergoing a massive proliferation of new toolsets, most of which will no longer be here five years from now. All technology industries go through a similar proliferation-and-contraction cycle as the best (or best-marketed) toolsets are adopted, laggards fail, and rollups bring many toolsets into broader platforms.
This recent proliferation is being driven by machine learning (ML), and will inevitably lead to improvements in the QA ecosystem that have been desperately needed since the inception of Selenium. As prevalent as Selenium may be in web application testing, it still often leads to high instability and runtime. Non-ML toolsets like Selenium require rare, bespoke talent and a level of experience and intuition that makes testing more “art” than data-driven science.
The new rolled-up platforms that emerge will end up with toolsets that span various forms of application testing: API, functional end-to-end (E2E), visual, cross-browser, performance, load, accessibility, and (in time) possibly user-experience (UX) and security.
The platforms that emerge victoriously will take on one of two forms, depending on the success of the businesses themselves as well as the software industry’s willingness to innovate:
- A platform of “better Seleniums”: Toolsets that are easier for less ‘artful’ engineers to use to build and maintain tests that they have themselves devised, or;
- A platform of autonomous data-driven services: These platforms would, in their mature forms, make QA automation engineering obsolete.
Better Seleniums
These toolsets will prevail as long as the software industry believes that humans with intuition and experience will make better testing decisions than machines with data. In this mode, test strategy and test case development will remain in the hands of committees of Product, Engineering, and QA. The outputs from those committees will be turned into automated tests by engineers who have an easier life than they do in the Selenium mode. In this mode, machine learning is limited to improving stability and reducing maintenance in test suites designed and built by humans. This means some human efficiency gains.
Autonomous Systems
These toolsets will prevail if the software industry gains confidence in data-informed machines to make better decisions than humans about what is important to test, and how, rather than merely executing instructions that humans provide. The data that will be used in more mature versions of these toolsets and will include the following:
Behavioral analytics to inform:
- What are (and are not) important E2E test cases from the user’s perspective, and how to generate them
- Performance tests that are tailored to user actions, and intelligently compare performance changes to baselines to better uncover key performance regressions
- Load tests that are tailored to understand the impact of loads that are scaled relative to current loads
- What cross-browser/cross-device testing grids to spin up, and which not to
Experience from previous testing engagements and datasets to inform test generation on an arbitrary web application:
- How to generate mock data and assertions in E2E tests
- What workflows in an application are high-importance or high-risk despite being lower usage
- What to look for in User Interface (UI) changes that have broken tests (to yield greater self-healing)
- What performance/load thresholds should be considered critical
- What visual changes in an application should be considered critical
- What an arbitrary API format needs to be tested for full functionality
- How applications do and do not impact the experience of those with accessibility needs
- Where high-risk security vulnerabilities are likely to lie
Such results would virtually or entirely eliminate the QA engineering function from software development teams.
I believe that the latter platform outcome will likely win out over the former, given the apparent improvements in engineer efficiency, testing focus, and maintenance/update speed. Novel applications of machine learning have natural challenges–namely, they require large datasets that do not yet exist. The software companies building these need to be highly capitalized and need to process a large number of customers using less efficient, man-behind-the-curtain approaches to allow machines to learn enough to take over the work. But I believe that this challenge makes the success of the autonomous platform play a matter of time, the availability of capital (which is in turn dependent on the vision of venture capitalists) and the pace of progress in machine learning techniques.