The E2E Testing Pyramid
In 2015, Google directly criticized the role of End-to-End (E2E) testing in the QA process. Google correctly noted that E2E tests are comparatively unstable and require reworking whenever there is an update to the application. If that isn’t t done, or is done incorrectly, testers and developers will not only be stuck sifting through the weeds to locate and fix bugs, but also wondering whether the test result was even genuine. Instead, Google says, there should be a heavy emphasis on Unit and API tests which are more accurate and minimize testing time. In Google’s view, E2E testing is treated like its counterpart at the top of the now-outdated food pyramid—fats and oils—”use sparingly.”
Sometimes readers take Google’s advice aggressively, cutting E2E testing from their entire process. However it still plays a critical role in quality that must not be ignored.
Why E2E Testing Still Matters
Unit and API tests do indeed run faster and cost less to maintain, and therefore have every right to win the bulk of testing mindshare. However, E2E tests remain the single best way for testing how the application will work in the hands of an actual user. Unit tests only address a single block of code at a time. API tests model the interaction between features. But your app’s true functionality depends on how every feature, every element, and every layer works together, not in isolation.
E2E testing is the only way to incorporate all of the network and server effects that influence your application’s functionality and user experience—a major factor in the overall performance of an application. What’s more, a complex user flow touches a myriad of points across the application, changing the application state and data in the database multiple times along the way. These user behaviors are diverse, unpredictable, and complex—they introduce states and inputs in ways that developers simply can’t preemptively imagine. Different code modules and API calls need to operate together seamlessly for a user to be able to properly navigate the app.
That being said, E2E tests cannot be an afterthought if your aim is to deliver quality code that works for its intended purpose.
How to Make it Happen
E2E testing is hard not only because of its resource intensity, but also because it’s difficult to test accurately. Conventionally, testing teams must imagine what users might do in order to test for their behaviors. They need to decide: should we cover what we think are the core use cases? Should we attempt to imagine and cover every edge case? The first approach can lead to poor coverage; the second can lead to significant test case bloat (and also poor coverage).
There’s a better way. If you want to test your application for how users intend to use it, let your users tell you what to test.
Find out how users are actually using and navigating your application, and test that. Product analytics tools have existed for years, but have not yet been accessed to their full potential in quality assurance testing. They’re used by marketing religiously, by product sometimes, and by quality never. There’s no reason the same product analytics data that drives UX decisions shouldn’t drive testing decisions.
The most common pitfall when it comes to testing is that developers and testers are testing the intended behaviors of their users and missing the data that is actually worth finding out—the unexpected ways in which users are actually navigating through the application, whether or not those behaviors are originally intended. If you can test actual user behavior, you will have a lean, mean testing machine that ensures your application’s core functionality always works, and that app-breaking bugs are always caught before they get out the door.
As Google suggests, you should have fewer E2E tests relative to others. We suggest, however, that E2E tests should be the crown of your testing process, not simply an oily “use sparingly” afterthought.
To use end-to-end testing to build your very own lean, mean testing suite, consider the following tips:
- Use analytics to discover the few dozen key user flows that truly matter
- Build E2E tests only around those user flows—don’t be tempted to add new tests for every possible bug
- Retire tests aggressively—when user flows change and old tests become obsolete, get rid of them (At the very least, comment them out)
- Make a rigorous (but quick) check of your current testing suite vs. known user flows as part of your regular maintenance cycle
- Consider whether automating the process might be the best solution for your team