Dan Widing

What is End-to-End Testing?

Extremely simply put, end to end testing is a type of software testing to determine if software is behaving as expected. This guide is an overview of how we approach end to end (E2E) testing, and why it matters.

“You can test every servo, every wire, every component; the hardware, the firmware, the software. But you have no idea whether your robot is going to work, and whether it’s going to work in the wild, until you let it out into the wild and let it fail. And it will fail, even when it seems like every component works individually. Even well-built things get hairy when you connect them all together.”

We didn’t have a truly applicable analogy for end-to-end testing until we heard it from a customer that had previously built robots before moving over to web software. Sure, there must be some sort of theoretical, Platonic ideal of testing in which exhaustive testing of components and subsystems will guarantee that the robot—or your web application—will work without needing to be run “in the wild.” But, we’ll wager, nobody’s found it yet. The perfect software testing method doesn’t exist, but we can get close.

Why Do We Perform E2E Testing?

This is the essence of end-to-end (E2E) testing, and why it’s so important. Your web application is probably less complex than a Mars rover, but you similarly won’t know whether it’s going to work once you put all the bits together until you’ve done just that. Your unit tests will test individual blocks of code for their core functionality. API/integration tests will make sure that your “subsystems” are working as intended. But, E2E tests are intended to test the entire application as real users would use it, in conditions similar to how real users will use it.

Why End to End Testing is Important

Therefore, an E2E test will actually launch the program in a browser and interact with it in a way that will test every layer of the application: the user interface itself, the browser (and compatibility with it), the network, the server, the APIs, the codebase, any 3rd party integrations, and any hardware—the whole kit. As with the robot, you don’t really know how all of these components and layers will work together until they’re doing just that—working together. You therefore don’t want to be shipping changes to your application without testing it end-to-end (unless you don’t mind bugs sneaking through).

E2E tests can assume many names, in part depending on their level of rigor. They can be called browser tests, smoke tests, user acceptance tests (UATs), or (less accurately) UI tests. Typically these all mean the same thing: you launch a browser to interact with the application, and check that specific behaviors still work as intended.

There are two ways to launch this browser and interact with the whole application: the first is with a human who checks for failures by clicking around, referred to as manual testing. The second is by having a machine virtually simulate a human, using predetermined validations to check for failures, referred to as automated testing.

What is the Optimal Number of End-to-End Tests?

Ideally, QA teams should not simply be tasked with guessing at how users are using their applications. They can and should employ advanced product analytics to understand these user journeys and how they are evolving with time. In this way, focused testers are then able to fully understand which E2E test cases are most relevant and write the best corresponding tests to ensure quality without bloating the testing suite.

The best way to decide how many end-to-end browser tests to perform is to determine how many different ways users actually interact with your application. If you were to graph the distribution of cumulative observed user behavior with a histogram, it would look much like the graph above: a steep curve of early behaviors, and then a bend towards a steep asymptote. After about 60-70% of total observed user behavior, the incremental coverage of each additional test case becomes negligible. From our own research, we find that this long tail of behavior doesn’t typically represent uncommon feature usage–most of it is behavior that doesn’t align with features at all. It can be ignored.

In the end, while your biggest obligation is to provide a quality product to your customer, your second obligation should be to do so quickly and cost-effectively. There exists a “just right” space of testing what matters, and not testing what doesn’t. Data is your guide to finding this golden ratio. If you can identify the core use cases in your application, you are no longer picking between a false choice off “high coverage” and good runtime / high stability: the notion of trade-off ends and you’re getting the best of both worlds.

Data-Driven Testing

And as with our Mars rover, the ideal test case simulates real-world usage as precisely as possible: testing the application in the same way that your users are using it, or are going to use it. This requires having data which tells you how your users are in fact using your application. Data integrity is essential here. Utilizing real user data is always possible when testing for regressions. But, user behavior needs to be estimated (or, frankly, guessed) when testing brand new features because you don’t have data about real usage quite yet.

Some teams might be tempted to do “kitchen sink” testing and try to test the application in every possible way, rather than in a way that reflects user behavior. We discourage this elsewhere in more detail elsewhere, but the primary consideration is that E2E tests are the most expensive, least stable, and slowest tests you’ll run. Having too many is going to incur dramatically increased costs for steeply-diminishing returns.

How to Build E2E Test Cases

Developing end-to-end (E2E) test cases poses a substantial challenge compared to writing unit and API test cases. Blocks of code and APIs can be tested against a well-defined, limited, and predetermined set of business rules. Test-driven-development (TDD) techniques can empower developers to write relevant tests alongside their code.

Developing E2E tests requires a fully different approach. This level of testing is meant to replicate user behavior that’s interacting with many blocks of code and multiple APIs simultaneously. Below we recommend a process that will help you build accurate, effective test cases for your E2E testing regime. Note that we will not cover test scripting here, but only test case development.

There are four considerations, each of which will be explored in turn:

  1. How to Scope End-to-End Testing
  2. What Bugs to Target
  3. Which User Flows to Follow
  4. How to Design Test Cases

How to Scope E2E Testing

The team should focus their efforts, and test only how users are actually using the application. Doing so yields the optimal balance of achieving thorough test coverage without expending excessive resources or runtime or relying on an expert to predict how customers use the website. This approach requires user data, rather than an expansive exploration of the different feature options in the application, to manage. To mine user data, you’ll need to use some form of product analytics to understand how your users currently use your application.

What Bugs to Target

E2E testing is meant to ensure that user interactions always work and that a user can complete a workflow successfully. E2E test validations should therefore make certain that an interaction point (button, form, page, etc) exists and can be used. Then, they should verify that a user can move through all of these interactions and, at the end, the application returns what is expected in both the individual elements and also the result of user-initiated data transformations. Well-built tests will also look for JavaScript or browser errors. If tests are written in this way, the relevant blocks of code and APIs will all be tested for functionality during the test execution. 

Which User Flows to Follow

To prevent test suite bloat, we suggest splitting your test cases into two groups: core and edge. Core test cases are meant to reflect your core features—what people are doing repeatedly. These are usually associated with revenue or bulk usability; a significant number of users are doing them, so if they fail you’re in trouble. Edge cases are the ways that people use the application that are unexpected, unintended, or rare, but might still break the application in an important way. The testing team will need to pick and choose which of these cases to include based on business value. It’s important to be careful of writing edge case tests for every edge bug that occurs. Endlessly playing “whack a mole” can again cause the edge test suite to become bloated and excessively resource-intensive to maintain.

How to Design Test Cases

Every test case, whether it is core or edge, should focus on a full and substantial user experience. At the end of a passing test, you should be certain that the end user will have completed a given task to their satisfaction. If you build E2E test cases with this process in mind, you will achieve high-fidelity continuous testing of your application without pouring unnecessary or marginally-valuable hours into maintaining the suite. You will be able to affordably ensure that users can use your application in the way they intend to do so.

Who Should Determine End to End Test Cases?

Due to the complexity of effective end-to-end testing, the ideal person to determine and execute end-to-end user tests is someone whose core expertise and focus is in understanding the entire user journey and the outcomes thereof, not someone who is asked to tack on end-to-end testing as an afterthought. E2E testing should be driven by an independent group with a mandate to focus on it and the time invested to maintain it: this can be the product team or it can be QA as a whole (a QA analyst, QA automation engineering team, etc).

These groups can, with the help of tools and data, wrap their arms around the different user journeys, develop end to end test cases for them, and write tests designed to catch bugs at the user journey level, and maintain them over time. This level of testing doesn’t require intimate understanding of the underlying modules of code behind the application; it’s instead meant to ensure that users can always use the application as they want to. Software teams should leave testing of lower levels of the application to those lower levels of testing—unit and API/integration testing.

Ideally, QA teams should not simply be tasked with guessing at how users are using their applications. They can and should employ advanced product analytics to understand these user journeys and how they are evolving with time. In this way, focused testers are then able to fully understand which E2E test cases are most relevant and write the best corresponding tests to ensure quality without bloating the testing suite.

E2E Testing Limitations & Challenges

Finally, a word of caution: E2E testing has limitations. It’s great at testing that the application will generally function: that users can always move through workflows without errors or breakages. Early in the software development lifecycle, it can be a lifesaver. It’s great at ensuring that all of the gnarly bits of code are working together when a user checks out or logs in or executes an analysis. But E2E testing isn’t great (or efficient) in testing that the right data is going to be displayed for a user or stored in the application—this is a place for unit tests to thrive. E2E testing also isn’t great at showing you where in your codebase your bug is hiding—just where in the user’s journey they’re going to find that the application is broken. Finally E2E testing isn’t great at telling you whether your page is formatting incorrectly or “looks wrong.” It can do a bit of this, but it’s a heck of an expensive way to do so. We recommend using testing tools like Percy.io for testing visual regressions, instead.

In short: ignore E2E testing at your own peril, but over-relying on it won’t do you any favors, either.

Dan Widing
Latest posts by Dan Widing (see all)

    Leave a Reply

    Your email address will not be published. Required fields are marked *