“Not even Ares battles against necessity.”
-Sophocles
When designing a holistic testing strategy for any application, the QA strategist has to first answer, “Which testing methods should I utilize for which types of bugs?” Some bugs are rendering errors, some involve the application returning the wrong data, others are functionality issues (the user simply can’t do what they intend to), and others are application-level/browser errors, each requiring some specificity in approach.
Often, businesses rely too exclusively on either end-to-end (browser-level) testing, or too exclusively on unit testing, without properly accounting for different kinds of bugs. Some businesses try to bake various data validation checks into their end-to-end testing. At ProdPerfect, we’ve heard business leaders suggest, “if all of your unit tests are well-written, there’s no need for end-to-end testing.” Both of these approaches, though, are flawed. QA teams need both end-to-end and unit testing, and they should be applied differently.
Unit Testing vs. End-to-End Testing
Unit testing checks code blocks (typically with a black-box mindset): variable X is the input; variable Y should be the output. Unit testing efficiently checks for the functions or calculations that provide resulting data—a numerical value, a text string, etc. End-to-end testing tests all layers of the application at once; it’s best-suited to make sure buttons, forms, changes, links, and generally entire workflows function without problems.
Here’s an example to illustrate the proper approach to testing decision-making:
“Llamas R Us” is an E-Commerce company, selling llamas online by subscription. Their software includes a sales tax calculator. When a customer selects their ship-to location during the checkout process, the sales tax calculator automatically calculates the tax and applies it to the total cost of the customer’s monthly llama purchase. What needs to be tested here is that the right sales tax is applied to the llamas being purchased, depending on which state the would-be llama farmer is living in.
Llamas R Us may be inclined to assign testers to manually complete the checkout process, select different locations, and calculate whether the correct sales tax is being applied. To perform these tests, they may consider using automated end-to-end testing, writing unique code for each different state’s sales tax to ensure full test coverage of the calculator feature.
However, though it seems comprehensive, this system is not in reality optimal. For one, it’s inefficient, as it’s difficult for end-to-end testers, even with automated tests, to read the resulting data from such a web page and verify it—humans are imprecise and machines aren’t great at scraping raw data off of a web page. It’s also simply inefficient to test a checkout process 50 times for 50 different states. Doing so would cause testing time and human testing efficiency to implode.
Data Validation vs. Application Functionality
At its heart, testing the sales tax calculator is a data validation test. The hypothetical calculator is reliant on a particular set of inputs (states, countries, etc.) to generate a particular set of outputs (the sales tax multipliers). Thus, it is an ideal candidate for unit testing instead of end-to-end testing. In this case, Llamas R Us should create individual unit tests to verify the functionality of their sales tax calculator. These tests run much faster, require less work to set up, and don’t need to be changed each time the user interface is tweaked—they should always work until the sales tax calculator code is itself changed.
End-to-end testing, automated or otherwise, is ideal for testing the functionality of an application, rather than the data being sent to the user. For Llamas R Us and their E-Commerce application, end-to-end testing is ideal for ensuring that a buyer can add products to their cart, can navigate shopping categories, and can access product details, images, and reviews. In this case, Llamas R Us isn’t testing to make sure the right data emerges, but that the workflow can be completed, consistently, as the application changes. Because of the various application layers and the many interacting blocks of code being tested simultaneously, end-to-end testing is efficient and invaluable as a tool for testing features like these designed for the usability of a web application. Simply testing at the unit level will never provide the full picture of whether or not the whole application works together. The reality is, you never really know it works until you see it working.
Data validation questions and raw functionality questions are very different issues which must be approached with different kinds of tests. The simplest answer to the question, “Which testing methods should I utilize for which types of bugs?” is that if you’re testing something which produces a given (and by its nature predictable) set of output data based on a given set of input data, unit testing is likely the most efficient. For testing the stability and functionality of a feature or workflow, end-to-end testing is likely best.