“Even if you’re on the right track, you’ll get run over if you just sit there.”
– American Humorist Will Rogers
Continuous development (CD) has become the new gold standard in today’s tech marketplace, and with good reason. Continuous development is fast, efficient, and allows for a company to get the latest version of their product to their customers as quickly as possible. It furthermore allows developers to get rapid feedback on small chunks of code through continuously testing that code as it’s developed, as long as continuous testing is also in place.
But with any change comes new challenges, especially if a development team is transitioning from a longer (for instance, monthly versioning) development cycle to a continuous development cycle. As part of this shift, a team must tackle rearranging end-to-end testing cycles accordingly, moving from monthly testing to (ideally) continuous testing on every build. Every release must be adequately tested, and with continuous development multiplying the number releases, the quantity of required test runs rises in turn. Each new change or release carries the risk of breaking an application, and each new change or release must be tested.
Because of this challenge, many testers turn to automated testing solutions for help. With regular continuous updates, manually testing an app prior to each release is now infeasible (not to mention unbearably expensive). Automation makes testing each deploy possible, and removes the primary testing burden from the development team and allows them to focus instead on other priorities like product improvement. Though automation is relatively simple to implement in unit or API testing, a proper approach to automating end-to-end testing requires further consideration.
Outsourced Testing
If a team’s end-to-end testing is outsourced to a manual testing company (crowd-sourced or otherwise), they are most likely to pay per test. This cost might be bearable for a team shipping and testing code once per month. However, a team deploying five or 10 times per business day is suddenly looking at 20 to 40 times the cost of the team deploying once per month. We discuss some other pros and cons of outsourced/crowdsourced manual testing in this blog post.
Test Code Automation
Instead of employing outsourced manual testing, a development team can alternatively use “lean and mean” testing code to automate their tests. To keep up with a CD release cycle, they need to develop a core regression testing suite that covers important workflows: it needs to run in minutes, rather than hours. This means prioritizing tests to focus on workflows that impact the most users, impact core functionality, and/or impact revenue. We at ProdPerfect have already discussed choosing the correct number of end-to-end tests, and that information is just as relevant here as otherwise.
What About Less Common Flows?
Continuously-testing teams will only be testing core workflows with each build; what about less common flows, edge cases, and the like? These tests should be maintained as a second suite altogether, and run out of band, at timed intervals, on a stable testing/staging environment. This strategy can also be used for cross-browser/device testing, which if extensive will also take more than a few minutes. By splitting your testing suite in two, you get the benefit of continuous testing on continuous delivery, without losing the ability to find edge case bugs.
It may feel like a daunting task to begin a CD cycle for the first time, but with foresight and intelligent testing plans, any team can take full advantage of its benefits to improve their product and make their customers ever happier.