Breaking Iron Triangle Thinking: A New Approach to Agile vs. Continuous Testing

 

THE IRON TRIANGLE OF PROJECT MANAGEMENT 

When I used to work as a project management consultant, I would hear brilliant leaders around me say: “Speed, quality, cost. Pick two.” In other words: if you want a higher quality product, you have to sacrifice speed, cost, or both. And everyone would nod along: this was true, and it was obvious. In this frustratingly pervasive “iron triangle” decision-making philosophy, true improvement is not possible—only trade-offs are possible. We are doomed to do only as well as we are doing now; our only options are to choose which priorities will be sacrificed to others. 

Because the leaders I met believed this model, they missed opportunities to actually improve their project—for example, to find cost improvements without negatively impacting quality or timeline. They never put in the effort to find these opportunities because they implicitly believed these opportunities didn’t exist. I have learned over my career that our implicit, unchallenged beliefs about the world have a powerful impact on the choices we make, for good or ill. 

 

                                                       

 

IRON TRIANGLE THINKING IN SOFTWARE DEVELOPMENT

In software development, people may not talk about the project management triangle, but this same trade-off-only mindset—an implicit belief in this Iron Triangle—still exists. Iron Triangle thinking means that engineers often feel stuck between prioritizing quality of code, speed (time to market), and cost. Of course we will always have choices that we can make between speed, quality, and cost. But the fallacy of Iron Triangle thinking is believing that the parameters of these choices are fixed. When we believe that improving speed means necessarily sacrificing quality, cost, or both, we limit our potential for innovation. 

The pervasiveness of Iron Triangle thinking means that many engineering leaders see only a hard trade-off between the cost and speed benefits of continuous development, and the quality benefits of deploying slower and testing more. They believe these competing demands are irreconcilable. As a result, many development teams who want to improve cost and speed are hesitant to make the move toward continuous development because they’re afraid of impacting code quality. They feel forced to choose between developing in two-week sprints—with a massive test suite that is expensive and slow, but assures quality at the end of the sprint—and moving to a Continuous Integration/Continuous Delivery (CI/CD) mode that is faster and cheaper, but sacrifices quality with less testing. 

 

QUALITY: THE “AGILE” APPROACH (BIG AND EXPANSIVE)

In traditional end-to-end (E2E) testing processes, there is no pressure not to add tests because you want a minimal chance that any bugs will make it to production. Typically, in the sprint-based (“agile”) development approach, quality is optimized at the price of cost and speed.

Most leaders are accustomed to running E2E testing at the end of a sprint, so the test suite can run as long as it needs to. Once E2E test suites grow to a certain size, you have to run them less frequently, as they can take several hours and simply can’t be run every time developers check in code. This means that there’s a feedback lag: developers discover bugs in their code days or weeks after they’re written. At that point, it’s impossible to pinpoint which deployment even caused the bug, so you can’t toss the buggy code back to the developer who wrote it. Often, leaders assign these bugs to more junior developers who are forced to mine the code to understand the intent of the original developer. In all, these bugs require a bunch of time and resources to fix.

 

SPEED AND COST: THE CONTINUOUS APPROACH (LEAN AND MEAN)

In a CI/CD environment, a team supports continuous development with continuous testing: every deployment is tested individually before it is staged or shipped to production. With fewer tests, the test suite runs in a matter of minutes, feedback lag is minimal, and buggy code returns directly to the engineer who wrote it. With this approach, developers are more fresh on what they just wrote so that when bugs do get caught, they’re able to pinpoint exactly where the problem is and resolve it in minutes, rather than hours or days. Developers kick out code and move on, producing more code in the same amount of time. Thus, with the continuous approach, the KPIs that the engineering leader maximize are developer velocity (faster deployments) and cost (fewer tests to maintain). 

The drawback to having fewer tests, of course, is that you’re ensuring less quality before your code goes out the door. Fewer tests simply means fewer opportunities to catch bugs before they hit production. A purely continuous testing suite sacrifices quality in favor of speed. 

 

A NEW APPROACH: AGILE AND CONTINUOUS

As engineering leaders, as long as we continue to subscribe to Iron Triangle thinking, we will never be provoked to ask ourselves if we can actually improve a process without making a trade-off. When we break Iron Triangle thinking, we can find a third way: an option that is not some form of compromise between competing priorities. We can find paths that give us the best of everything. In software development, choosing a third way means believing that it is simply not the case that every improvement in either speed, cost, or quality must necessarily come at the expense of the other two. Breaking Iron Triangle thinking requires that we do the harder thing and choose a smarter, more strategic approach to deployment design. 

One such way to break the Iron Triangle is to simply choose the best parts of testing from both agile development and continuous deployment. What gets us both the speed of the continuous environment and the quality of the agile environment? One possible design change to your deployment structure is to implement continuous testing early in your SDLC with the quality benefits of the end-of-sprint testing that occurs in agile development practice. 

In this structure, you preserve the benefits of continuous development: your developers can still contribute small chunks of code and run tests continuously for rapid feedback. In addition, you can run a larger test suite in a staging environment against a number of builds to catch bugs that the continuous test suite missed. This approach means many of your bugs—and if your continuous test suite is well-targeted, your priority 1 bugs—are caught sooner and fixed faster. Anything missed here is caught by the larger test suite later. This way, your developers get the speed advantage of continuously developing and testing, and your application gets the quality benefit of running a larger test suite—all without costs beyond maintaining the larger suite that would’ve been written for an agile team anyway. 

 

MOVING FORWARD: A PRACTICAL APPLICATION 

How do we manage both agile and continuous test suites? At ProdPerfect, our team uses data to maintain a continuous test suite focused on our customers’ most commonly-used features, while they manage their own larger test suite. Whatever your prioritization mechanism is, you can start the process by building your total regression test suite and tagging your top priority tests. Then, you can configure your CI to continuously run only the tests that are tagged as priority, and run the whole kit daily (or at whatever frequency works for you) on your staging environment. Compared to the “agile” testing approach we explored, this new approach to deployment design improves speed and cost with no negative impact on quality. Compared to pure continuous testing, this approach improves quality with zero to minimal impact on speed and a positive impact on cost (you’re catching more bugs before production). Instead of choosing one approach and sacrificing the other, you are implementing both intelligently and improving multiple parts of your project triangle without negatively impacting the others. You’ve broken the Iron Triangle and get the best of all worlds.

As engineering leaders and teams, you can break the Iron Triangle of project management by structuring your improvement cycle not around trade-offs, but on continuous improvement. A great engineering leader will use the momentum and mindset shift gained from their first Triangle-breaking win to push their team to challenge the assumptions that have held them back from improving all parts of their engineering practice. 

ProdPerfect Removes the Burden of QA Testing

There are typically three levels of quality assurance testing maturity. One is the classic waterfall approach where it takes weeks to get a deploy ready. Then, there is the continuous development and continuous delivery approach where QA engineers are put in place to handle automation. The most mature way of tackling QA is removing QA engineering as a separate practice, and making all your engineers responsible for the quality of features.

The problem is that none of these levels of maturity seem to be able to get QA right.

“No one has a good answer. Enterprise are failing in waterfall structures. Agile teams are failing or running into difficulty hiring and maintaining QA engineers. Silicon Valley is having to hire only the most senior folks, and even then it is through force of will and pain they are able to keep test suites to a point they are happy with,” said Dan Widing, founder and CEO of the automated QA testing provider ProdPerfect.

Automating QA
There is a better way. ProdPerfect removes the struggle it takes to set up a QA engineering department, and automates QA testing using live user data. This is “dramatically cheaper, dramatically faster, gets you a result faster, [and] is going to nearly guarantee that you catch bugs as part of your process,” Widing explained.

ProdPerfect is able to obtain live user data by analyzing web traffic and creating flows of common user behavior. That behavior is then built into an end-to-end testing suite that ProdPerfect maintains, expands and updates based on actual user data over time.

According to Widing, QA testing is “incredibly difficult, painstaking work that almost tends to be underappreciated by the organization itself,” and the folks who are having to deal with this are just overburdened with work. “We have a mechanism that lets us shake out the environment the customer needs us to test against… and then we are using a testing framework that lets us plug in our learnings from these steps to produce an automatically updated test suite,” he continued. “The experience the customer gets is a black box QA engineering department… What you get at the end is an auto-updated test suite that can run continuously in your CI system that just tests your application.”

ProdPerfect covers every core workflow with applications, provides 95 percent or more of test stability, less than four-hour regeneration of broken tests, and less than 48-hour test coverage for new feature sets.

“You don’t need to do anything to build, maintain, or expand the testing suite. We got it. You need to respond to bug reports, of course, and keep a stable testing environment up and running for us, but that’s all. Very frequently people call this ‘magic’ or ‘too good to be true,’” the company stated on its website.

Getting the right metrics
ProdPerfect not only works to ensure QA testing is covered, but also works to help teams understand what the right metrics to quantify success are.

“That is something we put into our service every step of the way. What your browser automation should be doing is catching as many significant bugs as possible whatever stage it is testing at and then otherwise staying as much out of the way,” said Widing.

You will know you have a solid testing foundation in place when you don’t ship a fire drill-style bug and have to wake up in the middle of the night and figure out how to deal with it or who is on top of it, Widing explained.

Since ProdPerfect is already analyzing what users are doing, it can project how things should be working and make sure they stay working. The solution tests features continuously, detects any significant bugs and verifies the feature set is actually working.

“We aim to stay out of the way by crafting what are the other metrics that are important to make sure you are not slowing down the software team,” said Widing.

Additionally, the solution will measure against minimum-frequency thresholds to confirm its performance.

“If you don’t set up your design and data strategy or set up the right tooling, everything falls apart and you have to work particularly hard to make sure all the pieces work together otherwise any singular improvement is not going to help you at all,” Widing said.

This article was first published on SDTimes.com.

What is Regression Testing and Why is It Important?

An ounce of prevention is worth a pound of cure.”  -Benjamin Franklin

This article is for everyone that asks, “why would you want to keep testing the features that are already live? Shouldn’t we focus on testing new features?” Those of you who already know the importance of testing both can probably focus on other content instead.

Testing new features that go live is certainly critical to making sure your application works. Brand new features should be tested extensively, not only for functionality, but also for user experience, conversion (as appropriate), performance, etc.

Your current feature set needs to be tested for regressions. regression is what it sounds like if you took statistics in college: your application regresses from its current state to something worse. It is deviation from expected state. This happens in one of two ways:

  1. You’re maintaining, altering, fixing, or improving a feature (rather than introducing a new one) and break it.
  2. You’re changing just about anything and end up breaking something completely different in the application.

The first form of regression is fairly obvious; the second can be a head-scratcher. The short version of why this can happen: almost any application is deeply interconnected. There’s a concept called DRY – “Don’t Repeat Yourself.” Good developers don’t copy code; rather, they make that code accessible to all features that touch it. Any area of an application depends on many others to function properly. If you break something while working on inventory management, you might wreck checkout. Updating infrastructure might break login. It’s not that every change can truly affect every part of the application, but any change might impact multiple parts of the application if a bug is introduced.

Regression testing, therefore, tests to make sure any of these regressions are caught before they make it to production. Generally, you run a suite of regression tests by testing every unit of code, every API, and core user pathway across the application at the browser level, with every build. Some teams can’t run their regression suite fast enough (or are using manual or crowdsourced testing, which has incremental cost per test run because you’re throwing extra bodies at the problem) that they run their browser regression testing suite on a less-frequent schedule. The obvious downside of this is that you’re more likely to let regressions into production before they are caught.

Automation vs. Manual Testing

If you’re considering automating your browser regression suite, start with features that are least likely to be changing rapidly: building automation takes time to pay off, so you want to make sure that these tests are going to be run more than a few times before they need to be re-written. For brand new or rapidly-evolving features, manual testing may be your most efficient approach.

When you do decide to automate: automated browser regression testing suites are built in a number of ways. The most basic is scripting them in SeleniumCypressCapybara, with Javascript in TestCafe, or using other such frameworks. They can also be built using record-and-play tools such as Selenium / TestCafe IDE. Machine Learning is making record-and-play stronger and less time-intensive, and will eventually allow record-and-play to drive itself using web traffic data.

What is Continuous Testing? (And Why Should I Care?)

“Move Fast and Break Things” – Facebook’s Old and Very Catchy Developer Motto

“Move Fast with Stable Infra” – Facebook’s New, Less Catchy Developer motto

Continuous testing is the holy grail of software quality assurance. Get it right, and you get to both move fast and also not break things. But it’s not only the holy grail because it will give your application everlasting life quality, but because it’s elusive, requires a lot of work—work which might end up killing you (think: Indiana Jones and the Last Crusade).

Continuous Testing Complements Development

More specifically, it’s a practice that pairs with continuous development (whether or not your team is deploying continuously). Briefly, continuous testing means that you’re developing continuously and testing every commit as you develop. Slightly more specifically, in continuous development, every developer should be committing fairly small changes, very frequently. Every time a developer commits work to remote, a regression testing suite (at the Unit, API, and E2E levels) runs to make sure nothing in the application has broken. This testing can either occur on a feature branch (perhaps during a pull request) or after the feature branch has been merged on a pre-production environment. If the developer is modifying or introducing workflows, each workflow also gets tested end-to-end. That’s continuous testing.

Here’s why (beyond the obvious benefits of testing generally) it’s so great: your developer has just committed code when it gets tested. Not only have they just committed it, but what they committed is necessarily a fairly small amount of code. Your developers, therefore, get instantaneous feedback while they are still contextually aware of what they just wrote—it’s fresh. And because it’s a small amount of code, there are only so many places to look to find out what went wrong. Your developers can thereby rapidly find, understand, and fix the bug. It doesn’t get to production, it doesn’t make it to a JIRA list, and it doesn’t languish for weeks. It gets fixed in minutes, before production, and generally becomes a non-event.

“Dan, how do I drink the soul-nourishing waters from this mythical cup?”

First, you need to have the infrastructure and culture to support continuous deployment. This is hard and we don’t want to trivialize it, but we won’t focus on it here. If you have the infrastructure and culture to support continuous deployment, then you’re ready for continuous testing.

The biggest challenge to continuous testing specifically is that your tests need to be comprehensive in scope yet also run quickly. These tests ideally need to cover the entire application, and they also need to finish running in minutes. It would conventionally seem like you need to choose between runtime and comprehensiveness, but focus can get you both.

Improving Browser Testing Runtime

By far the slowest part of your testing suite will be your browser tests, so this is where you need to do the most work to limit the number and parallelize. We discuss how to assess which user flows to focus on for browser (E2E) testing here, but in brief: focus on the core workflows your users are actually following frequently in your application. This maximizes bang-per-buck and lets you cover everything important without extending runtime so much that it gets in the way of continuous testing. After you’ve built the E2E tests you want, parallelize the ones you can, using a parallel-friendly framework such as TestCafe.

What if my Needs Are Complex?

If you want to test for edge cases, test cross-browser and cross-device, or otherwise have a very large browser testing suite, you may need to prioritize which tests get run with every build and which tests get run out of band, perhaps every few hours. This will allow you to make sure no regressions get to production that would affect core user flows, and any edge cases or low-impact regressions are caught a few hours later at worst.

All of this requires a lot of work even when you have “made it.” It can feel like a truly gargantuan task to get there if you’re not close. Read on, and we’ll be able to help you break the process into discrete steps.