B2B software sales and marketing teams love hearing the term “artificial intelligence” (AI). “AI” has a smoke and mirrors effect. It sounds impressive. But when we tell our buyers “this is the AI working” or “the AI is doing this,” they often know so little about AI that they never ask the hard questions. Sales and marketing teams love this too: they’d rather not answer those tough questions.
But in many industries such as our own devtools space, it is crucial that buyers understand both what products do and what their limitations are to ensure these products meet their needs. After all, the purpose of AI is to make good decisions for humans. To accept, simply, that “the AI is doing this” is to accept that we don’t really know what the product is, how it works, or if it is making good decisions for us.
When we’re in the buyer role, we often don’t hold ourselves responsible for understanding AI and machine learning products because these technologies are intimidating. They’re incredibly complex. This article addresses the limitations of AI and machine learning, so that software buyers can ask the right questions to truly understand what it is they are buying.
The Test Oracle Problem
One limitation of some AI or machine learning products is that for certain applications of the technology, there is no source of absolute truth to compare the accuracy of the output against. This is true for our product. People often ask my team: “How do you know your test cases are right?” This is the classic test oracle problem: no one knows or can know the perfect set of end-to-end (E2E) tests for any application. There is no objective standard of truth. As a result, neither humans nor machines have an answer for how to produce a perfect set of test cases. Explaining this reality to a buyer can be anxiety-inducing. No one wants to introduce this kind of uncertainty into their sales process. Yet our buyers deserve thoughtful, well-informed answers about the nature of our products. So we dive right in.
ProdPerfect’s buyers are some of the most forward-thinking, innovative folks on the market. They are thinking about software testing at a very high level and want to understand exactly how our product delivers results. They won’t accept “the AI is doing this” as an answer. Our sales team has to do our best to help buyers understand our product’s logic, as well as its limitations. Though we can’t say that our methodology is the only reasonable way to build a test suite, we can say that it is a really good one for producing a highly reliable, user-centric set of test cases. We believe that using machine learning logic based on user behavior is an extremely robust way to identify the patterns that truly matter to customers. The AI in Testing sub-industry has also agreed that machines are much better than humans at producing this set of test cases. As Jason Arbon, CEO of Test.ai, has said, when we rely on humans to attempt to come up with the “right” test cases, the result often depends on whether or not they’ve had their coffee that morning.
We explain to buyers that our product’s advantage is its methodology: it uses product analytics data rather than human intuition to make decisions about what to test. As a buyer, you also need to understand the intended advantage of your seller’s “AI” product before making a purchase decision. Is the “AI” meant to make a decision that is more accurate–against an objective standard—than a human? Or make a decision faster, and with less cost? Or introduce an alternative methodology that uses new data in a new way? Answers to each of these questions influence how you will use the product and what value it provides your organization.
“AI” vs. Machine Learning
Though AI is commonly accepted to mean, “any machine that uses math to make decisions,” true AI is self-taught. AI has a neural net that mimics neurons in a human brain which allows it to teach, update, and evolve itself. Because of this, true AI is difficult to build and is often experimental rather than commercial, with a few rare exceptions.
More often, what’s being described when sales and marketing teams say “AI,” is machine learning. Machine learning is human-taught: machines learn through human feedback using a probabilistic decision-making process that improves via ongoing correction. Machines “make decisions” or generate results through data science: they take in data, run algorithms against it, and output a decision or series of assertions based on probabilities. Humans correct the machine by telling it whether it was correct or incorrect in its assessment, and the machine updates itself. For example, in machine vision in autonomous vehicles, humans use tags to teach machines to identify the objects in a landscape. The machine applies a set of algorithms to probabilistically determine if an object is a human, car, or street light. As it receives accuracy feedback from humans over time, machines learn to make better decisions. Just like natural human intelligence, a machine’s artificial intelligence is only as good as the quality of its current processes, logic, and inputs. And because machine learning is based on probabilities, buyers must acknowledge that these products will sometimes give the wrong output or make the wrong decision.
Based on how you plan to use a product, you need to determine how rigorous its accuracy–the probability of producing the right output–needs to be. How often a machine can make the wrong decisions and still serve its purpose is incredibly application-specific. Self-driving vehicles must be nearly perfectly accurate to be adopted. Paralegal machine learning tool sets likely need to be slightly less accurate. What about machine learning-driven algorithmic tool sets designed to scrape contact data for your sales team? How accurate does the product need to be?
Asking the Right Questions
Regardless of how you plan to use a product, it is important to ask your seller the right questions not only to better understand the product, but to build resiliency around its current accuracy levels. The next time someone tells you “the AI is doing this,” you can ask the following questions to get more information:
- Is this product a machine learning product? Does it need to be, in order for me to get a meaningful result? To be considered machine learning, a product needs to learn through human feedback, not just make decisions using math and probabilities. Do you just need a product that uses logic to make decisions, or do you need a product that improves in accuracy over time?
- How is the accuracy of this product calculated? What is the desired output of the machine, and how is accuracy to this output calculated? What does 100% accuracy mean? You won’t know if the machine is more accurate than humans if you don’t know the conditions used to calculate accuracy. If a machine is “30% more accurate than humans,” who assessed the accuracy of the machine? Was it humans? If so, how did they know? If you don’t know what accuracy means, you won’t understand the stakes of buying the product.
- How do you know when the product makes the wrong decisions? Given a clear definition of accuracy, you will have to embrace that the product will sometimes produce the wrong output. You will need to build processes so that your business is resilient to this wrong output, and you will need this preliminary information from the seller to get started. Typically, their most successful customers have already adopted business processes to build this resiliency, and if the seller is sophisticated they can help you adopt these as well.
- In its current state of teaching and testing, how often does the product make the wrong decisions? Knowing the frequency of mistakes and the stakes of those mistakes will be crucial to deciding how you use the product–and whether it’s safe to do so at this stage in its development.
- How many hours of teaching have been put into this product? Since people are the ones teaching and correcting the machine, how many human hours have been put into the teaching process? This number will provide a simple, quick approximation of how much effort has gone into making the product more accurate. A low number can be fine, depending on the application. If the number is low, you should expect an early-adopter price or price grandfathering arrangement.
- How does my usage improve the accuracy of this product? As a buyer, you are an integral part of the machine testing and teaching process. As such, you should be willing to use your data to improve their accuracy, because you want these products to improve in the future.
Why Knowing AI and Machine Learning Matters
Not only is there a lot of “AI” out there that isn’t AI, but there is also algorithmic technology out there that isn’t machine learning. Those products may not even have any data science behind them. Thus, it is more important now than ever for buyers to know enough about AI and machine learning to ask the right questions to understand how these products make decisions on behalf of humans.
Ultimately, there are limitations to all machine learning products, though these limitations will differ by the product that you’re looking at and the way it’s being applied. Especially when a product’s accuracy levels are unknown, all you can do as a buyer is ask if the philosophy and methodology behind it are valid to use for decision-making: Does it have access to better data than humans? Can it make smarter and faster decisions than humans with this data? If the answer to these questions is yes, you should definitely consider buying the product as an alternative to having your people do the same work.