1. What led you to ProdPerfect?
I’ve always loved math. In fact, ProdPerfect Cofounder Dan Widing and I became friends on the math team and chess team (state champions!) growing up. I majored in math and economics with a minor in psychology. I used that to design financial software before eventually getting into a consulting company which gave me a lot of exposure in the startup world. I worked on everything from water conservation startups to conversion optimization tools to the e-commerce site for a brewery called “Mobcraft Beer.” I went to get my master’s degree in data science. All the while, Dan and I kept barely missing each other with job opportunities. Finally, at the end of my master’s program, my other co-Founders Dan Widing and Erik Fogg called me up and said: “Hey, let’s make a company. Let’s actually try and automate QA.” What do you do when they say that? I said: “Yeah, let’s do it.” That’s how I got here: nerdiness and wanting to solve cool problems.
2. Can you paint for me a picture of what life is like in your role?
As head of Data Science, there’s always the need to explore both the data we already have and the new stuff we can do with it. But, I am far more of a mathematician than a data scientist and slightly more of a product/UX designer than a mathematician. Therefore, I’m working to hire people who are going to be substantially better than me at data science, and that’s great. One of my goals in life—much less ProdPerfect—is to build my team to the point where I’m no longer needed. I’m excited to focus more on product work: there is so much beautiful user-experience (UX) exploration, data visualization, and design to work on. By improving our transparency and visibility of some of our analysis, we’re going to improve the value we provide to our customers. We have the opportunity to transform the QA ecosystem. In the nearer-term, we can provide information about our test suite, our analysis of customers, and the bugs we find in a way that provides serious value for Quality Assurance (QA). Longer-term, we can do this not only historically, but predictively. We can give tools to QA teams to figure out what parts of their applications are the most vulnerable in a way that is focused on customer behavior. It prevents QA teams and engineers from burning out. I love it. Best of all, we have a fantastic team of people whose experiences and skills we can draw upon that are just as excited as me about this.
3. What are some elements of a “typical” interaction with customers and/or product?
This has certainly changed over time. Initially, there was constant interaction with customers. That’s because when the company was first founded, we had to figure out what customer data we could even collect, much less how to use it. In contrast, a lot of the work I’ve been doing this past year has been focused on making the job of maintaining customers easier; providing better or different information, making it easier for us to access and interpret data that comes from running test suite. Now, data science is finally getting to a place where I can once again start focusing on how we’re actually establishing value to the customer. As well as cool stuff like: “What’s the new value that we can provide to the customer?” So I’ve gone the full gamut from very closely interacting with new customers, to diving deep with a single customer, to having months of my customers and stakeholders being internal. Now, it’s finally looping back.
4. What intrigues you about your role in data science at ProdPerfect?
We are in the true golden age of data ethics. We get to figure out what data ethics means to the world at large. Data scientists have a concept of data ethics. But when it comes to consumer protection, personal privacy, GDPR—you name it—there’s so much to be navigated. At the same time, we are in a golden age of exploration in data science. You can log into an online open course like Coursera and within hours feed in the entire works of Shakespeare or Anne Rice and make a Twitter bot that spews out content. It takes longer to understand how it works, but the tools are available. How amazing is that? Every day there are more tools laid at our feet. At the same time, I am a data scientist who has something of an aversion to data science. If you ever bring up machine learning or AI or data science to most engineers, you’ll likely be greeted with eye rolls and apprehension. AI isn’t perfect, machine learning isn’t perfect. Unless you are careful, you can bake black boxes into your system that are hard to fix and difficult to get five 9’s out of. Today’s culture has pushed the allure of machine learning without necessarily being able to back it up with improvements in this consistency. At ProdPerfect, we’ve applied machine learning. And it’s fantastic. But we’re doing it in a reserved manner because recklessness with these tools can lead to poor results that are difficult to diagnose. I’m a data scientist who loves data science, but remains skeptical of it. That’s why I encourage us to use it sparingly.
5. How do you explain to people the value of ProdPerfect?
We do hands-off QA testing. For someone who doesn’t understand Quality Assurance, I’d say: “We teach robots to make sure websites don’t break by feeding them anonymized behavior of your users.” There are thousands of analytical tools out there right now. You can use those to gain thousands of discrete insights. But none of those are really telling you what should be tested on your website. None of those are telling you: “What—if it does break—is going to hurt the use or usability of other parts of your website?” If I just say that, people get it. We’re using what users are actually doing on peoples’ websites to figure out what needs to be tested. That’s the insight people are incredulous about. Here’s an example: my mother sometimes orders mail-order steaks online. The website only allows you to apply one coupon. Guess what? The customer service line often allows you to apply more. She uses the website differently from those that designed it. I don’t understand how my mother can use a website in a manner that seems different from all other humans on the planet. But she has a penchant for breaking web apps in beautifully unexpected ways. And if she’s doing it, other people are doing it. Keep in mind I used to design web apps; I’ve done the work of a senior QA engineer to give direction on test suites. And by looking at what ProdPerfect showed me, I realized I was consistently wrong about what should be tested. I’ve been in those shoes, I know a person deciding what should and shouldn’t be tested is going to make mistakes. Your users can and will surprise you.
6. In your own words, how would you differentiate ProdPerfect from competitors?
Any other service out there I will loosely group into two groups. One is the throw-everything-at-the-wall-and-see-what-sticks group. Those are the companies that make thousands of small regression tests which run continually. Even if you can run those all in parallel, your test suite is going to take a heck of a long time to pass. Then you have other customers where it’s not actually hands-off QA. You still need someone at your own company who understands QA and your website to say what should be tested. This second group is great at maintaining tests. And they are fantastic at automatically healing tests. But that’s not the same as hands-off QA testing. We’re filling this niche of telling you what should be tested in a data-driven manner. That’s something no one has ever done before with this level of granularity. And make no mistake, bugs will get into production. You can have the best engineering team, the best QA tools, the best automated QA tools. Bugs will get into production. But can you prevent the critical bugs that will affect a critical mass of users? That’s something that we can say with increasing confidence as time goes on. Yes, we can prevent those bugs.