Customer expectations about modernization costs and timelines are changing fast now that AI is in the picture. Our partners tell us they’re fielding requests for legacy migrations under $100K — numbers that would have been unthinkable three years ago, and in most cases still are.
These expectations are hard to re-manage. They are not the customers’ fault. Customer expectations are a result of the industry’s marketing.
Arthur C. Clarke’s line about sufficiently advanced technology being indistinguishable from magic applies directly here. If a vendor pitches AI-driven migration as magic, customers are going to act like it’s magic. Then they build a budget and a timeline around that assumption. Perhaps all my vendor is doing is clicking a few buttons? Why should I pay $5M for that?
We saw this up close recently. A prospective partner presented on their use of “agentic AI” for equivalence testing. We were intrigued, so we asked them to go deeper. The salesperson had to bring in an engineer. The engineer’s answer was, in essence, “We don’t actually use agentic AI for testing.” Certain vendors will do anything to win the bid, hope complications don’t arise, and ask for expensive change orders when those complications invariably do arise.
We work with partners who have a broad commitment to precision and honesty, who respect the buyers’ intelligence throughout the buying process. They’re the folks you want to work with. The problem is that you, as a buyer, don’t yet know who those are.
It’s the wild west again. As a buyer, you need to ask the hard questions that will press the vendor and help you assess whether they are the real deal. Below are some of the questions that you’ll want to bring to the first and second sales meetings.
Scoping and Discovery
What artifacts does your process actually ingest? Source code alone tells you what the program is supposed to do. It doesn’t tell you what it actually does in production. A serious process pulls from source code, JCL, transaction logs, live database access, scheduler configuration, and whatever documentation exists. Broad context is needed to find the important details in the hidden corners of 30+ year old software. If the answer is “we analyze the codebase,” ask what happens to everything the codebase doesn’t capture. How reliant is the vendor on your SMEs telling them the answer?
How do you handle the exception path nobody documented? Every legacy system has at least one calculation, override, or edge case that’s been running quietly for a decade with zero documentation and no one left who remembers why. When an AI model runs into this case, it might generate a new function that’s statistically plausible and syntactically clean. Ask specifically: when your process can’t find ground truth for a code path, does it flag that for human review, or does it silently produce its best guess? The honest answer involves a review queue. The evasive answer involves the word “robust.” There are too many unique edges and “gotchas” in legacy systems to safely assume that the AI will discover and model all of them accurately.
Code Translation vs. Functional Equivalence
How do you distinguish between “the code compiles and runs” and “the code is functionally equivalent to the legacy production application”? Make them define both terms separately, out loud. Code translation is a real capability, and it’s genuinely gotten faster — dependency mapping and documentation generation that used to take months can now take days. But a modernization program that stops at “it compiles” is stopping after 10% of the real work has been completed. You need to find out what the vendor’s tools are doing beyond clever LLM prompting.
What’s your actual comparison methodology for validating output? “We do extensive testing” is not an answer. “We run parallel/shadow processing and reconcile transaction-by-transaction, screen-by-screen, batch-by-batch against the legacy baseline” is an answer. Ask them to describe the mechanism in detail.
How do you handle the parts of the estate that aren’t really a mainframe problem? A meaningful share of enterprise COBOL runs on distributed platforms, not on IBM Z — and a vendor who treats every COBOL engagement as an identical mainframe problem is telling you they haven’t looked closely at your environment yet.
The “Agentic AI” Claim Itself
This is where the BS concentrates, because “agentic” has become the default marketing coat of paint for 2026. Independent estimates put the number of vendors with genuine agentic capability at a small fraction of those claiming it — the rest are existing tools (chatbots, RPA, scripted automation) rebranded with new vocabulary. If someone uses the word “agent,” dig deeper. If there is no agent underneath, then you have found that you cannot trust this vendor’s integrity and therefore its claims. There is, unfortunately, a lot of vaporware, and we have met vendors who are trying to sell capabilities that they hope to have soon.
What decisions does the agent make autonomously, versus what requires a human to review and approve? Get a specific example of the decision boundary. A real answer names a concrete action (“the agent operates the terminal application, keystroke by keystroke, to capture baseline behavior, and a human signs off on the test cases before execution”). A BS answer stays abstract (“it leverages AI to intelligently automate the process”).
Can your engineer walk me through what the agent literally does, step by step? Not your salesperson — your engineer. This is the exact question that unraveled the pitch we described above. If the technical answer contradicts or hedges on the commercial pitch, you’ve found your answer.
What’s your production deployment rate for this specific capability, versus pilot or demo stage? There’s a well-documented gap right now between organizations experimenting with AI agents and organizations that have actually put them into production for anything that matters. A vendor confident enough to be honest will tell you where their own capability sits on that spectrum.
Hallucination Handling
When the model can’t verify something against ground truth, what’s the fallback? There are exactly three honest answers: it flags the gap for human review, it refuses to proceed without more input, or it has a documented, auditable rule for what it does instead. “It’s trained not to hallucinate” is not one of the three.
Estimating the Workload After the “Easy Part”
This is the question category that protects your budget, because the AI-driven code conversion is reliably the fast, cheap-looking part of the program, and reliably not the part that determines whether the project finishes on time. Generally, a low bid doesn’t mean much unless it’s a fixed-price or has some other provisions for driving accountability with the vendor. Without those, anyone can make a lowball bid and then charge you later for “surprises” that they should have found in discovery.
Once conversion is done, what percentage of total cost and timeline is testing, remediation, and cutover, for an engagement like mine, specifically? Testing and correction is typically the majority of both migration cost and timeline, and effectively all of the risk. If a vendor can’t break that down for your environment and only offers an industry-wide number, they haven’t scoped your program yet, and are attempting to give you false confidence.
What’s your estimate-to-actual track record on the post-conversion phase? Ask for an assessment of the past 10 engagements: what did they estimate testing/remediation would cost, and what did it actually cost. A vendor with a real track record will have a real number, even an uncomfortable one.
What triggers a scope change mid-project, and how is it priced? AI-accelerated refactors are notorious for looking finished right up until testing surfaces everything the “easy part” didn’t touch — batch dependencies, downstream integrations, data quality issues, performance under real load. Get the change-order mechanism in writing before you’re in it.
Proof and Accountability
Can you produce a traceable audit trail tying every migrated function back to a validated legacy behavior? A comprehensive, ready evidence package a compliance team or auditor could review line by line.
If something breaks in production after go-live that testing didn’t catch, who owns remediation, and is that in writing? This is the question that separates a vendor selling a deliverable from a vendor selling an outcome.
Ask the questions in the sales meeting. If you get adjectives instead of mechanisms, you have your answer. Demos should not be slides and they should not be a video. They should be able to show most of this to you live, and if not, they should be able to send you sample artifacts. Anyone hiding about any of these questions is likely not worth your time.