AI Is Only as Good as the Data Behind It. Five Questions to Tell If Yours Is Ready
By Sam Palmer, co-Founder and CEO of Libero Labs
Simple Subscribe
Subscribe Now!
Over the past several months, I’ve sat down with data and technology leaders at banks and credit unions of every size, and I’ve noticed something that surprised me. Institutions that seem to have nothing in common — a small credit union still running spreadsheets alongside its core, or a bank several times its size with a six-figure cloud data bill — keep arriving at the same question: Can we actually trust our data enough to build on it?
That question matters more than it used to, because AI doesn’t forgive bad data the way a person can. A loan officer who gets handed a stale number will usually catch it or at least know to ask. An AI system will use it anyway, with total confidence, and hand you an answer that looks just as clean as a correct one would have.
The Assembly Line Problem
Most of the data quality work happening at financial institutions today looks the same. Someone checks a report after it’s built, notices the numbers look off, and starts tracing backward to figure out what broke. It’s the equivalent of a factory that only inspects finished products at the end of the line. By the time you’ve found the defect, it’s already been packaged, shipped, and possibly used to make a decision.
Toyota ran into this same problem decades ago on its manufacturing lines and solved it in a way that still holds up as one of the better ideas in quality control. Instead of only checking finished cars, the company gave every worker on the line the authority to pull a cord and stop production the moment they spotted a problem, rather than letting it move downstream and multiply. What made it work wasn’t the cord itself. It was that the authority to use it sat with the person closest to the problem. If a defect can’t be caught and stopped where it happens, it might as well not be caught at all.
Key insight: Data pipelines at most financial institutions don’t have anything like that cord. Problems get discovered downstream, in a dashboard or a report, long after the pipeline that produced them has moved on. The fix isn’t just checking the data more often. It’s building the ability to catch a problem at the point where it’s created and stop it before it spreads, the way that assembly line workers could.
Why AI Raises the Stakes
There’s a reason this problem has become more urgent instead of less, and it comes down to how AI systems actually query data.
An AI system asked a direct question about your data can take one of two approaches. It can write its own query against your raw tables and guess at what your data means, or it can go through what’s called a semantic layer, a governed set of definitions that tells the system exactly what “active member” or “delinquent loan” actually means in your institution, rather than letting it infer an answer from column names.
The difference in accuracy between those two approaches is real, and it shows up in a specific place. A 2026 benchmark from dbt Labs comparing the two methods found that on questions within the system’s modeled scope, raw queries against unmodeled data landed in the 84 to 90 percent accuracy range, while the same questions run through a properly modeled semantic layer scored 98 to 100 percent.
Key insight: The gap that matters more shows up on questions that fall outside what’s been modeled. There, a raw query will still produce an answer, sometimes right, sometimes not, landing anywhere from 70 to 100 percent depending on the question. A semantic layer, by contrast, recognizes what it doesn’t know and declines to answer at all. That’s a conservative stance, but it’s the correct one. An agent that guesses might get lucky. An agent that knows what it doesn’t know won’t hand you a wrong answer with total confidence.
Even the AI Companies Are Learning This the Hard Way
If you assume this is a problem for financial institutions to solve and not the AI companies themselves, it’s worth reading what Anthropic has written about its own internal analytics. The company that built Claude has publicly described its own struggles getting reliable answers out of its own data, reporting that internal query accuracy fell as low as 21 percent before it built out a governed data layer of its own. Even after that work, the company is candid that so-called silent failures, answers that look correct but aren’t, remain a problem it hasn’t fully solved.
I find that admission more useful than most vendor marketing I read. If the people building the models are still fighting this problem on their own data, it’s not something a financial institution should expect to solve by simply pointing an AI tool at its warehouse and hoping for the best.
What Regulators Are Already Signaling
Examiners are already moving in this direction. In April 2026, the Federal Reserve, the OCC, and the FDIC jointly issued revised guidance on model risk management, known as SR 26-2, replacing a framework that had been in place since 2011. It applies most directly to banking organizations with over $30 billion in total assets, and it carves out generative and agentic AI explicitly, calling them “novel and rapidly evolving” and outside its scope for now. But the principle behind it, that institutions need to be able to show their work on how models and the data behind them are validated and governed, doesn’t stay contained to $30 billion balance sheets or wait for a footnote to catch up. Smaller institutions and CU-specific AI governance are already being addressed elsewhere — the NCUA maintains its own set of AI governance resources aimed specifically at credit unions navigating this territory.
A Practical Checklist Before You Start
None of this requires a particular vendor or a particular platform. It requires asking a few honest questions about where your institution stands:
- Can you trace a number back to its source? If a board member asks why a figure changed, can you show the path it took, or does someone have to reconstruct it by hand?
- Do your key terms mean the same thing everywhere? “Active member,” “delinquent,” and “at risk” should mean one thing, not a slightly different thing in every report that uses them.
- Where would a problem get caught? If bad data enters your pipeline today, does anything catch it before it reaches a dashboard, or does a person have to notice first?
- Do you know what a change will break before you make it? Whether it’s a core migration or a new data source, can you see downstream impact in advance, or do you find out after something breaks?
- Is sensitive data controlled as it moves, not just where it starts? Governance that only covers data at rest doesn’t say much about what happens once that data gets copied, joined, or exported.
None of these are really five separate problems. They’re the same problem showing up in five different places: a gap between where your data lives and where decisions get made from it, with nothing governing what happens in between. Close that gap and all five questions start answering themselves. The answer to each one has to be something you can point to and demonstrate, not a feeling you have about your own data.
