Your Bank’s AI Is Measuring the Wrong Thing
By Dvir Ginzburg, founder and CEO at Encore AI
Simple Subscribe
Subscribe Now!
For most of my career, I built recommendation systems, First at Facebook and then at Microsoft, I helped financial institutions tell their advisors what to offer a customer next.
Picture an advisor at a large retail bank, seconds before a call connects, receiving a prompt suggesting a certificate of deposit. It seemed perfect, except conversions on the prompts ran under one percent.
We were good at the first half of the problem. I could tell you which customer to approach, when to approach them, and what to put in front of them. The message was written for one individual and highly personalized.
Yet, we weren’t able to hold the conversation. The recommendation landed on a screen, but the technology stopped there. Everything after that depended on whichever advisor happened to pick up the phone, and that turned out to be the whole game.
Reality check: Most advisors never read the prompts, and among the ones who did, only a fraction could transform a prompt into a conversation that ended with a new account. The same mistake is now running at scale across an entire industry, and it has a name.
Need to Know:
• Deflection rate is a cost metric, not a growth metric. it tells you nothing about the revenue you are losing.
• Most banks cannot state their completion rate: the share of started applications that end in a funded loan.
• AI usually sits with the COO or support leader, so it gets measured against operating expense instead of originations.
• The gap between what a top banker converts and what an average employee converts is the largest unmeasured number in most institutions.
• One auto refinancing lender raised digital application conversion by more than 30 percent without changing product or pricing.
Why “Deflection Rate” Is a Cost Metric Wearing a Growth Metric’s Clothes
The industry is measuring the wrong thing. Most banking AI is judged by how many customers it keeps away from a human being. These bots track cost metrics like calls contained, tickets resolved, and staff hours saved. These KPIs tell you nothing about revenue or what opportunities you’re missing.
Key insight: When banks push customers who call about a loan to an AI agent instead of a person, the deflection number goes up, cost per contact drops, and the metrics surpass KPIs. What we’ve learned is that certain big-ticket financial decisions, like taking out a loan, are decisions people don’t complete alone. Most of those customers never come back to finish their application.
Judged on deflection, that operation looked excellent, but judged on originations, it was turning away thousands of applicants a month without ever speaking to a single one of them. This is what happens when AI is treated as an efficiency plan, which is still how most banks, credit unions, and lenders see it. The owner is the COO, or whoever runs customer support.
Those are the people who are responsible for operating expenses, so operating expense is what the system is measured against, and it performs well when measured that way.
In this example, the conversation should be with the chief credit officer. That is the person who is likely unhappy with conversion rates and collection rates, and who has generally never been told that the AI is doing great but not translating to loans being processed.
What to Do Next
• Identify who owns each customer-facing AI system today, and whether that owner is accountable for operating expense or for originations.
• Ask your chief credit officer what the same system looks like from their side of the P&L.
• Check whether any of your current AI reporting contains a single revenue-side number.
The Largest Unmeasured Number in Your Institution
Then there is the question of what happens during the interactions that do go through. What every customer wants, and very few of them get, is your best person paying close attention at the moment the decision is being made. Your best people cannot be everywhere, and there are not enough of them. The gap between what your top banker converts and what an average employee converts can add up to a significant amount of lost revenue.
My mother holds a PhD in chemistry. Still, every time she opens a certificate of deposit, she calls me first. A CD is one of the simplest products a bank sells, and she still wants somebody to walk her through it before she commits. Most banking customers don’t have access to human support 24/7.
Yet, every bank knows how to deliver personalized services, which is why the private and wealth management business exists. The industry’s challenge has been to deliver that same personalized service below the threshold of the wealthiest customers.
At one institution we work with, a banker has been doing the job for nineteen years. She senses where a conversation is going before the customer does, and she knows which of a dozen approaches will land with this particular person. Everyone else in the office sits somewhere behind her as they pay close attention to how that senior banker performs and closes the deal.
Key insight: Very few banks I speak with can tell me their completion rate. We should be tracking metrics like the share of started applications that end in a funded loan, what share of onboarding journeys end in an active account, or what a conversation in a given channel is worth. Instead, we’re relying on measuring cost centers.
It’s a simple mindshift that can have a big impact. One auto refinancing lender we work with raised conversion in its digital application process by more than 30 percent. The product did not change, and the pricing did not change. What changed was that applicants stopped being handed a form and started being walked through a decision.
Time spent in the customer journey has a similar trend line. There are lenders taking 25 days to approve credit. Bringing that under ten counts on the time saved metric, but if you look closer, it’s a magic number for approving a new line of credit. These are the details and focus that move today’s AI systems from deflection metrics to revenue drivers.
What to Do Next
• Pull your completion rate: the share of started applications that end in a funded loan, and the share of onboarding journeys that end in an active account.
• Compare your top performer’s conversion rate against the team average and quantify the gap in dollars.
• Establish whether anyone has ever studied what your highest performers do differently on a call.
• Join your call recordings and chat logs to the outcome data already sitting in your CRM.
If you are sitting down with an AI roadmap, ask all your leaders who have customer-facing systems if they are measuring deflection data or completed applications and recovered relationships instead. Then, ask what your highest performers do differently on a call and whether anyone has ever actually studied it. Almost nobody has, and that is the piece that most AI systems are missing. The material is already sitting in the building while the recordings and chat logs are filed away with the notes from the CRM. Somewhere in those files is also the outcome. Was the loan funded and was the balance ever paid?
Join those two halves and the answer is clear. You can see which conversations ended in revenue and which ended in nothing. Analyzing what questions and conversations drove a positive outcome and what didn’t is the difference maker.
Bottom line: Banks are very good at measuring cost per call, but never built a way to measure what their best people were doing right. With the latest data and tech advances, now they can.
