Financial institutions are rapidly adopting artificial intelligence (AI) to enhance operations, refine marketing strategies, determine creditworthiness, and develop new products and services. However, this technological leap comes with a crucial caveat: the need for continuous data input, including information gathered from customer interactions where consent may be unclear.
As integration deepens, banks, brokerages and other financial service providers wrestle with when and how to obtain explicit customer consent to use their personal data to train AI models. Financial institutions must develop solid ways to sanitize that data to safeguard customers’ privacy without degrading the value of the data itself. And they must remain vigilant against a new breed of cyberattacks targeting AI solutions.
Failure to address these issues proactively could expose these companies to significant legal and regulatory challenges, potentially undermining the very innovations they seek to achieve.
From Algorithms to Verdicts: Defining Privacy in Gray Areas
Banks, credit unions and their contact centers are closely following a spate of lawsuits challenging the use of third-party AI solutions fed consumer data without explicit consent. Among them is a class action lawsuit filed earlier this year in a California federal court that claims Google’s AI-powered customer service platform violates the California Invasion of Privacy Act (CIPA). By recording and analyzing client Home Depot’s customer calls without their consent, the lawsuit alleges, Google’s Cloud Contact Center AI tool intercepts, records, and retains conversations to train AI models, constituting wiretapping under CIPA, which was enacted in 1967.
Recent rulings in similar cases suggest Google may face challenges in defending its actions. Judges in lawsuits brought against Salesforce and Amazon Web Services have allowed wiretapping claims to proceed against the tech companies that use customer data for AI training.
These cases point to a common phenomenon when laws and regulations lag technology, creating gray areas around data use and consumer privacy. The Privacy Act of 1974, borne out of a distrust with the federal government and its electronification of personal information, and the more financially focused Gramm-Leach-Bliley Act (GBLA) of 1999 were among early attempts to protect consumer privacy. More recently, state laws like the California Financial Information Privacy Act (CalFIPA) of 2004 and California’s Consumer Privacy Act (CCPA) of 2018 have introduced the concept of consumer data ownership and consent, flipping a formula that had favored corporate entitlement.
“With CalFIPA, you have to opt in; with Gramm-Leach-Bliley, people get an opportunity to opt out. Either way, if you’re sending something to an unaffiliated third party, it’s a relatively straightforward process of saying you’re going to have to let the customer know,” says Seattle-based attorney and cybersecurity expert Scott M. Giordano, AIGP, FIP, CISSP, CCSP, CIPP, CIPT, CIPM.
He noted that financial institutions found to have mishandled consumer data faced substantial penalties — up to $100,000 per violation for companies and $10,000 per violation for board directors. “So, on paper, the potential for being punished is pretty big. The problem is: How do you find out that customer data is being misused? Unless someone blows a whistle or a bad guy breaks in or there’s some kind of failure, it’s hard to tell that any of this is being done,” Giordano says.
To date, 19 U.S. states have adopted omnibus privacy laws, with 16 of the 19 state laws either in effect or taking effect by the end of 2025. This will add even more complexity to the regulatory environments in which AI technologies operate.
“Banks, brokerages, … all of these institutions have a fiduciary obligation to their clients,” says former U.S. District Court Judge Noel Hillman, who now practices with law firm Gibbons P.C. in their white collar & investigations group. “When it comes to what data they gather, who sees it, and how they use it, these companies put themselves at regulatory and financial risk if they tread too close to the line or cross over it.”
Hillman’s law firm colleague, attorney John Wolak, CIPP-US, emphasizes that compliance obligations with existing laws remain firmly in place regardless of newer data usages. “Those types of things don’t go away just because you’re feeding a transcript into an AI model,” he says. “In fact, it gets even more heightened when you’re using an AI model because of where the model may reside and who may have access to the data.”
AI’s Growing Role in Financial Services
In recent years, AI solutions like natural language processing have attracted companies seeking to innovate and remain competitive. Customer service is one such area integrating AI capabilities to better serve bank customers, clients or members. A typical use case involves an inbound call or online chat being transcribed in real time and sent through an AI model to improve both the customer experience and the model’s efficacy.
Most users are notified at the onset that their call or chat that the interaction may be recorded for “training purposes.” But there is an assumption the training involves humans, not machines. That’s why full, explicit disclosures are important now.
“The consumer should never feel fooled,” says Christina McAllister, a Forrester Research analyst who recently co-wrote a blog post on privacy advocates’ focus on contact centers’ use of AI without explicit consent.
AI has a role in improving and personalizing financial services, she said, so long as everything is done ethically. “This isn’t a reason for companies to shy away from leveraging AI in the contact center. But they need to be concerned with how they create a transparent disclosure and consent mechanism for customers,” she says.
Learn more about monitoring AI in banking:
- How ‘Voice Cloning’ Will Disrupt Client Verification
- AI Now: Real-world Lessons to Enhance Your Marketing, Immediately and Safely
- An AI System Built for Everything is an AI Built for Nothing
Contending With a New Attack Vector
While large financial institutions may have the resources to build their own internal AI models, most others turn to vendors to handle the heavy lifting, particularly developing algorithms and machine learning models required for advanced analytics and automation. In recent years, the rise of generative AI, particularly publicly accessible large language models (LLMs) like OpenAI’s ChatGPT, has expanded the threat landscape in numerous ways.
LLMs train off queries and data scraped from both public and private sources to refine responses, which means any proprietary information fed into a request is at least theoretically now public. That’s why financial institutions should approach any public AI tool with extreme caution. Not only could private data be leaked, but AI algorithms have been known to give biased results due to flaws in their development that impacts decisions, such as someone’s creditworthiness. Additionally, LLMs can “hallucinate” and convincingly dispense fake or false information.
Threat actors use LLMs to refine their attacks, whether it’s auditing malware code, fixing grammar and punctuation in phishing messages, or tricking AI into revealing customer information through a technique known as a prompt injection.
“That’s an entirely new threat vector, and it’s something that’s not necessarily easy to defend against because if you want to expose that LLM to the public — or at least expose it to your business partners —then the bad guys could potentially break in,” Giordano says. “It’s really a semantic attack where they make a very clever query and get information based upon that. It’s something that must be dealt with, and I’m not sure how well it is.”
Building Trust Through Simplified Language and Clear Consent
From a cybersecurity perspective, financial organizations should be conducting regular risk assessments to ensure they have the appropriate safeguards to prevent data breaches and data leakages. This can include red teaming, or adversarial role playing, to find exploitable weaknesses before the bad guys do. The lawyers interviewed also recommend doing due diligence when vetting vendors and testing a chosen AI solution for biases through audits.
Additionally, financial organizations should anonymize or remove personally identifiable information not needed to yield accurate results. And remember: “There is no 100% security solution. There is no 100% privacy solution,” Wolak says.
Disclosure and transparency are paramount.
“If a company’s doing something with customer information that isn’t disclosed, not only may that raise a regulatory and compliance issue, but more importantly it may raise a customer relationship issue,” Wolak says. “You want to treat your customers in a way that demonstrate to them that you are sensitive to the fact they are entrusting you with a lot of information and expecting you to act in their best interests.”
Adds Hillman: “At the end of the day, it boils down to consent, consent, and more consent. The more fulsome you are in your disclosures and the more honest you are, the less risk you have.”
McAllister recommends financial organizations review and update their privacy policies and disclosures to better align with how they now handle customer data. Then look at their vendor contracts carefully to understand any given permissions for data use, given these agreements may have been signed before the rise of AI.
Within the AI models themselves, examine – and use – robust redaction mechanisms and understand how customer data is stored once transcribed and recorded. All experts interviewed by The Financial Brand for this story recommend pushing to remove clauses giving vendors liberal use of any data shared with them.
McAllister cites a Forrester study that examined consumer reactions to two major banks’ privacy policies – one using plain language and another, legalese and jargon. Thirty of the 32 research participants said they were more willing to open an account at the bank with a clearly understood policy.
“I think it’s worth companies investing in making that experience really clear, especially if their competitors aren’t and if consumers are becoming more sensitive to AI use and data use,” she says. All these technologies will impact them when they’re sharing sensitive financial information. I don’t think there are any downsides to being more transparent and making things easier for your consumers to understand.”
Anne Saita is a technology writer who specializes in cybersecurity and data privacy.