The phone lights up during dinner, and the voice belongs to a stranger who knows too much. A relative in Nakuru, hundreds of miles from the person who actually borrowed, hears a caller recite a loan balance down to the last shilling and warn that an arrest is coming. The relative never took the loan and never agreed to vouch for it. Their number was simply harvested from a borrower’s phone the moment an app was installed, months earlier, when the interest still looked reasonable and the approval was instant.
The same silent machinery runs far from any lending app. In an American nursing home, an elderly patient is told her rehabilitation coverage ends on a date no doctor chose , a model picked it. In a drugstore aisle, a shopper is followed, searched and ordered to leave because a camera decided her face matched a thief’s. On an airline’s website, a grieving passenger is told about a refund that does not exist. None of these people agreed to be handled by an algorithm. Most never knew one was in the room.
Put the cases side by side and a single shape appears under all of them, one that survives every change of industry: AI deployed on people without their consent is not a free efficiency. It is an efficiency financed by an invisible loan against trust , a loan the business never records as a liability, and one that is called in full, all at once, on the day it is discovered, with the interest rate set by whoever discovers it. Call it the trust overdraft. The gain lands in this quarter’s numbers. The cost is deferred, kept off the books, and it does not arrive as a gentle decline.
What the borrower actually consented to
Start with the lending case, because it states the consent problem most sharply, and because Africa has run the largest live experiment in it. The credit ecosystems built on Kenyan products like M-Shwari, Tala and Fuliza went viral across the continent, and with them a set of collection tactics that spread from Nairobi to Lagos to Johannesburg. The models are genuinely clever at the part everyone can see , instant loans, no collateral, credit for millions the banks ignored. The other half runs quietly. When a borrower falls a single day behind, the system reaches into the phone and messages the entire contact list, sometimes labelling the borrower a thief to people who never asked to be involved. One Nigerian borrower took about twenty dollars at a fortnightly rate that annualises past 560 percent, then found her contacts turned into collection agents.
The tempting objection is that the borrowers consented, they did tap “allow,” most without reading, desperate for the loan. But look at what the consent covered, and for whom. The borrower agreed to a loan. The people in her phone agreed to nothing, and it was their trust , the ordinary assumption that being saved in a friend’s contacts does not make you collateral , that the model spent.
Now watch how quickly the same gap opens in industries that never asked for a signature at all. Between 2012 and 2020, the American drugstore chain Rite Aid ran facial-recognition surveillance across hundreds of stores, matching shoppers against a homemade watchlist and never telling them the cameras were doing it; employees were actively discouraged from revealing the system existed. Acting on false matches, staff followed customers, searched them, ordered them out, called the police, and publicly accused them of theft, sometimes in front of their own families; the technology misfired most often against women and against Black and Asian shoppers. No one tapped “allow.“ They walked into a pharmacy and were entered into a criminal database by a camera.
Consent to a transaction is not consent to being processed, and often there is no transaction to consent to at all. That gap , between what a person agreed to and what the machine did in their name, is where the overdraft is drawn.
When the machine speaks for you
Move the idea into a boardroom in a wealthy country and it stops looking like predatory fintech or reckless retail and starts looking like ordinary cost-cutting , which is why it is instructive.
Air Canada put a chatbot on its website because a chatbot is cheaper than a call-centre agent, and it answered questions in the airline’s voice, on the airline’s page, with the airline’s authority. After his grandmother died in November 2022, a passenger named Jake Moffatt asked it about bereavement fares, and it told him he could apply for the discount retroactively, within ninety days , a policy that did not exist. He booked on that assurance and was refused the refund. He sued.
The airline’s defence is the part every executive should tape to a wall. Air Canada argued that its chatbot was “a separate legal entity responsible for its own actions,” and so the airline could not be blamed for what it said. The tribunal member called this a remarkable submission: a chatbot is not a separate person, it is simply part of the company’s website, and a company is responsible for everything on its website, whether the words come from a static page or a machine. The award was small , about C$812, but the principle was not.
Here is the overdraft in its cleanest form. Air Canada captured the visible saving, automation instead of salaries, by quietly borrowing against one specific trust: the customer’s belief that when the company’s website speaks, the company means it. Running the counterfactual, marked as a scenario , had the airline disclosed the bot’s fallibility and simply honoured what it told a grieving man, the episode costs nothing anyone would remember. What made it a landmark was the instinct to disown the machine at the moment of reckoning, the reflex of an organisation that had spent trust it never knew it was spending.
The overdraft that took down a government
The largest version of this story was not run by a company at all, and it shows what the debt costs when the account is a whole population and the drawdown runs for years.
The Dutch tax authority wanted to catch childcare-benefit fraud, so it built a self-learning risk model to flag suspect claims. Among the factors it weighed was whether a claimant held a second nationality; parents with dual citizenship were systematically scored as higher-risk, and officials treated the flag as guilt. Between 2005 and 2019, roughly 26,000 parents were wrongly accused of fraud and ordered to repay their benefits in full, often tens of thousands of euros, driving families into ruin. Children were taken from some of these homes into state care. None of these citizens had consented to being ranked by a machine on the basis of where their parents were born. Most never knew the machine existed.
The efficiency was real while it lasted. Fraud recovery had a budget and a target, and the model served them , until the account was called. In January 2021, the entire third cabinet of Prime Minister Mark Rutte resigned over the affair. The state acknowledged institutional racism, and a parliamentary inquiry branded the episode an unprecedented injustice.
Whether the discrimination was designed or merely emergent is genuinely contested, the tax authority insisted it held no racial data and so could not have been profiled by race, while investigators and courts concluded the effect was discriminatory regardless. That dispute matters legally. It does not touch the overdraft, because intent is not the mechanism; concealment, scale and time are. And the coda is the sharpest part: the same governing parties returned, a version of the same profiling quietly resurfaced in vulnerable neighbourhoods, and civil-rights groups were still fighting it years later. Trust overdrawn at this scale does not reset when a cabinet falls. That is the interest: you repay with the added cost of everyone now assuming the worst about the next system you build, deserved or not.
Why the debt stays invisible until it doesn’t
The reason intelligent organisations keep taking this loan is that it is genuinely hard to see. Trust is not a line item; it shows up only as its own absence, suddenly, later. And no case exposes the accounting more plainly than an American health insurer, because there the sum was allegedly worked out on purpose.
UnitedHealth used an algorithm called nH Predict to project how long an elderly patient “should” need in a nursing facility, and , according to a class action and the reporting behind it , pressured staff to keep actual stays within one percent of the machine’s estimate, cutting off payment when the number ran out. The tool’s judgement was poor. Plaintiffs say that when patients bothered to appeal, roughly nine in ten denials were overturned. Here is the overdraft stated as arithmetic: the company allegedly kept using the model anyway, because only about 0.2 percent of policyholders ever appealed , so the savings from the many who simply paid out of pocket or went without dwarfed the reversals won by the few who fought.
A US Senate subcommittee later found the insurer’s denial rate for this kind of care more than doubled after the tool came in. UnitedHealth denies the algorithm makes coverage decisions, calls it a guide, and says the suit lacks merit; a judge has let the core contract claims proceed. But the structure is the point: a concealed system, a population that mostly will not push back, and a saving booked today against a liability recorded nowhere.
That liability compounds in the dark until something drags it into the light , a headline, a fine, a five-year ban, a court order. In 2026 a federal magistrate ordered UnitedHealth to hand over documents showing how nH Predict actually works. That is the moment the loan is always called: when the machinery is finally made to explain itself.
This gives the pattern a sharp, testable boundary, which is what separates a real idea from a slogan. Concealment stays cheap only while it goes undiscovered, and the odds of discovery climb with scale and with time. So hiding an automated decision can be survivable, even rational, for a small, brief, low-stakes deployment. It turns ruinous exactly as the deployment grows large, persistent and consequential , the direction every successful AI system is pushed by its own economics. The organisations most tempted to run on hidden leverage are the ones scaling fastest, and scale is what calls the loan. The strategy fails hardest where it looks like it is working best.
There is an honest tension worth naming, because the tidy answer , just disclose everything , carries a cost of its own. A fraud model published in full also teaches the most resourceful how to slip beneath it. Transparency is not free. But that is an argument for careful disclosure: telling people they are being assessed by a system, and keeping a human accountable for the result, without handing over every weight. The overdraft is not the price of using AI. It is the price of hiding that you are.
Consent is the cheapest capital in the AI economy
Which turns the whole subject, at last, from a warning into an opportunity , because the organisation that treats trust as a balance-sheet item holds a structural cost advantage over the one that doesn’t, and that advantage is widening in every industry at once.
The rules are arriving, and they converge on a single demand: tell people. Africa, having felt the harm first, is legislating fast. Kenya made lender harassment a criminal offence from January 2025;its data commissioner has fined lenders for raiding phonebooks, and by 2025 more than half the complaints reaching the office involved lenders misusing data. Nigeria’s tougher new lending rules force every operator to register, ban automatic pre-approved loans, mandate plain disclosure of terms, and outlaw harassment in collection. The wealthy world is moving along the same axis. In the United States, the Federal Trade Commission barred Rite Aid from facial recognition for five years and ordered it to notify the people it had secretly enrolled.
California has required businesses to disclose when a bot is standing in for a human since 2019, and Colorado’s incoming AI Act reaches exactly the terrain of these cases, demanding disclosure whenever AI drives consequential decisions in finance, healthcare, insurance, housing or employment. The patchwork is contested and still being redrawn, but the direction is one-way. A business built on disclosed, human-backed AI is not scrambling to comply, because it was already paying as it went. A business built on the hidden version is carrying an undisclosed liability that any new statute or lawsuit can call.
So the move for a founder, a bank, a hospital, a retailer or a regulator deploying these systems now is not to slow down. It is to refuse the overdraft on purpose: to say plainly that a machine is in the loop, to keep a human who will answer for what it does, and to treat consent not as a checkbox but as the cheapest capital available , trust paid for openly, in small amounts, as you use it, rather than borrowed in secret at a rate you neither set nor see until it comes due.
The African lenders thought they were selling loans. The insurer thought it was managing costs. The drugstore thought it was stopping theft. The airline thought it was saving on staff. The state thought it was catching fraud. Each captured something real and immediate, and each was, without recording it anywhere, running up a debt against the one asset none of them owned outright: the belief, on the other side of the interaction, that someone knowable and accountable was there.
That belief is the most valuable thing any organisation holds, and the easiest to spend without noticing. It is also where Africa has an unusual chance. The continent adopting these systems now is early enough to choose, having watched what discovery costs everywhere else , and it can build on something the imported templates lack. Long before the loan apps, African credit ran on cooperatives and community funds rooted in social trust, reinvestment and accountability; the opportunity is to digitise those traditions rather than replace them with extraction dressed up as inclusion, and, ideally, to set a common standard across the African free-trade area before the next wave of systems is built rather than after. The machine has made trust cheaper than ever to spend. The businesses that last will be the ones that decline to borrow against it.
English 
简体中文 
Čeština 
Deutsch (Sie) 
Français 
