Hook
Forty-four percent. That is the number that matters — not the 57%.
A British fintech firm called Saturn pushed 121 personal-finance questions through the leading large language models, repeated each one five times, and collected more than ten thousand answers. The headline: a 57% failure rate overall, 88% on the hardest questions, and even the top paid model — labelled "Claude Opus 5" in the coverage, a name I cannot verify against any public release — failing 39% of the time. Those figures moved fast. But the quieter number is the one that should follow you into every conversation about AI and money: 44% of users believe AI-generated financial information is regulated. It is not.
The advice lives in a legal vacuum. No ombudsman. No compensation. No liability channel when the number is wrong and the pension is smaller. That gap — between what users assume exists and what actually exists — is the real defect. It is also a defect the crypto industry has inhabited for a full decade, and the on-chain ledgers have the receipts.
Context
Saturn's methodology deserves the same scrutiny I apply to any protocol claim. The premise is simple: take real questions a person might ask about tax, savings, and pensions, run them through free and paid conversational models, and score each answer on correctness, on awareness of current tax rules, and on disclosure of risk. That last criterion matters more than it looks. Including "missing risk warnings" as a failure condition mechanically inflates the failure rate, because consumer chatbots routinely omit disclaimers — so any domain that legally requires a caveat will score poorly by construction, independent of reasoning quality.
The corroborating survey work is where the picture sharpens. An EY study found 49% of users already lean on AI to help with savings and investment decisions. PensionBee reported that nearly six in ten users would act on an AI recommendation without independent verification. An FCA study found 56% of respondents trust AI more than they trust television and radio — while, in the same cohort, 44% wrongly assume the financial information is regulated. High trust, low comprehension. That is not a knowledge gap. It is a structural mispricing of risk, and the market is charging users for it without labelling the product.
There is a regional asymmetry buried in the setup that nobody has priced. If the 121 questions are weighted toward UK tax and pension rules — and the HMRC penalty example suggests they are — then models trained on data with a thin British footprint are being marked down for a knowledge boundary, not a reasoning failure. That distinction changes the remedy entirely: one is an engineering hole, the other is a geography problem. The coverage treats them as the same number.
The downstream consequence is a grey zone. If the UK tightens, AI advice migrates offshore, and users reach across borders for guidance no single regulator owns. Meanwhile the fix that would actually help — high-quality, domain-specific training and annotation in tax and pensions — is a supply chain almost nobody is building, because it is unglamorous and slow.
Core
Here is where my own work becomes relevant. I spent four months in 2017 reverse-engineering the contract logic of a failed ICO — fifty thousand lines of C++ — and I learned that the most dangerous component in any financial system is never the one that visibly breaks. It is the one everyone assumes is being watched. Four years of ledgers never lie, only distort — and the distortion in this AI-finance test is the same one. The failure is technical and quantifiable; the loss is behavioural and invisible until it is already booked.
Map the failure modes and they line up precisely with the on-chain patterns I have catalogued. Saturn names three: arithmetic errors, ignorance of tax-rule changes, and fabricated rules that do not exist. Translate those into ledger terms. Arithmetic error is a rounding bug in a settlement contract — small, cumulative, catastrophic at scale. Ignoring tax changes is a stale oracle feeding a lending protocol a price from last quarter. Fabricating a rule is a smart contract executing a function the whitepaper never implemented but the documentation implied. The code whispered what the whitepaper hid, and the user, trusting the surface, never audits the internals.
The gradients in the data are more instructive than the headline. Free models failed 63% of the time; paid models 49%. Fourteen points for a subscription fee — and on the hardest questions, the free tier collapsed to 93% failure. That is not a pricing curve. That is a capability ceiling being sold as a service tier. When I mapped recursive collateral cascades across Compound, Aave, and Uniswap in 2020, I found the same shape: a system that looked robust under normal load and revealed its single point of failure only under stress. The AI finance test is a stress test, and the stress test says the load-bearing wall is missing.
Now bring in the regulatory dimension, because this is where the crypto parallel stops being metaphor and becomes method. The CEO of Saturn has publicly called on the FCA to act, noting that AI financial advice is unregulated and that consumers lack the redress a human adviser would carry. He is correct on the law. He is also a stakeholder with a product that would benefit from the rules he is requesting — a fintech firm whose competitive position improves the moment "licensed and supervised" becomes the entry ticket. I am not accusing anyone of bad faith. I am pointing out that the entity auditing the failure is also positioned to sell the cure. I flagged the same structure in 2021, when whale tails flickered in the NFT gallery shadows and 12% of a blue-chip collection's supply sat in thirty wallets that consistently bought the dip and told the market the dip was organic.
Consider the sequencing. Satoshi's original framing — peer-to-peer electronic cash, no intermediary, no permission — carried the same implied promise this AI adviser does: the tool is neutral, the code is law, verify for yourself. Then the ETFs arrived, and Bitcoin became a Wall Street instrument. The promise of self-custody survived in the protocol and died in the user base. AI financial advice is replaying that arc in compressed time. The technology is sold as a democratising force; the adoption data — 49% using it for real savings decisions — confirms the demand; and the reliability data — 39% failure at the very top — confirms that the trust is arriving years before the competence. Adoption sits near 50%. Reliability failure sits near 50%. The two curves have not converged, and nothing in the current paradigm guarantees they will.
That last point is what the coverage underplays. In crypto, the mitigation for a fragile protocol is a vertical fix: better oracles, audited contracts, time-locks, redundancy. The same applies here. Retrieval-augmented generation wired to a live, authoritative tax source; tool-calls to a deterministic calculator instead of letting a probabilistic model perform arithmetic in its head; structured output that refuses to answer outside its verified knowledge boundary. These are engineering remedies, and they are not mysterious. Their absence from the discourse tells you the conversation is being steered toward "trust the model" or "don't trust the model" — a binary that serves the incumbents on both sides and leaves the actual fix unbuilt.
And the layer-two lesson applies cleanly. I have argued for two years that "decentralised sequencing" has been a PowerPoint slide while the sequencer remains a single centralised node in practice. The pitch and the plumbing diverged, and users paid the spread. Financial AI is now making the identical trade: the promise of a knowledgeable, always-available adviser, delivered by a system whose core reasoning is non-deterministic, boundary-blind, and unaudited for the exact question being asked. When I built the 2025 institutional flow tracker, 70% of institutional volume landed during low-volatility windows — smart money waiting for calm, never buying the headline. Retail does the opposite. It buys the narrative at the exact moment the narrative is least reliable, and an AI chatbot that answers with confidence is the most efficient narrative-delivery machine ever built.
Contrarian
The obvious reading of this report is that AI cannot be trusted with money. The data-driven reading is narrower and stranger: the report may be measuring the wrong failure, and it was written by a party with an interest in the answer.
Start with correlation against causation. A 57% failure rate is a measure of answers; it is not a measure of harm. The two are linked by a chain the report does not trace — user receives bad answer, user acts on it, user cannot recover, loss is material. Every link in that chain is assumed, not evidenced. I have watched a flash-loan attack predicted with 95% accuracy and still mistimed by the market, because the model was right and the humans were slow. Failure-rate theatre has the same half-life.
Then there is the unreported number. Every question was asked five times. That repetition was designed to capture variance — and variance is the finding nobody published. For a financial decision, inconsistency is more dangerous than error. An AI that is wrong the same way every time can be calibrated; an AI that answers the same pension question five different ways gives the user no way to know which version to act on. The report scored correctness and stayed silent on consistency. That is the blind spot I would have led with.
And the stakeholder problem is not minor. Saturn ran the test, Saturn's CEO framed the conclusion, Saturn sells the alternative, and Saturn did not disclose its own product's failure rate. I have seen this exact silhouette in token launches: the team that publishes the damning audit of competitors is usually the team raising the next round. Read the method, not the press release.
Takeaway
The forward question is not whether AI will advise on money — 49% of users have already decided that. It is whether the framework to catch the error will arrive before the losses do, or, as with crypto, three years after. Watch the FCA's next move, watch whether the replication data holds the variance, and watch where the first real harm claim lands. The ledger always settles. The only variable is who is holding the position when it does.