A finance professional warns that opaque AI systems in accounting could trigger another Enron-scale collapse by making audits untrustworthy. Black-box AI creates the same complexity that led to Sarbanes-Oxley after the original Enron and AIG failures. The fix: human-in-the-loop controls and SEC rules requiring auditors to personally attest to AI-generated numbers.
They told us that AI will solve the efficiency problem. They told us that we should master AI before it masters us. But the deeper I dig into AI and finance, the more I see a darker truth. The black box effect. This black box is threatening to bring about the same collapse in investors trust we saw in 2001 with Enron. So what is the black box effect in relation to accounting with AI? According to this IBM article, a blackbox AI is an AI system whose internal workings are a mystery to its users. Users can see the system's inputs and outputs, but they can see what happens within the AI tool to produce those outputs. Now fast forward to today. Companies everywhere are adopting AI not just for marketing and chatbots, but inside of the finance and accounting systems that run our businesses. Agentic AI can process millions of transactions and make decisions faster than any accountant ever could. Which sounds amazing, right? But here's the catch. AI is often a black box. It spits out results without a clear explanation of of how it got there. And in accounting that's a big problem because transparency and audit trail are non negotiable. That's why we see articles like this one from CFO Drive titled Deploying AI in accounting poses 12 audit risks. The researcher found that auditors for companies that use generative AI often confront a black box challenge where they can either interpret or explain how the technology generates information. The whole point of an audit is to provide credibility to the financial information. And when this credibility is gone, investors cannot fully trust the financials. So what is the relationship to the Enron collapse? Enron went from a Wall street darling in the biggest fraud in US history almost overnight. And on its way down it took down the public's trust and SEC regulations. Enron purposefully created complex business dealings and entity structures in order to bury their fraud. Which in turn caused the SEC to issue the many rules of Sarbanes Oxley known by the short name Sarks 404 rules. Another prime example of creating complexity to cover fraud is AIG which nearly filed for bankruptcy in 2001. AIG's near collapse was caused by massive losses from its Financial products division due its significant exposure to complex derivatives. For me, based on my experience as a financial auditor and controller, I believe that using AI and a black box, as many companies are doing today, will land us in a socks 2.0 moment where the public no longer trusts financials that are produced using AI agents. The second and more important issue in my view is that auditors themselves are over relying on AI to audit the same information that was produced by AI. This is like having your cousin vouch on your behalf to the girl that you like and tell her how great you are. This article by Racontour finds that new evidence suggests that auditors are using AI systems they cannot fully control or explain. This threatens to erode the credibility of the auditing profession. Look, I understand why large companies need to use AI to produce reports. Just have a look at the complexity of some of these businesses. Consider Amazon's business model. Massive scale retail plus marketplace. Millions of products, millions of third party sellers, billions of transactions. To top it off, they have prime subscriptions that gives free delivery which creates accounting nightmares when you have to reduce the merchant's revenue. Without AI tools you couldn't even venture to create the this elaborate business model. So we need AI, but we still need an adult in the room. That's what we call human in the loop. What is human in the loop? It's when an AI system isn't fully automated, but instead has a human involved at key points in the process to review, approve, override or provide feedback on AI's output. Why this matters in auditing, it prevents over reliance on AI. By ensuring a trained auditor evaluates whether the AI's conclusions make sense, it adds accountability. A named person is responsible for judgment and not just the system. And it helps catch AI's hallucination biases or data errors before they make their way into financial reports. Another way to look at this goes like this. Investors are humans. They have human level comprehension. For them to trust a financial report, another human expert needs to have read it and use their own human level comprehension to judge it and conclude that it is fair and accurate. Can you imagine yourself investing your money in a company relying fully on an AI tool that concluded that everything looks legit? I don't think so. So here are my predictions. The SEC and the PCOB, which are the two main governing bodies for public companies in the U.S. will likely issue a series of rules that will be as stringent as SOX404 in 2002. Because at the end of the day, investors don't invest in efficiency, they invest in trust. Remember that SOX 404 require the one management establish, maintain and annually assess the effectiveness of internal controls over financial reporting and 2 management must issue a report confirming this assessment and 3 external auditors must independently evaluate and attest to management assessment. So we will likely see new rules around human in the loop aspect where companies will need humans to attest to the work of AI. So if you think AI is taking over finance. Remember, investors need to trust financial reports. Can you as an investor put your hard earned money into the hands of an AI agent? This might happen down the line, but as of now, my personal answer is a big no. See you next time. If you enjoy my content, I think you like to dive deeper in my career tailored courses. I train next generation controllers and FPA professionals using hands on demo and exercises in my Controller Academy. My courses are CPE eligible which means all curriculums are meticulously created for the biggest impact on your career. Go check it out. Link in the description and I'll see you on the inside. SA.
