Ep 772: AI You Can Trust: When Good Enough Isn’t Actually Good Enough

This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life.

You know what most companies pitching AI agents aren't actually doing? Deploying AI agents. Sage just did the opposite at Sage Future. They expanded their AWS collaboration to put real AI agents into real finance workflows built on Amazon Bedrock Agent Core and available in the AWS Marketplace. So that means enterprise-grade AI for your finance team without the IT headache. Check it out at sage.com.

What good is AI if you can't trust it? I think as agentic AI grows at an unprecedented rate, Sometimes there's a rush to just do more, create more. But when was the last time you went through and audited your AI? Because if it's good enough, is that good enough? Well, that's what we're gonna be talking about today on Everyday AI. For our podcast audience, you might not see this, but, uh, we are at Sage Future and I'm excited for today's show. So, uh, we're gonna jump right into it. So help me welcome my guest, livestream audience. We have Jeremiah Edwards, the head of Sage AI. Jeremiah, thanks so much for joining the Everyday AI Show.

No, great to be here in San Francisco with you.

All right, so we're gonna get to everything that's new here, uh, at Sage, uh, at Sage Future. But let's start with the big question. When it comes to explainability in AI, when it comes to trust in AI, how can people, whether they're business owners, whether they work in finance, How do you actually get to the point where it's not just a race to do more, produce more, more outputs, but how can you get to that actual trust and explainability piece?

No, it's super important. I think, uh, focus on correctness and focus on explainability is, is a big gap in a lot of AI tooling today. Um, you know, we kind of stopped and said, oh, this, it's amazing that the, the dog can play piano, but, but does it sound good? Um, I think, you know, I would encourage every business user to look at their AI systems and say, okay, how did it calculate these results? Is it correct? And is that better than the process I was using before? Is it really saving me time? Because, you know, there's a lot of noise out there.

Yeah, absolutely. And let's actually rewind a little bit. So, and zoom out. So for those that aren't familiar with Sage, obviously a global player in the game, but, you know, explain a little bit, you know, what Sage is and what you all do.

Yeah, so Sage makes accounting and back-of-office software for small and medium businesses around the world. You know, we focus on that accounting function and give a CFO tools that they need to run a business.

Yeah, absolutely. And with Sage Copilot, obviously a lot of new announcements and new capabilities coming out of Sage Future. Can you maybe highlight a little bit what was announced? Because I know there's a lot, right? But highlight maybe a little bit on the Sage Copilot side, the new capabilities that Sage customers now have.

Yeah, so Sage Copilot is our user interface for all things generative and agentic AI. And, you know, we released that a couple of years ago, but these days there's this big transition from generative, you know, kind of chat agents to agents that can really take action within the software that people are using. So in particular, our finance intelligence agent is helping people close the books faster, keep track of their budgets. And answer questions about their business that they couldn't before. Yeah.

And so last year, you know, I had a great conversation, uh, with Aaron and, you know, at that time it wasn't that the models themselves, uh, you, you know, were the biggest hurdle, but, you know, looking at the models from a year ago to the models that power the technology today, you know, there's definitely been a step change, uh, you know, more agentic by nature. Smarter, more capable, et cetera. On the flip side, for, you know, people in finance, because, you know, a lot of day-to-day operations, they start and they end with the numbers. How do you balance that, you know, the rapid growth in agent capabilities even within Sage Copilot with the potential for maybe getting that one number wrong? 'Cause if you get that one number wrong and that's your starting point, things can go downhill quickly.

Yeah, no, I think the, there's been this kind of obsession for the last 3 or 4 years on the model, the model, the model, the model. And, and I think that's in one part justified because Anthropic and OpenAI are, are releasing some really cool stuff every day, every week. It gets a little bit better, a little bit faster. But, um, in order to, to get like finance bulletproof accurate numbers, it's not just the model. I think there's an entire system of software, uh, that has to come together. And more and more around agentic AI, I think the industry is realizing this and catching up. It's not the model, it's the model plus the agentic harness, plus the tools that it has access to and the data sources that really matter. And the, the best AI model in the world cannot, cannot do your accounting for you if it's not connected to the best accounting software in the world. Um, connected via, you know, standard protocols like MCP or Um, and without that, you're, you're, you have no hope of getting, getting a good answer. Yeah.

And, and that, I think that's a great point because, you know, you even brought up things like the Model Context Protocol and the standard. Um, right. And it seems like there's also, for people that maybe don't have a background in AI and machine learning and math, right, like you, but maybe business, uh, business owners in the C-suite that want to be involved and understand how the AI interacts with their numbers. Can you even explain a little bit and specifically, you know, even about the math capabilities of these models, right? Because maybe 2 years ago, people, you know, you see all these kind of horror stories, people share something like, oh, you know, this model can't add, you know, 2 2, but we're not there today. Can you just give everyone else a quick update on just where these foundational models are now when it comes to being able to understand the basics that you need to have in finance?

Yeah. So all, all the LLMs in the world were trained on a, on a vast corpus of text data, right? They know how to complete text and, and at a very fundamental level, they are translators. They're able to translate from a question to an answer. They run into a lot of trouble because there's a lot of, there's a lot of lies on the internet. I don't know if you knew Jordan, but I've, I've heard, uh, not everything on the internet is true and, and Errors like that, factual errors are in their training data. The training datasets are so big that that can't be filtered out or corrected very easily. That process takes a long time. So enter the dragon. In order to get correct answers out of these systems, we have to give them access to tools. We have to give them a calculator to do arithmetic. And without that, you're definitely gonna get incorrect results. And I think it can be hard to find the line where, you know, that training data was good and correct and where things start to go off the rails. Off the rails.

So I do wanna get back to making sure your AI that deals with the numbers gets on the rails. But I do also wanna talk about, uh, a pretty big partnership that you guys announced, uh, with PwC. And in that partnership, you know, the study that came out that said 71% of finance leaders would reject AI that can't explain itself, right? So talking about this explainability, there's obviously things you know, at the baseline model level and then at Sage Copilot level. But can you just talk a little bit about the importance of explainability and trust, right, when it comes to the numbers? I know, like, it maybe just seems like it goes without saying, but I want you to say it directly because I think some people need to hear it.

No, I— at the end of the day, the CFO or a small business owner is accountable for the integrity of their books. You know, when their taxes are filed, when an audit comes up, it's, it's not Claude that's sitting in the hot seat with the IRS. It's the CFO. And because of that accountability, uh, the expectations that they have of any software system, including AI, are very high. So if it's not completely correct, it's, it's not good enough. Um, close is not good enough. And because of that, I think, you know, it's critically important that, that in all of the software that Sage builds, in all of the AI that we bring to market, We think of explainability as a, as a first-class design principle. And, and what, what does that mean? It means that every step the AI agent took to answer a question is there. It's fully auditable. It's, it's right there for someone to see, including data sources, including API calls, and including reasoning that these models can bring on top of that. Yeah.

And you know, even speaking of reasoning, right? So not to get too technical for our non-technical audience, right? But what does it mean now that the, you know, the models used by Sage Copilot, you know, can reason by default, right? Whereas 2 years ago, you know, that wasn't really an option for anyone, you know. So what does that mean when it comes to increasing the trust and increasing the explainability when you can see here's how Sage Copilot got to this answer?

Yeah. So I'll use another term of art. I hope it's approachable, but I think more and more chain-of-thought reasoning, you know, is a first-class consideration for anybody using these systems. You know, you're not just calling an LLM once and getting an answer. I hope not. You're asking it to critique its answer. You're asking it to dig in. You're asking it to look at the other tools available and pick the right one. And you can look at, and hopefully everyone wants to surface this reasoning to users, we certainly do inside Sage, you know, what the model was doing. And why it was taking the actions that it was. But that's, that's the only way that you're going to get AI that a, that a CFO will use. Yeah.

And, and yeah, FYI, you know, audience, you, you, you've heard me be the old man shaking my fist on the chair all the time about looking at the chain of thought and, you know, just, just FYI, you know, Jeremiah, one of the smartest in the game, said to do the same thing. But also you bring up a good point, the transparency of being able to look at that inside of Sage Copilot, which brings me to time. It brings me to agency, you know, for people, you know, I consider myself mid-career, right? But for those people. Yeah, yeah. But, you know, for those people that have, you know, been, you know, a CPA for a decade or a CFO for, you know, 25 years, you know, and now they see these agentic models that can do the type of work that they've kind of built their career around, right? So I think people look at it two ways. One way they say, this is great, right? And I can, you know, check the work inside of Sage Copilot. It explains how it thought. Uh, great. The other people are like, wait, what does this mean for my job, for my time, for my agency, for my expertise? I know this is a much bigger question that we could probably talk about for hours, but How do you begin to unravel what this means for your own experience and for that human agency?

Yeah, well, first of all, I want to acknowledge that this is a universal problem. I mean, as a mathematician, as a software engineer, these are problems that I also face. You know, wow, that the computer's getting awfully good at what I used to do for a living. Goodness. Um, but no, I think, uh, within, within finance, within accounting, You know, there's, there's actually, you know, a shortage of talent these days. Fewer people are coming out of CPA programs ready to jump in and, you know, and help small and medium businesses get that job done. And I think AI is kind of rising to meet that challenge and fill some of those gaps. And to anyone who's fearful of that transition or looks at this and is like, I'm not sure where this is going, I'd say you don't have to, to eat the whole enchilada at once. I think you don't have to use every, every piece of AI out there. All at once. And I, I don't advise that people do. I think, um, you can start with task-based AI, process automation. Um, you know, I love to, to focus on kind of AP, accounts payable for that. Um, people used to have to, to look at invoices and manually enter data into the accounting system. No human has to do that anymore. We have tools for that now. And then that's great. That is AI. It, it, you know, a lot of people don't think of that as AI anymore. Because, because it's been around for a while, we've gotten comfortable with it. But, um, for anyone who's, who's skeptical of that transition, start there. It can save you a lot of time. I think our customers say, you know, it can double the number of invoices they can process every month, which is, which is huge. Uh, getting to your point though about, you know, how does that change the nature of strategic thinking within finance and, and the role of the CFO? I'd say it frees up a lot more time for looking ahead instead of looking back. When you're crunched for talent and under a lot of time pressure, often you get into these sort of batch update cycles. We have to look at the last quarter, make sure everything's correct, turn it into an update. And by the time you do that, those numbers are out of date. And I'm hoping that AI, and certainly the AI we build at Sage, can help them be more forward-looking and really kind of meet this dream of continuous accounting where the data is available. The data's available to the people within the business who need it to make decisions. That takes work off the finance function and, and allows them to, to really focus on, on what matters for them. Yeah.

And you know, you bring up a good point because it does seem like as agentic AI becomes more powerful, as you know, layers of intelligence like Sage Copilot you know, help it become more explainable and, you know, more utility as well. Obviously, you know, you talked about, you know, it will free up a lot of time, but I'm wondering, you know, can you get maybe a little tactical and, you know, so where should people be spending that actual time? Is it, you know, more on the front end? Is it more on the back end? Should we be spending more time? Like, like me, I'm a dork. I love reading the chain of thought and, you know, trying to make it a little bit better or get better outcomes. You know, where should people, you know, specifically whether they're using Sage Copilot or not, but if they are using agentic AI in finance, where should those experts be zeroing their time in to make sure that they're getting the most out of the AI output, but also to make sure that they're still keeping those crucial skills sharp?

No, absolutely. I think, let me give a very finance-oriented answer first, and the short version is governance and control. Um, you know, any AI system, you know, that, that people are working with should give you the confidence you need to, you know, in the data, in the results it's bringing to you. And if it doesn't, don't use it. But the, you know, it should also give you a control plane for, for deciding, you know, using critical judgment to, to decide what to, what it should do on your behalf. And if, if the human is not in control of the AI system, you shouldn't use it. Yeah, that's another design principle. But ultimately, I think it's about a point we touched on earlier, which is accountability. You know, if the finance function is still accountable for the financial health of the business, you know, that means tasks are gonna be a lot more review-based. AI is the doer, the human is the supervisor, the human is the reviewer. And you have more time to do that because you're not stuck on kind of you know, lower-level tasks. Yeah.

And, you know, a lot of success probably comes from, you know, constantly auditing and improving the people, the process, and the technology. How do you schedule those, uh, you know, feedback loops, right? When the, when the technology does change so quickly, right?

Yeah.

How often should you be visiting change management within your financial organization? How often should you be you know, re-auditing some of these agentic workflows that are now possible this year that maybe weren't possible when I talked with Aaron last year, right? What does that kind of iteration loop look like in the age of AI everywhere being very impressive?

Quick pause.

Want to know where AI ROI actually lives? It's not the flashy demo. It's the work nobody wants to talk about. At Sage Future, Sage tackled the painful side of finance system rollouts, implementation, financial data migration, chart of accounts mapping, configuration, the work that stalls real transformation projects. Sage is using applied AI to make that faster, more accurate, and auditable with humans staying in control. That's responsible AI doing real work that actually moves time to value. If your finance team is daring down a migration, check out Sage at sage.com.

No, it's, I wanna answer that question two ways. One is because, you know, as a builder of AI systems, I certainly have opinions about, you know, how it can help save people time. But I'm also, you know, an AI user, right? We all are. We all wear that hat. Like I'm using AI to try to, you know, make make my workflows more efficient and also, you know, transmit that to my team, also learn from my team. Um, and I think how you do that effectively, you know, is gonna look a little different in different organizations depending on your risk appetite. But I think at the very minimum, I'd advise that, that everyone be looking at AI tools and thinking about how, you know, how that can change the operation of the team. And I really do love to think about it as the, at the team level and sharing information is the bare minimum here. I think what naturally arises out of that is some use cases, you know, are gonna be better than others, you know, and as a leader, you can identify, you know, which ones those are, which ones you wanna promote within your organization and incentivize people to, you know, to be creative, to bring those to you. Because otherwise I think you, you end up in a, in a situation where, you know, there's, there's shadow IT, there's, there's people using tools that are kind of outside of your visibility and control, and, and that's worse for everybody.

Yeah.

Um, you know, obviously, you know, vendors like Sage are gonna make the, the best AI integrated with our accounting systems as we can, but we, we exist in an ecosystem and, you know, while we can provide the best solution inside, inside Sage Intact, for example, you know, um, like people are going to, to look at other solutions and, and, and use them for different use cases. Yeah. And so you really have to give yourself the visibility into that and, and then advise from there.

So you brought up a couple great points there, both about shadow IT, uh, right? And which I think is important to, you know, or maybe let's just do this. Can you explain, right? Because I think a lot of people think, oh, I can only get the best models if I go to, you know, Claude or OpenAI or Gemini or whatever. Can you explain about how you just bring those best models inside? I think we maybe skipped over that point, but I think it's important to talk about, yes, you still get world-class AI capabilities within Sage Copilot.

No, absolutely. I think we, you know, we have, we have technology partnerships, you know, with, with all the best players. We, we do use models from, from Anthropic and OpenAI and, and Microsoft and open source. But you know, by, by integrating them into our AI platform, form by adding a layer of control and governance, we're able to ensure that all the answers that come from Sage Copilot are grounded in real data, you know, and are relevant to what the user's trying to accomplish. And that's, you know, that's really because we have the context within the accounting system and within, you know, the user identity of who's using that system, you know, to give them the best data and answers we can.

Yeah, absolutely. And that does, you know, help fight against the shadow IT, which I know is always a big problem. But you brought up something else interesting that I, I'd really like to discuss, just the risk appetite. Yeah, uh, right. So, uh, for— I think obviously in the enterprise we're past that conversation for the most part, right? I think most large enterprises that are moving fast, you know, they've got it, you know, somewhat figured out when it comes to, you know, finance and AI. But I think a lot of SMBs are still maybe struggling with that risk appetite, right? How much, you know, should we be pushing AI, you know, when maybe they don't have a team of experienced, you know, machine learning engineers that can come in and, you know, help set this up. So with that, for maybe those, you know, small, you know, maybe those small and medium-sized businesses, when they're looking at something like Sage Copilot and maybe they're still doing it the manual way, Right. And they're saying it might be too risky. How do you address that?

No, absolutely. I think it, um, I approach this, this whole decision space very much with a mind towards like, use the right tool for the right job. Mm-hmm. Um, you know, um, you can, you can maybe just use ChatGPT directly for some functions, you know, um, ideation, creativity, um, marketing copy, a first draft, not the final draft. Um, that, that's fine. But when it comes to finance, and you really do need to, to have a higher standard, and I think people are right to, to, to look at that risk and be like, wait a second, do I, do I want the robot anywhere near my general ledger? You know, and, and again, my answer to that is just that it's not an all or nothing type of thing. I always suggest that, that people look for individual use cases, subfunctions where AI can bring bring whatever amount of value you're comfortable introducing today. And yet more than 10 years ago, the first AI that we built at Sage was, was outlier detection. It, it just combs over every general ledger entry and says when something is anomalous, when something got misentered, or when fraud can pop up. Mm-hmm. That should not be controversial. It's not changing anything. It's not posting transactions. It is simply there as a watchdog, as a stopgap, and that's still running today. And now, like, we can do a lot more, obviously, with agentic AI. But even within that system, our outlier detection on the GL is one of the core tools behind our MCP layer, behind our finance intelligence agent that makes it work. That's what grounding means in, in the real world. Yeah. You know, we are grounding our agentic answers with real anomalies detected with good old machine learning.

Yeah. And you know, you, you said something important there that I wanna point out, you know, doing. The best with today's agentic AI. But, you know, obviously you can't be looking at the past if you want to be a future-forward, a future-facing organization that's keeping up with, you know, bleeding-edge AI. So, you know, obviously you can't, you know, lay out the playbook for next year's Sage future, right? But what should business leaders, you know, on the finance side and otherwise, be doing today, specifically when it comes to their books, their processes, so they can be ready for whatever comes next?

Yeah, I mean, if you live on the bleeding edge, you get cut sometimes. It's, um, I'm there with technology and, you know, our, our customers are there with their processes and, and with, you know, like all of the changes in this space. Uh, so my, my best advice is, um, you know, hold onto it loosely. I think there are some problems that we can solve and AI is not one of them. AI is bigger than that. It's bigger than you, it's bigger than me. And we can orient ourselves around that change and be ready for that change. So I think my best advice is start using AI that you're comfortable with. Keep an eye on all the AI as best you can, you know, through Everyday AI and other resources, right? So that you're aware of what's going on in the ecosystem. And then when you do see something that's a good fit, You know, when finance intelligence agent can help you close your books, you know, 10 days faster than you ever could before, which it can, you know, jump on that, right? Be ready to move in any direction, but only adopt AI that has real value for your business.

All right. So, Jeremiah, we've covered a lot in today's conversation from what's new with Sage Copilot, what's being announced here. We've talked about risks and guardrails and everything in between. But as we wrap, what is your one most important piece of advice, right? Getting back to trust and good enough, maybe not being good enough. What is the big takeaway to make sure that people can have AI that they can feel confident in and they can trust those outputs?

Yeah. Oh, I just want to share that people shouldn't give up on this. I think there's a lot of skepticism around AI that's arisen from, you know, overinflated promises. But I think, You know, all the AI we build at Sage is built with grounding and with access to real answers and real data. So if you've seen it, you know, some, something else hallucinate, that's not good enough. You, you can, you can maintain a really high bar there. Um, there is AI that you can trust out there in the world. It's being built by Sage. And, and what does that really mean? It means that you're confident in its answers. As a human, you control what's going on and that AI is accountable and auditable. For, for what it's done inside your finance system.

All right. What an amazing conversation. So we covered a lot in today's show, but there's going to be a lot more. So make sure to check out today's daily newsletter for everything else that happened at Sage. Jeremiah, thank you so much for taking time to join the Everyday AI Show. We really appreciate it.

Thank you, Jordan.

All right. So if you miss anything, like I said, make sure you go to youreverydayai.com. We're going to be recapping everything that happened at Sage Future and a lot more. Thanks for tuning in. Hope to see you back tomorrow and every day. For more everyday AI. Thanks, y'all.

Every AI vendor in your inbox is promising to revolutionize finance. Most of them have never set foot inside of a controller's office. Sage has. At Sage Future, they double down on AI you can trust for finance. AI that gives your team confidence with results they can explain and verify. Control so people stay in charge of the decisions that matter, and accountability so every action can be traced, plus a faster way off legacy desktop tools and into the cloud alongside AWS. That's how you actually adopt AI in mission-critical work. Check out sage.com for more.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going! For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers, and we'll see you next time!