Unlock Financial Success with AI With Jon Brewton

Welcome to the Future Finance show where we talk about why Android analysts dream.

Of electronic spreadsheets.

When implementing AI decision making process. Why is that traceability and explainability so.

Important if we are going to be able to scale generative AI into different places and let's just say it doesn't have to be. I think traceability and explainability are a big part of that. These features are crucial for maintaining our understanding of these AI systems. Our system review actually starts and ends with Knowledge Graph data. 99.9%. From an engineering guild chain or finance perspective, maybe 80% is good enough. Gets you from a starting point anywhere from 60 to 65%. And if you continue to contextualize those models, you can get quite closer to.

How you're using some hallucination resistant agents, you're using some of those with your platform.

Hallucinations are actually quantified in a couple of different ways. It's like how contextually relevant is the answer that was provided. The non factual basis of all of those in native performance is about 73%. Don't think you can get there without really.

Future Finance is brought to you by qflow AI, the strategic finance platform solving the toughest part of Planning and analysis. B2B Revenue align Cells Marketing and finance seamlessly speed up decision making and lock in accountability with qflow AI Foreign welcome to Future Finance. This week we cover three topics during our sections. First, Paul in the monologue talks about spreadsheet LLMs and the exciting announcement by Microsoft. Next, Glenn Hopper tells us why our job is safe for now because we're only on level one of the five levels to reach artificial general intelligence according to OpenAI. Then finally we welcome our guest John Bruton, who owns Data Squared and is an expert in machine learning and AI solutions. Join us for this exciting episode. It was recently announced that Microsoft is working on its own spreadsheet LLM. The reason for this is spreadsheets are hard for traditional LLMs to interpret and and over time this could be a game changer for what AI is able to do in spreadsheets. Some of the challenges right now for LLMs and spreadsheets is there's a lot of characters that can often exceed the characters they can digest at one time. Spreadsheets are two dimensional and usually LLMs use a linear and sequential input approach and they're not usually trained to interpret cell addresses and specific spreadsheet formulas. All of this creates challenges for LLMs. So Microsoft Research has used a multiple step technique to parse the spreadsheets they've used a compressor. In some cases they've been able to reduce the number of tokens that requires to analyze the data by 24 times. The methodology allows them to have the LLM look at the compressed spreadsheet they use JSON and combining, taking that data once they've compressed it so that they can feed it into the LLM. So what does this really mean? What's this all about? Well, what it means is we're making progress, we're not there yet.

They continuously learn and adapt, refining their strategies over time. So these are self improving AI. And the significance of these autonomous AI agents is it's going to be in their ability to automate more complex tasks. You know, right now we have to give very specific step by step instructions. But with an agent, you'll tell it, I want you to do this, it will go off and do it and then come back to you when it's complete. And think about these agents, the power of them. They're operating continuously without breaks and you know, much more efficiently than humans are. We're going to look at reduced labor costs, minimize human error. And the idea here in the perfect scenario is by having these agents do this, human workers are able to focus on the more creative and strategic and complex tasks. And you know, this has been the holy grail for the short term is to come up with these autonomous agents and it'll certainly revolutionize many industries and professions. But we're not quite there yet. The role of human decision makers, creative thinkers, and those in positions requiring a personal touch that remains crucial in everything we do. So your skills in these areas are still very much needed and valued. But now what happens when we get to level four? OpenAI calls level four the Innovators, where these software programs can transcend task completion and drive their own innovation independently. These systems will generate novel solutions, conduct research, create original works. And their creative thinking and advanced learning algorithms will make them powerful contributors to fields like I don't know anything, science, product development, art, really any of the jobs that we do today. And they'll on their own, these systems will hypothesize, experiment, analyze results. And you can see how this will further accelerate innovation and integrate knowledge across disciplines from everything it's looking at. And this stage will be huge. But I think we're still a long way from machines independently driving innovations across the board. You know, level five, this, this is the end game for OpenAI. And it sounds very much like science fiction because as of today it is science fiction. But OpenAI calls level five organizational equivalence, where AI systems are functioning as entire organizations. With strategic thinking and adaptability. With strategic thinking and adaptability. Imagine an AI running a whole company. This level envisions AI systems managing more complex systems and achieving organizational goals, independently transforming industries and the global economy. I mean, this is, you know, this is true sci fi stuff at this point. These systems could understand and navigate complex business environments, evaluate competitive landscapes, identify opportunities for growth and innovation.

At level five, AI would be able.

To efficiently handle day to day operations of a business, managing resources, optimizing workflows, ensuring seamless coordination across departments. You know, looking at supply chains, finance, hr, customer relations, and a computer system that's able to link all these together. So I don't, when we get to this point, what does this mean for.

You and your job and me and my job? I mean, I think the key takeaway.

Right now is while AI is advancing rapidly, we're still at the very beginning of this journey.

The leap from where we are now.

To true AGI is enormous and it's going to take some time, likely many years, if not decades. I think people, you know, it's always dangerous forecasting the future. But I do know from forecasts I've made, it does seem that technology is coming faster and faster. And I've never seen anything as broadly adopted or advanced as rapidly as AI has since the transformer, since generative AI became mainstream. So, you know, I think we know that it's coming, we sort of plan for it. And I think that for the short term though, we look at what are the skills that make us uniquely human. Our creativity, our emotional intelligence, our complex decision making and adaptability. These are what's going to be in demand now and for the foreseeable future. And I think we look at AI as a tool. I mean, it's a powerful tool, but it's not replacing the human workforce at a wholesale level anytime soon. That said, it's crucial to stay informed and be adaptable. The job market is going to evolve just like it has with previous technologies. And as AI capabilities grow, we're going to really refine what systems can do and what we as humans can do.

And I think if we look back.

At every prior technological revolution, new jobs and opportunities are created too. So we hope and we think that that's what's going to happen here. So while OpenAI and others are making strides towards AGI, we're still in the early stages and your job is safe for now. But it's always smart to keep learning, growing and adapting to the Changing landscape.

Welcome to Future Finance. Our guest today is John Bruton. John is the founder and CEO of Data Squared, a leader in machine learning and AI solutions. With a background spanning operations management, digital transformation and engineering. John has held key strategy and technology development roles at global energy giants BP and Chevron. His work has led to significant operational efficiencies, including $500 million in savings at Chevron's Southeast Asia division. John holds degrees in operations management, industrial and systems engineering and an MBA from NYU Stern and the LSE with executive education from HEC Paris, Harvard Business Analytics Program, MIT's CSAIL and Stanford SAIL. A U.S. air Force veteran, he combines military precision with tech expertise. At Data Squared, John is pioneering AI solutions emphasizing traceability and explainability which are crucial in today's data driven financial landscape. John, welcome to the show.

Glenn, thank you so much. Appreciate the opportunity. Look forward to getting into a discussion today about how the things that we're doing apply to the sectors of finance and a couple of other interesting places as well that we're playing in at the moment.

Yeah, that's great. And one of the things I love about this show is I get to talk to people who are even dorkier. Is that the right word? Nerdier? Geekier? I don't know than I am. But looking at your, your, your background, I think that's a, probably a moniker that you carry with pride.

So no doubt about it, I guess.

To start, I think all three of us do. That's what makes it great.

It's nerdfest. So John, I guess to get started, tell me a little bit about Data Squared, what you guys are doing and sort of your journey, your path to founding this company.

Yeah, thanks. Look, I think Data Squared story is interesting. I find it compelling and interesting because you know, well, we're a part of it, but in general, look, we're a service disabled veteran owned small business in the government sector specializing in gen AI product development and deployment. Additionally, we have some interface in the commercial markets, really paying attention to what's happening in the energy space. Our company started in, let's say early 2023, but really didn't get kicked off in the way that we're positioned now and the things that we're working on now from a product development perspective until August of 2023. So we're really young, we're working in a couple of different places. We have active work going on in the United States and we also have active work going on in the energy sector, specifically building oil and gas solutions that are heavily reliable, traceable and explainable. Just a little bit about the company in the background. We've started with four people. There's three common traits in the founder pool. We either served in the military. I'm an Air Force vet, like you mentioned earlier. Our co founder, Eric Costantini is a Marine. And our other co founder, Chris Rohrbach is a retired Navy SEAL commander. He did 26 years as a Navy SEAL commander, SEAL Team 2610 and a variety of other positions. And Jeff Dalglish, our Chief Technology Officer. We worked together at Chevron for several years, specifically in digital solution development and deployment at scale within the company. So our three common traits are we either studied together, Eric and I at Harvard, Chris and I at NYU and the London School Economics. And Jeff and I, we work together. Eric also has a mining background, at least in the energy sense and we served in some capacity. So those are really the three common traits and the things that really shape our perspectives and the things that we focus on on a day to day basis, primarily energy and defense. But what we started with at Data Squared gave us an opportunity to really start to look at how we could design something that's industry agnostic. And the thing that we built to this point, which we'll get into a little bit more later, is exactly that. I think it's particularly and uniquely positioned to exceed and exceed exceptionally well in places like high reliability sectors, specifically places like engineering, defense, healthcare and finance. These are places where the stakes of these environments are exceptionally high and rapid ability to analyze information can really be the difference between success and failure.

Appreciate that overview. I'm feeling a little left out in this group. Everybody but me is a military veteran. I will say though, I did work on a Navy base for four years and I was employed by the government, but I wasn't enlisted, so that's as close as I can get.

Yeah, no, it works.

I like it. You talked about traceability and explainability. I'd love to kind of dig into that a little bit around AI. How crucial those features for financial institutions when implementing AI, you know, making decisions, decision making process. Why is that traceability and explainability so important?

Well, for us I think it's important for a multitude of reasons, but it's really informed and directed by the industries that we're targeting to play in most. And it aligns with our subject matter, expertise, our background and the things that we're passionate about and we understand well. But these are industries that have a preoccupation with failure, a reluctance to simplify they're sensitive to operational changes and environmental changes. They have a commitment to resilience. Think of the nuclear Navy. That'll ring true to Glenn. And a deference to expertise. You know, these are things that are cultivated traits of the environments that we need to play in. So if those are the minimum hurdles that we have to actually achieve to sort of operate in these spaces and operate with scale, well, then traceability and explainability are absolutely crucial for understanding how to implement these tools and getting and generating value out of these tools in a scalable fashion. I don't see how that's much different for financial institutions or financial markets. Whenever you start looking at these things, every decision that can be made by AI should probably be understood fundamentally. So it needs to be traceable, needs to be verified. So it needs to be trustworthy and explainable. And these are really essential for maintaining trust and I think in general compliance in these highly regulated, high reliability environments. So those are really important things that I think are absolutely crucial to the environments that we want to play in and the environments that you obviously thrive in. When I think about how it applies to financial markets, I think about it in a context of regulatory compliance. We're trading, we're selling, we're acting on some of these things within these markets. There are a lot of regulations that we have to follow, we have to adhere to, and we have to manage outcomes to. As a byproduct of that. I would expect that traceability, transparency and explainability are almost imperatives for how you would deploy these things and deploy them at scale. I look at risk management by ensuring that we have traceable decision making processes and explainable outcomes, financial institutions better understand and mitigate risk. I think that really helps, I think in general, from an explainability perspective, identify potential biases in the system and understanding how things could potentially be overfit and lead you down paths that aren't necessarily the right paths to go down at any point in time. So you need to have the ability to dissect this information in a way that creates an opportunity to understand it at its core fundamental level and why these things were generated. I think that's a bit of something that we miss. The other areas that I think are really important just from a financial perspective are stakeholder trust and fidelity in decision making. You know, we operate primarily in the engineering construct and in the government construct, and there's a couple of key environments. You know, if you're going to enter into the kill chain, not to get morbid at all, you need to ensure that you have a very clear understanding of how these decisions, or at least these suggestions were provided to you. What's the underlying context? How valuable or how connected is that context? How can we quantify the impact of the decisions that we may make in any collateral damage? Same thing for contingency modeling in big large scale oil and gas operations. If you think about some of the projects that I manage for Chevron, some of these are up to $2 billion, they spanned five countries and had staffs of over 500 people. And if any one piece of this puzzle was designed incorrectly, well, like it interjects a level of failure that is almost intolerable. And so these environments really thrive on that. And so financial markets I don't see as being any different at all. As a matter of fact, I put them on the same level with the environments that I spoke about before. And health care is another area where we are going to deploy these tools. We need to ensure that they're explainable and that's a really hard nut to crack within the market as we sit right now. These tools, commercially, as they're available, have a lot of interjected bias because of the way they work. They're statistic models. So it's like what is more likely than not to be the answer that you're searching for? And in our spaces I'm not exactly sure that that really rises to the level that it needs to.

So you're saying your level of confidence in a kill chain is much, much higher than me, who's writing a post on LinkedIn type of thing. Right. The statistical factor of there being an error has to be magnitudes higher.

Agreed? Yeah, I think so.

And that's, you know, when you're talking about AI and machine learning in data, you know, everybody right now wants to talk about generative AI and that's the latest buzzword and it's the most approachable and accessible. But the funny thing is most people who are using that don't understand what it is or really how it, how the generative AI is giving them the result. But when in what you're doing, and certainly for military applications and for applications that are of great consequence, it's important that you have that explainability and understandability of the models. And I think that's something that we really need to think of. And I think that for these kind of Johnny come lately, people who just, they haven't used AI in any capacity until LLMs made it accessible to them, we're talking about completely different ball games here. And but that said I mean, there, there is an application for generative AI and I'm trying to find ways. And that's one of the things that I love about this show is having people who are at the leading edge with integration of AI into workflows and how we can apply it. And one thing I love about the show is having guests on like you where I can ask really geeky questions and get a good answer on it. So I'm thinking about generative AI now and I'm really, I'm pushing, you know, I do R and D for products in the, the finance marketplace and we have to overcome a lot of the issues that you're addressing as well. And one of the workflows I'm looking at, I've tried, you know, overcoming hallucinations with rag and that has limited success. And I don't know if you guys are working with this now or not, but one of the areas I've been looking into lately is graph databases, knowledge bases with rag. I know it's not necessarily something you're doing today, but I'm wondering how could you potentially see knowledge graph technology being applied to financial data, you know, to help us uncover hidden correlations or insights that traditional analytics might miss?

Yeah, it is something we're doing today. I'll explain the nuance of what we're doing. So our system review actually starts and ends with knowledge graph data models. So it is a fundamental part of our process, our workflow and how we actually store in GI structure information. For me personally, I believe that you have a couple of different variations. And like you said, I think traditional RAG methods have a good efficacy rate and good as being generous. Like the average returns range anywhere from 30 to 60% on efficacy in terms of how structurally fit the answers are to the things that you're asking. What's being interjected from models. When you use knowledge graph data models, you almost create a roadmap for how you can structurally look at the connections and information and the strength of those connections. And so I personally believe that if we are going to be able to scale generative AI into different places and let's just say it doesn't have to be 99.9%. Right. From an engineering guilt chain or finance perspective, maybe 80% is good enough. Well, graph rack can kind of get you there. So what we're doing is essentially using that same fundamental structure and the fundamentals of how you do that, but we're doing it on steroids and growth hormone at the same time. Think Barry Bonds 73, helmer season. That's effectively what we're doing. So we're using the same fundamentals in context for know process. How you get there. Graph Rag gets you from a starting point anywhere from 60 to 65%. And if you continue to contextualize those models, you can get closer to the 80% mark. So you can get a lot of fidelity out of the models just right away because you can see the inherent connections between node 1, node 2, and the strength of those relationships. What we're doing is essentially ratcheting up how we do that. Same principles, just ratcheting up. So we add these hyper semantically embedded layers of information and we also test information against every other piece of information in the data model. So essentially we prioritize data ingestion, compression, structure and testing on the initiation of what we do with the client. So we'll take their information, we'll do that, and we have a proprietary process for how we do that. And then we start to look at enrichment. It's like, what do we understand about the domain problem we're trying to solve? What are the typical questions we'd want to ask? And how can we understand the connections of this data model in the context of those questions? That's simplifying things. But I think the other big difference in what we're doing is usually when you look at Graph Rag, especially from a research perspective, people are doing that on unstructured data. So people are really having a conversation about words. And these models are great with words. I mean fundamentally that's exactly what they're built to do. But when you start looking at structured data or high frequency data, or even semi structured data, the performance of these models starts to fall apart. Our system effectively cuts through that noise. So we ground everything together fundamentally, we test everything so that we can connect all of these different data types together in really, really tangible ways. Now again, if we go back to the database model itself and understand how you can apply some of these concepts from a Graph Rag perspective, well, now we can start to do things that we weren't able to do. We can ground that model, we can effectively constrain that model, but we can hyper contextualize that model in a way where your returns from a system perspective are far in a way better than anything that you can get from just traditional rag approaches or even graph rag approaches. And so doing something that's slightly different, and I think it really is that toggle between different data types being ingested all at once, combined all at once, and then contextualized together. That gives you this level of fidelity that's almost uncomparable in the market right now. So Graph rag. Yes, definitely a good concept to apply to what you're doing in financial markets for sure. Go ahead with a follow up.

Yeah, I was going to say that to me it's it. It is trying to find those correlations between the. The unstructured data and the in the sort of word based and just how. How can I add value to. Historically our reporting and finance has been limited to this is what's in the gl. Maybe we have some exogenous factors, some. Some external data that we're going to bring in that we can find correlations to. But if you can find customer trouble tickets that are related to a certain thing, just pulling information unstructured from your other systems, I think it's a way to really expand what we can do in FPA with our data being able to link it to other information sources.

Ever feel like your go to market teams and finance speak different languages? This misalignment is a breeding ground for failure, impairing the predictive power of forecasts and and delaying decisions that drive efficient growth. It's not for lack of trying, but getting all the data in one place doesn't mean you've gotten everyone on the same page. Meet qflow AI the strategic finance platform purpose built to solve the toughest part of planning an analysis. B2B Revenue QFLO quickly integrates key data from your go to market stack and accounting platform, then handles all the data prep and normalization under the hood. It automatically assembles your go to market stats, makes segmented scenario planning a breeze and closes the planning loop. Create airtight alignment, improve decision latency and ensure accountability across the teams.

So on that point, so we've done a bit of financial modeling in the intelligence construct. And one of the things that I think is really interesting is whenever you tie financial data to other data, you know that could be human reporting data, it can be public data, it could be 10Ks, you can analyze 10Ks. I know that you've done some work in this space to look at how you can do that and how you can do it with a great deal of efficacy. Sentiment analysis is another thing you can look at and it's being able to visualize and understand the intricate web of these connections and how they in combination with one another, not at the same weight or in the same sort of factors, but together actually contextualize and drive market behavior. That's a really interesting thing to start looking at. I think the other thing that really that you get from just using knowledge graphs is the ability to uncover hidden connections, which can help from a pattern analysis perspective in fraud detection. And you can build these sort of network models which give you an understanding of all the inputs, the outputs, the connections between these models and the strength of those connections. And if you can do that, you can model those connections in a way that is really tangible and significant. I mean, I can't undersell this at all. That data model enables you to combine information that is just different. And by that combination you can really see like trends, patterns, you can analyze connections, you can really start to dive into proactive risk management, contingency planning, and get a real comprehensive view of how market conditions influence one another. You can also do this in preventative maintenance. If you're looking at say like grid control and you want to understand all the external factors that actually have an impact on how grids are managed and how energy is produced and how it's stored and how you can disperse it and what that means for like your total grid management, your production and everything else. The weather has an influence on this. So there's really no limit to the types of things that you can combine. But I just think that from a finance perspective that transparency, well, the data model enables a level of transparency that is crucial for audits, regulatory reviews and making accurate and informed decisions, especially when you're talking about prediction. Got it.

Thank you for that answer. That was very helpful. I want to ask question more around and this gets back to I'd say generative AI, but AI in general. Right. Hallucinations particular with Jenna Tren AI. And I see you smiling. I've got a big concern, I think a couple dictionaries. Last year, the word of the year was hallucinations. Somebody, somebody who didn't know anything about it as like what drugs? No, no, no, that's not, not what they mean. So obviously for finance that's a huge concern, especially if you're using it for anything that goes public. Releasing 10K, you know, releasing information to the street. So I thought if you talk a little bit about how you're using some hallucination resistant agents, heard you're using some of those with your platform and how you think we'll see that technology applied in finance, you know, fraud, anomaly detections with large data set.

Yeah, I think just to ground the discussion, it's really important to understand how these things are actually produced. I think whenever we talk about hallucinations publicly, we talk about it as is it Factual or is it not? And is it interjecting things that are made up or fabricated? The interesting thing about it is like how these models work. And I know that you guys are going to know this, but maybe the people that are listening don't quite understand this as well. Hallucinations are actually quantified in a couple of different ways. It's like how contextually relevant is the answer that was provided, what's the token probability that led to the answer that was created? And how much entropy is in the model uncertainty. Now you test these things on a variety of manners. Like you can effectively, there's a bunch of different question types. You can test things and say like hey, here's a simple question, you know, what is the birth date of a person within your data model? And you can recursively run that question a hundred times through the same model and then look at the differentiation and the answer and test the certainty or the efficacy of the returns of that. That gives you an entropy score and the overall token probability. So that's how hallucinations are really calculated in general. People think about it, is it fact based or not? And you have to really start with like, how are commercial models performing today? There's been a lot of great research in this lately and one of the better papers which I can send you effectively dives into, okay, let's look at every commercial model and let's look at how they're performing in the fact basis of their native performance. You know, given a certain number of factors. And one of these studies looked at, one of these studies looked at, okay, let's look at all the different models, llama models, opt GPTJ, OPO, GPT4O, like all the anthropic models, Gemini models, everything. And let's just run some testing on it. The non factual basis of all of those and native performance is about 73%. So what about if we just take an individual document and look at it, okay, it's an individual document we tested or sorry, this research paper tested 156 documents just doing traditional rag on it and looking at the answers that were provided and only 14.7% of the returns had a factuality score better than 80%. So these models in native performance are actually pretty terrible and they do interject a lot of nonsense. And the question is like, is the nonsense contextually relevant to the thing that I'm doing? Who knows? You have to really dive into the results on a one off basis to look at. But effectively we set up a testing protocol and we're testing all of these models recursively in the background against the things that we're looking at and trying to ground them. We have a patent pending approach to how we're doing that, but it effectively sets up hallucination resistant agents that can be deployed. Now a lot of that is actually helped by the database model and the structure of it itself because we're effectively paving a roadmap for how we can deploy these things and really defining the connections between point A, point B and the strength of these connections. And overall that cuts down on your hallucinated basis. But for us it is a thing that needs to be managed. We've been lucky because we were able to test things and I did some LinkedIn posts on this recently. Regardless of the model that we're applying, whether it's Lana, GPT4, Claude, Opus3 or Sonnet or the question type that we're asking, when we engage the system through our process, we can get 99.9-ish% results. You never want to say 100% because you know, these systems do work on a statistical basis still, but we do a good job in grounding them. So how does that apply to financial markets? For me, I think if we're going to do stuff like fraud analysis, we really need to be able to continuously monitor data sets and deviations from expected behavior. And if these models are interjecting false or inaccurate information, well, it really waters down the value proposition for something like that. Financial irregularities can be timestamped, fingerprinted and understood. And you can really trace back the leading factors for how you got from a point that preceded these things to the eventual outcome. And these systems are really good at understanding that stuff when given the right structure and the right grounding. And I think that's really important. The other thing that I think is just sort of looking at early detection for preventing financial losses, like how well can we predict market changes and the impacts they're going to have on the things that we're trying to do. The other big thing there from an hallucinations perspective, regulatory compliance is obviously a piece of that puzzle. But integrating some of these solutions within secure frameworks and systems is I believe, another area where if we combine sort of the right structure of information, we can get to a point where we have a lot of confidence in these systems and we have confidence in the financial decisions that we're going to make as a byproduct of them. I just think enhanced accuracy is a big part of it, especially when you're dealing with financials And I don't think you can get there without really ensuring that you have a low hallucinated base of returns. So I know it's a market problem. One is people don't understand it very well. But when we talk about factual basis in the work that you guys are doing and the work that the people that are going to be listening to this podcast are doing, I see it as a crucial imperative for scale.

Yeah, great, great points. And I. It's funny, I feel like with my first question as soon as we had you on, and it's because when we get someone with your background and what you're focused on, we like to dive deep. But we just jumped off the high dive straight into the deep end and got down into the weeds of this. And I think, you know, for a lot of our listeners, they are, this is the place they're working in. And this is, you know, we're, we're singing the song of their people. But for a lot of other people who are, you know, or FP&A and looking to expand and understand this, you know, we did go right into the deep end. So I think I'd like to maybe bring this home at the end with now that we've thrown everybody in the deep end and made them swim on their own. If I'm running a finance department and I'm looking to integrate advanced analytics and AI gen AI or not, you know, if we've known that this is coming and we're not already doing work with advanced analytics and using our data and incorporating AI into our financial processes, what advice would you have for someone in this position to start integrating and start moving in this direction where they can be talking meaningfully about graph rag and all the other stuff that we've talked about in the show.

Yeah, for me, it's a great question because I think it's a scalable question and answer. You know, whether you're dealing in finance or healthcare or engineering, you know, there's, there's a certain level of efficacy that you're going to chase to make sure that you have trust in what you're doing. But I think it goes back to, let's just really look at how we integrate advanced analytics and AI solutions in your business. And it starts with thinking about it strategically and understanding methodically how you do these things. You need to define your specific objectives and challenges clearly. I think there's not lip service, like if you're going to undertake deploying one of these solutions or building one of these solutions for deployment, whether it's improving risk management, enhancing fraud detection or optimizing investment strategies. Having clear objectives will guide the AI integration, development and all the things that you need to do to make sure it aligns with your business goals. I think traceability and explainability are a big part of that. These features are crucial for maintaining our understanding of these AI systems, ultimately what they're being fed and what's coming out of them. But on a more fundamental level, how that stuff's connected and the value of those connections really gives you an inference in how you can do some predictive modeling around these things. For me, I think that builds stakeholder trust and ensures that you're compliant with the things that you have to be compliant with. The other thing, and I know this is where people are going to say like, ah, it's just generic, but collaborate with experts. You know, if I'm going to take on a finance project with somebody, I'm going to call you guys just to talk about it because you have a different perspective than I do. Yeah, perfect. No problem. Whoever I need to get to first, you know, I'm going to talk with experts because one of the key principles I sort of live my life by, and I think it was informed by the military and all the stuff I did in oil and gas is like, perspective cannot be given, it has to be earned. And you can try to learn something, but if you just collaborate with experts, they have that perspective and you can really. We can get 99.9-ish% results. You never want to say 100% that. Okay, let's look at all the different models. Llama models, Opt GPT, J, OPO, GPT4O, like all the anthropic models, Gemini models, everything. And let's just run some testing on it. The non factual basis of sort of the right structure of information, we can get to a point where we have a lot of confidence in these systems and we have confidence in the financial decisions that we're going to make as a byproduct of them. I just think enhanced accuracy is a big part of it, especially when you're dealing with financials. And I don't think you can get there without really ensuring that you have a low hallucinated base of returns. So I know it's a market problem. One is people don't understand it very well. But when we talk about factual basis in the work that you guys are doing and the work that the people that are going to be listening to this podcast are doing, I see it.

As a crucial imperative for scale yeah, great, great points.

And it's funny, I feel like with my first question, as soon as we had you on, and it's because when we get someone with your background and what you're focused on, we like to dive deep, but we just jumped off the high dive straight into the deep end and got down into the weeds of this. And I think, you know, for a lot of our listeners, they are, this is the place they're working in and this is, you know, we're. We're singing the song of their people. But for a lot of other people who are, you know, or FP&A and looking to expand and understand this, you know, we did go right into the deep end. So I think I'd like to maybe bring this home at the end with, now that we've thrown everybody in the deep end and made them swim on their own. If I'm running a finance department and I'm looking to integrate advanced analytics and AI generative or not, you know, if we've known that this is coming and we're not already doing work with advanced analytics and using our data and incorporating AI into our financial processes, what advice would you have for someone in this position to start integrating and start moving in this direction where they can be talking meaningfully about graph rag and all the other stuff that we've talked about in the show?

Yeah, for me, it's a great question because I think it's a scalable question and answer. Whether you're dealing in finance or healthcare or engineering, there's a certain level of efficacy that you're going to chase to make sure that you have trust in what you're doing. But I think it goes back to, let's just really look at how we integrate advanced analytics and AI solutions in your business. And it starts with thinking about it strategically and understanding methodically how you do these things. You need to define your specific objectives and challenges clearly. I think there's not lip service like if you're going to undertake deploying one of these solutions or building one of these solutions for deployment, whether it's improving risk management, enhancing fraud detection, or optimizing investment strategies, having clear objectives will guide the AI integration, development and all the things that you need to do to make sure it aligns with your business goals. I think traceability and explainability are a big part of that. These features are crucial for maintaining our understanding of these AI systems. You know, ultimately what they're being fed and what's coming out of them. But on a more fundamental level, how that stuff's connected. And the value of those connections really gives you an inference in how you can do some predictive modeling around these things. Things for me, I think that builds stakeholder trust and ensures that you're compliant with the things that you have to be compliant with. The other thing, and I know this is where people are going to say, like, ah, it's just generic, but collaborate with experts. You know, if I'm going to take on a finance project with somebody, I'm going to call you guys just to talk about it. Because you have a different perspective than I do. Yeah, perfect. No problem. Whoever I need to get to first, you know, I'm going to talk with experts because one of the key principles I sort of lived my life by, and I think it was informed by the military and all the stuff I did in oil and gas, is like, perspective cannot be given, it has to be earned. And you can try to learn something. But if you just collaborate with experts, they have that perspective and you can really leverage that perspective to ground your understanding of what you're trying to do, the opportunities that are in front of you, and some of the things that you really need to think about. I think leveraging strategic partnerships with people that know this stuff well is also important if you have the ability to do it. Like we're leveraged up right now from a partnership perspective with Nvidia, Microsoft, AWS and Neo4J. And that really covers our CSPs, our compute capacity and our database model. I think you do need to focus on security, continuous learning and adaptation once you actually start to deploy these things. But for me, I think those are kind of the first principles I'd gather around. There's a lot of different things that you need to think about, including the efficiency gains that you're chasing and what it means for how you operate your system and your personnel. You really need to understand that stuff because if you deploy a system that takes 75% of the workload off the people that are doing the work today. A financial analyst is a really good person to think about in this context. But what are you doing with those people? How are you repurposing? How are you actually leveraging their talents and their skills in different ways now that you have excess capacity? So I think, you know, think about it from a first principle's perspective. If we're going to actually build and develop a new program, we're going to develop a new solution and deploy it. What do I need to think about first? What problem am I trying to solve? What Value am I trying to create? Who do I need to talk to that's going to give me the best answers and how does that inform how to develop and deploy the solution?

Now we're going to get to our personal section. So how this works is each week we come up with a different list. We've used different tools to come up with a list of 25 questions. This week it was done by Claude Sonnet. So we use them to generate questions. They have 25. Going to give you two options. One, you can pick a number between one and 25 and that's the question I ask you. Or two, I can use the random number generator and see what it comes up with.

I like random numbers. Let's go.

All right, give me one second here and we'll see what it gives us. Number 18. So let's go to number 18. Number 18. What's the most interesting or unusual job you have had before your current job, before what you're currently doing?

I'd say being honor guardsman in the United States military was probably the most interesting and gratifying job I've ever had. I wouldn't say that it was particularly comforting on a day to day basis. I think I presided over 47 military funerals whenever I did that. And yeah, those brothers in arms. But that was definitely the most interesting and rewarding job that, that I ever had.

Got it. Now I can see where that could be very rewarding but also very emotional. Presiding over some of the things that you did obviously can be difficult. So thank you for sharing that one over to you, Glenn.

All right, let's go. I'm, I'm gonna let Claude pick from these and we're just going to completely turn it over to the AI here.

So he's, he's more daring than I am. I'm not quite ready to give it all.

No problem.

Oh, this is so. This is a great one. It's actually, I feel like it may, this may be actually a good follow up to what your most rewarding job was. What's a charity or cause that you're passionate about?

We as a family really support the Ronald McDonald House and have for years. Both of my kids when they were young had medical issues and they were in the hospital, serious medical issues and that stuff. That organization is just like amazing. They do a lot to help families and anything we can do to help them, really, really meaningful. Sorry.

Yeah, no, that's.

Yeah, yeah, that I could understand being unemotional on that one. I used to do in grad school we go every month and prepare dinner for Ronald McDonald's, and I love doing that. You at the end, we'd be like, it was great seeing you. I hope not to see you next month.

Yeah, yeah.

Like, I like it. But we want your kids to be home with you. So I could imagine that's an emotional. Emotional memories there.

Yeah, for sure. But amazing organization. Like, they do incredible work for families. Really. That's a big part of at least our experience. And anything we can do to help them, we're for it.

Well, John, we really, really appreciate. I know it's. We went so deep, so fast on this, but it's just, I. I love having you on and your insights and everything you were able to provide and then, you know, learning a little bit more about you and. And what you're doing at Data Squared and on the personal side and just thank you so much for coming on the show.

Thank you, John.

Yeah. Thank you. Really appreciate the opportunity. Guys. Have a great day.

Thanks for listening to the future finance show and thanks to our sponsor, qflow.AI if you enjoyed this episode, please leave a rating and review on your podcast platform of choice. And may your robot overlords be with you.