Token Costs, Tokenmaxxing and managing engineers

Welcome to another Founder Wisdom podcast. We have, uh, Greg Wallen with us today. Greg's CTO at Proove.ai. Uh, he's in product technology, business, uh, line leader, uh, side of things, and all about AI, Gen AI, fintech in the Greater Boston area. Uh, Greg worked at a bunch of cool companies before Proove.ai, including AWS, Experian. He has a PhD in computer science and biomedical informatics, which is fascinating to me. So Greg, welcome to the podcast. Tell us a bit more about yourself and what you're up to.

Sure, sure. Thanks to— thanks for having me. Uh, yeah, so what we're up to at Proove AI, um, so in the, you know, during the course of trying to do development with, you know, generative AI tooling, meaning Claude Code, uh, you know, anything of the like, Cursor, uh, you know, what we've noticed and what we've seen is just how much pain these new breed of developers is experiencing as they're trying to productionalize and add to the stuff that they've created with Cloud Code. There's lots of things out there and lots of emerging mechanisms or procedures on how to make all of this stuff work, like spec-driven development, prototype-driven development, lots of discussion in the space. You know, what we're really interested in at Proove.ai is to say, okay, you know, if development has changed, the software development has changed and it's yet another big, you know, a big change, which is nothing new, right? That's happened several times during the course of my career. It's all cool, you know, then how do the processes and toolings change to support moving at speed without breaking things? Right. How do we, how do we, you know, productionalize and ultimately keep our sanity as we try to maintain, to add to right, the software that create with this new tooling, because by default it turns into a mess. So, so if any of you who have tried to, you know, build a prototype with, with Claude code or Cursor, right? Yeah, it's delightful in the beginning. And then as, as you try to add to it and get it to do exactly what you want and build exactly how you want it to be built, that you end up, you know, creating never-ending rules and assertions, things to do, things not to do. And over time it starts to become maddening. And yeah, we think there's a tool gap there. We think that You know, there's a much better way to, to approach this. And I think that as, you know, as development changes, there's going to be a real big market and need for, for tooling in the space. So that's what Proove.ai is trying to do. We're incubating products at the moment. We're very early stage, you know, and that's fine. But we're interested in talking to, to people who, you know, who like our vision, like what I'm talking about, and could be perhaps early users of what we're incubating.

And what's telemetry data? Let's start with that.

Sure, sure. So, so a lot of what we started with Improve AI was, was, you know, the telemetry problem. So, you know, how do you capture information? How do you capture the data that you need in order to do these, you know, in order to properly control for, for your AI systems? Um, that was a, that was a problem that we had seen emerge sometime in the late part of last year. So telemetry data, just in, in our case, means anything that you would need to make high-quality decisions about, like, you know, what either you or your agents or your team of agents, right, should do in response to supporting or to fixing, you know, your agents, right? So we start with this problem first because if you don't have, you know, if you don't have a good way to capture and to observe what's your what your agents or what yourself are doing with stuff, then fixing things and automating anything is really hard. We see this as a prerequisite. Some people are going to do it manually, some people are going to have a whole bunch of telemetry collection in place and a whole strategy about it. Some will just want their, ask Claude code to do it or something like that. Both of those are important approaches, but part of what we advocate for at Proof.ai is to make sure that you do have the right observability or the right telemetry collection in place as a prerequisite. And that's the step that people forget. It's the thing that they don't do when they actually start to do their prototypes. They're like, whatever, I'll get to that later. Forget about that. I just want to see the software work. And when they do that and they don't really have any forethought around, hey, how am I going to keep visibility into this? If you don't tell your agents to do it, or if you don't do it yourself, then you're going to come up with this big old observability gap pretty soon into usage, and it's going to grind everything to a halt. We've seen people I've seen it fail in this condition countless times. It's probably the biggest thing that seems to be killing GenAI prototypes. The whole big messaging last year, hey, X% or 90% of GenAI things are not making it to production yet. A lot of it's attributable to the fact that you get these GenAI prototypes really fast and they look like they work, they look like they're done, and then you go and you want to add to them, you want to support them. And bad things happen, right? Because maybe you didn't even— maybe you didn't build half of it yourself. Maybe, maybe agents built it, or, you know, maybe, uh, you know, maybe, maybe you're struggling with the non-deterministic parts of the, you know, of using LLMs for either, either make decisions or create output. So, so that's in a nutshell what it is. It's a prerequisite, and it's a, it's a sub-problem that we think is a natural starting point for a lot of, a lot of people.

And why are you the right guy to build this? Can you tell us about, uh, your career summary?

Yeah, sure, sure. So, um, yeah, I, I'm, I'm particularly interested in it just because, um, if you go back to some of my early, uh, you know, my early experience, like very early in the late '90s, right, I was building generative language systems as, as a, as a PhD student. Now those generative language systems, you can imagine don't look anything like today's, right? So we're talking rule-based things. Yeah, old stuff, right? Um, but, um, it was, you know, at the underlying to all of these things were, you know, algorithms to create text, but there were also things like inference engines to, to make decisions and things like that. And one of the, one of the things that just drove me mad was that as, as we're creating these really interesting models and things that did impressive stuff, we could never really ever license the technology or have any collaboration with an outside party because everybody would just say, oh, I don't really understand how, how that thing is making decisions, or I don't really understand what the chain of decisions was. I don't understand how that would work in my environment. It was just a horrendous hurdle in innovation. And it all came down to the inability to, you know, really have a sane discussion around telemetry, around data capture, around audit. Right. It was just like, sure, I could create something incredibly innovative, but the instant I try to put this somewhere else, it just goes and dies to death. Right. My personal experience was within medical, right, in hospitals. Right. So you can create a fantastic model that works on patient scoring for, you know, for one hospital. You can try to have a discussion on how would we roll this out and ultimately improve the quality of healthcare for everybody, you know, the quality of decision-making. Right. And you run into all these hurdles like, like, okay, but what data was used to train it? How do we know it was all above board? How do we know we're allowed to use that data here? Right. Because how do I— can't see the data. You know, you get into these really ugly discussions like, well, can we inspect the data? I mean, in today's world, all right, do you want to inspect the, you know, the file with, you know, the 2 terabyte file? Right. Probably not. Nobody wants to inspect that. And at this point, humans can't. So it's just, it's one of these things that's been bothering me more recently. You know, I was working at a fintech company who processes payments, right, part of which is fraud modeling. Um, again, it's still the, the problem is still there. It's, it's like if you're talking to regulators, regulators say, oh, you have statistical models that, that judge risk, I want to see, I want to make sure that this model only uses data from your particular from the country, because if you use data from another country to train your fraud models, that's a no-no, right? And yes, that's true, of course we follow the law, that's, um, that's what we have to do. But the burden of truth was just, you know, for these statistical models gets into a never-ending loop discussion, and ultimately what it does is it wrecks innovation, right? So I mean, those are two, two examples. Um, yeah, it's just, it's one of these things that's always been there lingering. We've ignored it for for decades at this point, right? It's not like this is a new problem. We've just ignored the problem and said, yeah, I guess we'll get to that, right? And now's the time when it's really gonna start getting in the way because statistical models are everywhere, non-deterministic models are everywhere, and blind models, meaning models that we don't know what data it was trained on, are everywhere. So it's a great time to finally fix this problem, right? And, you know, that's why I'm so interested.

And how did you meet Marina? How did you guys get to know each other?

Yeah, sure, sure. Just a typical recruitment type of, type of call. I think though that his, his pitch at the time was, was very regulatory-based and audit-based, which, which appealed to me. So ultimately he pitched an idea to me that I happened to want to buy, right? So as, as somebody who was serving as a, as a CTO at a fintech, right, he pitched an idea I hadn't really heard anybody pitch before. It really struck me as, yeah, I need that. I really need a better way to capture telemetry and ultimately build better on top of statistical models. It just resonated with me. But the meetup was via very typical recruitment and outreach channels here. I just got so excited by the prospect and that somebody was finally up for solving this big old problem out there that I I joined up.

What do you have to say about the, the token cost, uh, and yeah, companies like OpenAI is like sort of, uh, selling their tokens at a discount. Do you feel that, uh, tokens will cost more right now? What do you have to say about token maxing?

Yeah, yeah, so, so I mean, the— well, token maxing I don't think is a very good metric for managing engineers, right? It's, it sets the wrong You know, it definitely sets the wrong incentives here. I mean, I understand why it's there, but I mean, productivity and usage metrics for engineers have never been— it's not— never really been a great solved problem, right? It's one way to look at a picture. We get it, it might be useful, but it definitely doesn't set the right things here. Um, the— yeah, overall though, the token costs, right? So, you know, I would expect, you know, expect token costs to of course, you know, drop, right? Model commoditization is, is pretty much expected here. Um, So I don't think that that's really the key thing. The problem though is again around the visibility of those costs and attribution of those costs. So that's still a thorny problem right now. And that's why you need observability and you need better ways of working with your agents because we would expect, yes, so at the moment token costs can be considered subsidized depending on your package and depending on exactly what you're buying. Um, yeah, so I'd expect then people to, to need to watch and monitor their costs even more, just like any sort of commoditized tech cost, right? It's not really a new thing. The problem, right, really to focus though on isn't really, oh my gosh, what about token costs, or oh my— these are two big and high-level things to be worrying about right now. What you should be worrying about is, all right, well, it's going to cost what it costs, you know. Could— do I really understand where my costs are going? And does it matter, right? Where, where in the sort of, you know, ecosystem of stuff that I'm spending do I really need to focus on? Um, I mean, again, so I've seen too many people ignoring the actual problem here and instead saying token costs are too high, or oh my gosh, token costs might spike because they're subsidized. It's just not a productive thing to worry about. I mean, all of these are you know, what you're, what you're again doing is, is avoiding work, right? The work to be done is, all right, well, who's using what? Do we understand what token costs are attributed to what? And most importantly, and most difficultly, does— do we understand where token costs go as part of our development cycle, right? Because you might end up with token costs that look good, you know, at some point, but there's all these anomalous spikes that end up doubling or tripling your costs that you don't really have any visibility into, right? And if you don't think through how you want to report on these and how you want to monitor them, right, you're just going to be in a world of hurt, you know, later. Simple examples are things like, all right, I mean, I can make a, I can make a system that, you know, runs just fine, right? And runs at a reasonable cost. But every time I add, let's say, a new, you know, a new use case or I fix a bug, maybe I, maybe I burn like 10 or 20 times the tokens. I mean, that might be, that might be one of those things that looks like an anomaly today and I don't really have visibility into. But if that's the cost of your application, you need to understand that. And there's not going to be a magic tool that tells you that you're going to have to do that thought work yourself, just like you had to do for cloud costs, right? This is not— you know, it's not like all of us who used to say, oh my gosh, right, cloud computing, right? You know, certainly was no cloud computing when I was a developer. You know, it was later. And sure, in the beginning, it's not like there was this beautiful report that told you where your costs were going, right? This was a thing. This was This was, oh my gosh, it's so useful. And so, oh my gosh, I don't know where my— oh my gosh, how did my bill get so big, right? And oh my gosh, I don't know where my costs are going. Yeah, let's not do that again. Let's not— you know, we should know better, right? And, uh, it's, it's again the thing that people are, are missing out on because it's a hard problem. And instead they're, you know, instead the messages around, well, token costs are too high, they're subsidized, oh my gosh, right? Which is— it's, it's not really the thing to focus on.

Who do you target and how do you know that they're paying too much for tokens? Like, uh, what companies do you target that have a burning need for what you're building?

Yeah, yeah, yeah. So, so the— we just want, you know, we just want people— we're focused mostly not only on costs but on, on just giving people the tooling, you know, to, to, to create reliable, resilient GenAI systems that are maintainable and that you have visibility into. Too, right? So we're actually focused on a number of different actors at the moment. We're functions much like an incubator. We're incubating multiple products at the moment. But some of the most people— some of the people with the most acute pain are not developers at the moment, right? So they're specifically, you know, there's, there's a leading edge, you know, group of people specifically in, in, you know, working with finance, right? So people who use GenAI to build models in Excel or in some equivalent, right? Those people obviously they built, right? They write things. Maybe it's not software code, but it looks an awful lot like software code. You know, it's formulas. They have data provenance issues. But most importantly, right, there's a high degree of pain and downside if their models either use the wrong data or are wrong, right? So there's an interesting intersect in these non-engineer disciplines right now that use generative AI to build stuff where there's a high degree of loss for specific wrong things. So people building— people in finance is one, people in law is another, right? People in insurance is a third, right? So we are looking at those types of actors. And again, if there's anybody out there who, you know, who is a professional and you're using GenAI to build stuff and you're starting to get hit by this whole, oh my gosh, this is productive, but oh, this there's all these, you know, why did cloud code start pulling in, you know, this data point when it really shouldn't, right? And that I didn't see it until revision 10 and now I'm in trouble, right? Those, those types of things are, are really important. And in many ways they're easier and more precise to deal with than, you know, your software engineers out there, um, because, you know, they, they have a very narrower, narrower build space, right? Now we're a build surface and a much higher and acute problem if things go wrong with their cloud code. Yeah, that said, we're not ignoring— we still are addressing generative AI engineers as well. Absolutely. You know, absolutely a problem. But, you know, the individuals that typically get bit by this, you know, the most are, are going to be the individuals who are using cloud code as your primary way to develop. You're, you know, you're looking and you're putting out lots of prototypes, but you're at that point where you need to get You need to modify, you need to get something into production and you need to walk away from it. Right. So I think that when you're at that point where you need to move on to something else is where people start to feel the most pain. So typically, though, that means you've already built something, it already works. You're already pretty happy with, you know, its return. It's just now you've sort of been bitten by the fact that, you know, it needs babysitting, right? And it requires a lot more babysitting than you would have thought and walking away from it and moving on to your next project. Is Burdenson— sure, you would love to hand this off to somebody, but you can't, right? For a bunch of reasons, right? That's sort of that sweet spot of like, you know, you might know all of the specs and all of the stuff that you've put into your context and your instructions, but like, how do you hand that off to somebody else, right? In the software world, it's straightforward, right? We've built procedures and ways of doing this so that anybody can contribute to anybody's code. Yeah, we're not there yet with, you know, with coding assistants. So like if you're at that point where how do I hand this off to another person? How do I get out of this? How do I— every time I make a change, how do I not overwrite 3 different, you know, things that I did earlier? Hey, I forgot I did that, right? Because I'm doing something else now. How could I possibly remember every conflicting or convoluted instruction I've given to it? If you're at that point, that's, that's a good time to talk to us as well.

And this problem is probably gonna 10x itself when we're hitting the agentic age of things, when agents are fully independent and there's no human oversight.

Absolutely, yeah, exactly. So that's what we expect. We expect that as, as human, you know, operation continues to decrease, right, which is an assumption we make, um, and that you— that an engineer will be running larger teams of agents, or professionals will have more agentic interaction than Yeah, then let's think ahead, right? Does the problem get better or does it get worse? It gets worse, right? And, you know, it's one of those things that we're sort of just ignoring right now as if it's going to go away. And I think if you look and say, yeah, it's a little bit painful, but whatever. Yeah, just think forward to next year when you have, you know, when you have more agents, less people in the loop, right? Then what? So it's a good call to action to start today.

And what would— how would your infrastructure change when agents are in charge of making decisions? Would you have to speak their, their own language? And also, what do you expect by then in terms of token costs? Let's say in 5, 10 years from now, should the token cost go down? Should there be even another business model, right? Um, yeah, what would you see in terms of token cost in the future?

Yeah, yeah, it's, it's, I'm almost, I'm going to avoid the question. I don't actually know, right? I think that it'll stabilize most likely, but you know, a part of commoditization is the token costs will become what they're worth to everybody, right? Which means that assuming you're developing on par or better, or you're at least as efficient as your peers, it's just not something to, to worry about, right? Is it going to get more or less? It's, it's, it's almost irrelevant just because you don't really know what the work you'll get out of a token is in 5 years from now, right? So even if a token costs a lot more, what if you get more work out of that token? Um, so it's, it's just, yeah, it's, it's just that, you know, I don't think we'll end up in a— end up balancing out that, you know, you somehow you need to anticipate that things are going to go up or down. I think you need to anticipate that they will cost what people are willing to pay for them. And as long as your token efficiency— and you have an eye for efficiency and you understand what costs you're doing— you're probably going to do at or better than average. And if you're doing at or better than average, then it becomes a matter of don't worry about it just yet, right? You know, treat it like electrical prices or anything else, things that fluctuate. Um, as somebody building stuff, it's probably not the thing that it should be top of mind, but token efficiency should be, right? Because if you're vastly inefficient, right, you might find out that all of your peers are doing better than you, and token costs rightfully go up because, you know, because that's what people are willing to pay for them. Um, in that case, you are going to be in a world of trouble, in world of Hertz. So, you know, again, we— the best you can do in that situation is not worry about token cost, but instead focus on token efficiency, right? Where, where are your tokens really going? What work are you getting done in response to them? And making sure you have the visibility, right? So if you don't even have that visibility today, then that's the thing to start with, right? Start there.

How do you manage your engineers as a CTO with like a multi-decade experience?

Oh, what's up? Challenging question. Uh, yeah, so at the moment, right, at the, you know, the, you know, we're running a very early stage, you know, lab here at Proof.ai. So, you know, a lot of this is, comes down to giving engineers, you know, the flexibility and the ownership to innovate and to create. Um, that's not new. I mean, I think you've probably heard the same message from any, any big tech company, right? It's, it's It's a matter of putting in guardrails for engineers, instilling a culture of ownership, making sure that they can connect directly to customers and directly to people with the pain, hear firsthand, you know, what people are suffering from, build against it individually, remove barriers, and then ultimately get them to place— help them with that feedback, right? And help them with advice. Yeah, I don't really think I'm saying anything different than what we've we've heard from, from big tech, you know, companies in the last decade or so. It is the model that works. You know, what is different though, I think, in the GenAI world, right, as somebody leading this, is that they, you know, becoming as a, as a, as a manager, as a, as a leader, um, this is not the time to, you know, get out of the details of GenAI development. If anything, it's the opposite, right? I remember, you know, there was a bunch of things like infrastructure automation that, that did really revolutionize how software was developed. CI/CD, you know, infrastructure as code, all of those good things, things that really did change how we develop software. As a leader and a CTO, I'm not really sure it was necessary to get into the weeds and into the details of how that really worked, right? I think you need to know as a concept, what does it do? How does it work? Okay. You know, in today's world as a CTO, though, and how engineering practices have changed, right? The two big things that are different now are one is it's not the time to be high level, right? It's— this is the time to say, all right, yes, in principle you think you know how clogged code works and whatever. Yeah, don't do that. I would use it directly, experience the pain yourself. This is one of those, those rare times when it's actually a good idea as a leader to just experience the actual code, you know, the code coding challenges, the creation challenges, right? And understand what's, what's different because it's not obvious, right? I don't think it's— it doesn't really lend itself to leading a few— reading a few articles and then saying, oh yeah, I get it, it's different, right? It's not a, not a good time for that. Um, so getting back to creation is important, and the good news is it's much easier now than it was, right? So, um, you know, it's, it's not really— you don't have that, that burden as a leader to say, oh my gosh, yes, in theory I want to contribute, but I'm out of my game and now there's 50 things I don't understand that I have to learn. That constraint's gone, right? It's easy to create again. And, you know, that's one of those things as a leader that becomes unintuitive and something that I think you need to— yeah, I would recommend people do. You know, there's— I guess there's other things as a leader here that, you know, become important as well in leading the team, right? Throwing away past assumptions around how development work is really hard, but it's also necessary as a leader today. You know, accepting that development practices are potentially going to change, right? And sort of saying, hey, let's forget about a lot of the things we've learned over the last couple of decades is really hard. And I think that that's something you need to do now. It's just sort of like Again, start from first principles. If this is the tooling that you're given now, knowing that it's very productive and you can produce a lot, let's build, build from the bottom up. How would we work within this? You know, within this, don't go ahead and take the things that you've learned from the last 30 years and try to apply them to, you know, to how things work today. That's, that's a pitfall, and it's a typical way to approach, right? It's like, hey, cool, we have this software engineering org, we decades, we know what we're doing. Oh, cool, there's a new thing called coding assistants, but let's shoehorn them into our processes and make them work the way that we think we want to work. Nothing fundamentally wrong with that. It's going to cause pain and confusion for everybody. Even if it's as a thought exercise, why don't you say, let's pretend there are no constraints. If this was the tool we were given, let's build from the bottom up. What would we add on? Let's build. What was problematic about this build? What else do we want to add into the picture now? If you work another way, it's just as a leader, the natural attempt is to bring wisdom that came from 30 years ago to everybody and say, oh, but you're going to get bitten by thing X, Y, and Z. They're not going to get bitten by those things now. So you can't bring it up. So the best thing you can do as a leader is to say, forget about everything, let's build it from the bottom up and let's see what happens. Let's— how do we want to work as a team? And don't sit there ranting and raving about, you know, Scrum and this stuff that you, you know, stuff has worked for you for the last 20 years, because it might be a total— you might not bring anything to the table.

Love it. Greg, thanks for being on today. Where can people find out more about you?

Yeah, sure, sure. So I'd encourage everybody to go to proveai.com, and if you want to read some of our thought leadership articles around what we're doing process-wise, proveai.com/blog/pdd, um, prototype-driven development, PDD. I think those are a good way to, um, to, to read about what we're up to. If you are interested, so if you're in the legal profession, the law profession, and you're building with cloud code, or if you're a GenAI developer and, and these problems resonate you with you. We're not looking to sell anything right now. We're looking for early users and feedback providers. So the best thing is, is just to, you know, read about what we're doing, submit in, submit your information, and we'd be happy to talk, right? This is an early stage lab.

How is Greg in