Living Lindy: a No-BS Conversation on AI Agents with Flo Crivello

Hello and welcome back to The Cognitive Revolution.

Today my guest is Flo Crivello, CEO of AI agent platform Lindy.

This is Flo's 6th appearance on the podcast, but only his 2nd that goes deep on Lindy specifically. I reached out to Flo to request this conversation while I was preparing for my recent presentation on AI agents at Imagine AI Live, because I know that Flo lives and breathes AI agents as much as anyone. And yet he always cuts through the hype and shoots me straight on what's working and what's not working yet. For comparison, that presentation, which we ran as a recent episode, is a higher-level overview that I hope people will find valuable for structuring their thinking and perhaps even bringing leaders in their organizations up to speed. Whereas this conversation is a much deeper dive fit for AI engineers, tinkerers, and builders of all types and full of practical nuggets. Now, if you've been following the AI agent space, you know that it's been a bit of a roller coaster. We've gone from mind-blowing autonomous agent demos like Baby AGI and AutoGPT back around the time of GPT-4 launch in 2023, to a world today in which most AI agents, and certainly the vast majority of AI agents in production, are better described as intelligent workflows in which AIs perform specific tasks that do require fluid intelligence, but nevertheless exist within a structure and control flow that was designed and built by humans. That said, while it's taken longer to get here than I expected, 2 things are now becoming very clear. First, even in this more constrained paradigm, the practical value of AI agents is becoming huge for use cases across customer service, sales, recruiting, inbox management, scheduling, and information synthesis. And second, the more open-ended choose-your-own-adventure form of AI agents are also now starting to work. Flo describes a few that he's built for himself on the Lindy platform, but you can see this happening everywhere from OpenAI's Operator to Claude Code to Google's spreadsheet integration of Gemini 2.5, which I recently tried again for the first time in months and found to be dramatically improved. At the same time, these are only starting to work, and just how quickly these capabilities will continue to advance is of course hard to say. Flo, as you'll hear, isn't convinced that we should just extrapolate METER's historical trend analysis of AI task length doubling times all that far into the future. And perhaps as a result, he's not ready to go all in on multi-agent systems either. Beyond that, you'll definitely want to hear Flo's takes on model choice and managing model upgrades, fine-tuning, RAG, whether the tools that AI agents are given to use should themselves be intelligent and dynamic or dumb and deterministic, how Lindy helps users maximize performance by curating successful examples, and how and why, even assuming we do get AGI soon, Flow believes that scaffolding will continue to play a critical role in helping humans understand and control AI activity. On a personal note, I'm also glad to share that I do now have a Lindy doing meaningful work for me. I've previously mentioned that we're experimenting with sponsored episodes, but to date, these have been very few and far between. Now, with podcast guest pitches from AI companies' PR firms ramping up to a truly unmanageable volume, I finally decided to enlist the help of an AI agent to process them all. And so I built a relatively simple agent on Lindy that takes a PR firm email as input, researches the company they represent using the Perplexity API, evaluates whether they're a legitimate AI industry leader that our audience would actually want to hear from, which by the way does filter out a large majority. And then for those few that pass review, finds contact information for key executives via the RocketReach API, And finally, drafts personalized outreach emails explaining our sponsored episode offering, which are placed directly into the Turpentine business team's Gmail drafts folder for review and sending. One could, of course, build such a workflow on many different platforms, but I do find it valuable to see how an AI-obsessed team like Lindy is thinking about these things. And Lindy does a particularly nice job of having AI set all the configuration values for you, of making sure that AI has full context at each step, And of allowing you to chat with individual instances of a task. In any case, this is exactly the kind of task that's perfect for current generation agents and that every business should be experimenting with now. Well-defined, multi-step, requiring both research and judgment, and ultimately allowing us to scale a valuable activity that we previously couldn't.

Of course, I'll be transparent with you, the audience, by always identifying any sponsored episodes that we do bring you.

Today's is not one, and I'll maintain my own agency by remaining personally responsible for the companies that we do and do not choose to feature on the feed. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or leave a comment on YouTube. We welcome your feedback via our website, cognitiverevolution.ai, and also by DMing me on your favorite social network. Now, I hope you enjoy. This no BS conversation about the state of AI agents with Flo Crivello, CEO of Lindy.

Flo Crivello, CEO at Lindy. Welcome back to the Cognitive Revolution.

Yeah, thanks for having me.

Let's talk about agents. Uh, it's on everybody's minds. I've been studying the subject from a bunch of different angles, and I knew who I wanted to call to get an honest, real talk assessment of where we are in the development of agents. So maybe for starters, kind of a rudimentary question, but maybe at least wanted to set the terms and hopefully deconfuse the subsequent discussion. What is an agent? And there's so many different definitions, right? Everybody's putting forward their own definition. How much does that matter? And what's the definition that you work with?

My favourite definition is Harrison Chase's definition. By the way, first of all, I don't think the definition matters. I think you know it when you see it. You can operate perfectly fine without being too nitpicky about the definition. But if you really insist upon the definition, I really like Harrison Chasey's, the CEO and the founder of LangChain's definition. He says it is software which at least part of the control flow is defined by an LLM. That's it. And so I think part of what I really like about that definition, and I think it also pinpoints why it is sometimes hard for people to really define what is an agent, is it's a spectrum. It's like the more of the control flow of the software is defined by an LLM, the more agentic the software is, which I think is the same as for humans. It's like sometimes some humans in companies operate within very tight guardrails and they're not very agentic, they're not high agencies. Some others are like live players or whatnot and have very high agency. If you think of a Jensen Huang, the CEO has ultimate agency. Yeah, that's my definition.

Okay, so that would mean a— and I think the spectrum of definitions that I've heard maybe ranges from, like, Dharmesh on the one hand said basically anything that you go to that's an AI that helps you get stuff done is an agent. He doesn't care if it's a fully deterministic workflow. And then on the other end, you've got, like, I heard a good one from Amjad, which is that it's an agent when it decides when to halt. Yours is, I would say, a little closer to Dharmesh's in that any tool call would count, right? And any fork in the flow, like if-else type of logic, as long as that's entrusted to an LLM, you would count that.

Yeah. I think where my definition overlaps with Amjad's is decides. To me, decides is like there is this brain, right? To me, the brain is the LLM. And so it's like if you introduce the decision power made by the LLM inside your workflow software, whatever you want to call it, it's agentic.

Okay, so I think this is maybe something we'll kind of return to as we go. Not too much there in terms of autonomy or open-endedness required. And I think that is one of the reasons I'm kind of digging in on this is because I feel like when I talk to people who who aren't super deep down the AI rabbit hole every day, as both you and I are, they have seen your sort of baby AGI type demos, or maybe your ChaosGPT type demos, and they've kind of latched onto that in their minds of like, oh, wouldn't it be amazing if I could just give a real simple instruction to an AI and it could just go do everything and figure it out? and come back to me when it's done, right? The old kind of come back to me when it's done. That I feel like is leading people often quite astray in terms of what is actually realistic today. And also maybe in terms of just how much work is really needed to, if nothing else, assemble context for your agent or guide it to the necessary context so that it has a chance of doing what you want it to do accurately. But maybe you're going to destroy my worldview here in a second with telling me that you actually have tremendously open-ended Lindys running your life. So keep that in mind and tell me, what's it like living the automated Lindy lifestyle today?

Yeah, I wouldn't say running my life, but yes, I have a couple of very open-ended Lindys running. I think this definition of agentic is just on the far extreme of that spectrum that it's just defined. When you let LLMs take a lot of the decisions, then it's very agentic. And so that's just this definition. I was thinking about it actually the other day because the very first version of Lindy perhaps was overestimating the LLM's capabilities, was definitely overestimating the LLM's capabilities. And it was just these open-ended agents. And so since then, we've actually backtracked and let you set up more of that deterministic scaffolding where you can really force the agent to go through a deterministic set of steps where it's like, hey, suppose you want a customer support agent. You're going to receive a support ticket on Intercom or Zendesk or whatnot, and you want it to check your knowledge base on Notion before answering the support ticket. You're not asking it. You're not asking the LLM, please, I beg you, LLM God, go ahead and check the knowledge base. You want that to happen deterministically all the time. You want it to be hard baked into the cognitive structure. of the agent. So now, like, we— like, the current version of our product does let you decide however tight you want your guardrails to be. When do you let the agents roam free and so forth? I mean, I have some agents— like, it's funny, actually, my meeting scheduling agent is just a pure agent. Like, it's got very little to no guardrails. It's just like a very big prompt where I tell it the rules and I tell it how I like my meetings scheduled and so forth. But within these boundaries, it does pretty much anything it wants. I have another agent that, this one's going to sound funny, but, or very minor, but it wakes up every week. So every Monday morning, it woke up a couple of hours ago and it checks whether there is a new podcast from my favorite podcasters who are Dharkesh and Denis, and obviously The Cognitive Revolution. And it looks for the podcasts. And then once it finds them, it summarizes them and it sends me a summary. That kind of has to be an open-ended agent because this concept of like, look for the podcast, because there's no one source of truth. So you gotta kind of just like look for the podcast, go on YouTube, go on iTunes, just like figure it out, you know? And then once you found it, like ping me. So that's an open-ended agent.

One of the things I noticed as I was using the product recently is You've built a bunch of primitives, basically, like lots of integrations now, for one thing. So as time has gone on, it's become much more like a Zapier in terms of just like, depending on whatever app you want to use, there's an increasingly good chance it's going to be there. But then you have your own primitives that are like, search the web for something and not too much detail, given I obviously kind of want to make it seem like free and easy for the user, I guess, and maybe sort of keep some of your secrets for yourselves. But I guess I have a couple of questions on that. One is like, is that an agent? And if so, aren't we kind of running into some troubles in terms of how we draw boxes around these things? Because this is something I'm really looking at from a lot of angles in terms of Liability and testing and sort of even just design. Like, what's on me? What's a tool? When I call your Lindy official, search the web for something, should I think of that as kind of an open-ended agent? It seems like it might be doing a bunch of steps in there and I don't really know what they are, and it's kind of coming back to me when it's done.

No, it's literally a Google search. It's literally a Google search. We try to be careful. There is an agent, and then there is the tools that the agent In the early days when we were much more exploratory and trying to figure out what even was an agent, we did experiment a lot with agentic tools. So you can almost think of it as a subagent. And we learned that's a pretty bad idea. You really should draw a pretty sharp line between your agents and your tools, and your tools should not be agentic, basically. And the reason for that is because it's, one, empirically, it just doesn't work well. 2, it just makes it very hard to reason about your system because you're introducing— it's hard enough to make one agent work. Now you need to make 2 agents work. You need to figure out how to get them to interface with each other, which is really tricky. So anyway, no, I mean, the search web action is just a Google search.

Okay, interesting. So does that make you bearish on things like all these new agent protocols that are coming out? I mean, people are talking about, of course, A to A from Google. MCP is meant to be more of a tool thing, but I'm also seeing the pattern increasingly of smart MCPs. Not that often yet, but a familiar one that I've mentioned a couple of times is Augment created a sort of clone of Claude Code.

Yeah.

And in the Claude Code blog post, they say, well, we have a planning tool and it can call the planning tool and Augment was like, well, we don't have a planning tool. What should we use? Should we make our own? They looked online, they found Pietro's sequential thinking, or whatever exactly it's called, tool already packaged up as an MCP. And so they're using that. So they have the coding agent, but then it's tapping into this other sort of smart planning MCP, which obviously has interesting questions around what context is it fed and how much of its thinking do you get back? But Would you call yourself bearish on all these sort of multi-agent frameworks at this point?

No, I wouldn't call myself bearish. I've been very excited about multi-agent systems for a very long time. I do think though, they are a lot younger. It's much harder to make a multi-agent system work than it is to make one-agent systems work with a bunch of tools. It's just much harder. I think part of that is the models haven't been really ready. I think part of that is the tooling hasn't really been ready. And I think perhaps part of that is we've not really had I'm sure these protocols are going to help. I am excited about Google's work here. It seems like something like this is going to be necessary. It actually really reminds me of in supply chains and like e-commerce and shipping and all of that and logistics, they have this protocol called EDI. It's a very, very, very old protocol that's like, it lets people say like, it lets you formalize your relationship with the supplier. So it's like, hey, I bought this from you. Now you're going to confirm that you, like the purchase order went through and now you're going to confirm film that the shipping went through and, oh, actually quantity is different and all of that stuff. So it's very formally defined. And look, that's the backbone of the logistics of the modern world. Like, the whole world runs on EDI. So I do think we are going to need something like that for intelligent communication. I think it's going to help a lot to make these systems more sturdy.

Okay. We're just not quite there yet. Maybe they are in the same spot that you were a year ago where They're just a little bit ahead of the game, basically.

It's early. It's early for sure. But I mean, look, I think it's close enough to start thinking about them. I have some multi-agent systems that I use on a day-to-day basis in production.

Okay, tell me more.

Well, it's going to sound like toy examples, but I have— so obviously Lindy is in online meetings, and she sits in the meetings, she takes notes, she does a bunch of stuff. It's like my meeting note-taker Lindy is huge. And one of the things she does is going to sound weird, but I do a lot of interviews with candidates. And when I chat with the candidate and it's decidedly not a fit, and I just decide to like pass on them, like during the interview, sometimes it's more ambiguous. I'm going to talk with the team because I'm like the last round of interviews, but sometimes it's just like, this is, I'm going to veto this candidate, you know? And so what I do is the candidate leaves the meeting. And my Lindy is still there. And so I talk to my Lindy and I'm like, Lindy, let's just pass on this guy, send him an email. And so what she does is, so that's my Lindy note taker. I have another Lindy that I call my Lindy chief of staff. And that Lindy just does a bunch of things for me, like exactly these kind of tasks. She knows what I mean when I say pass on the candidate. She does a bunch of things. She like, first of all, she doesn't send the pass email immediately. She waits a couple of days. If this candidate had been introduced to us by a recruiter, she also shoots a note to the recruiters to let them know. Like there's a lot of stuff that she does behind the scenes. And so when I do that, my Lindy meeting note taker sends a message to my Lindy chief of staff to tell her like, hey, Flo wants you to pass on this candidate. Can you please do so? So that's like a simple example, but it's real. Like, I use it on a weekly basis.

How gnarly do those things get in terms of the overall control flow?

Well, I mean, the agent delegation and collaboration itself is just an extra step. But now agents as a whole, regardless of the multi-agent system, can get very gnarly in terms of orchestration, can get very, very complex.

Hey, we'll continue our interview in a moment after a word from our sponsors. Maybe before digging into another example or two, of course, everybody has seen the meter graph that's been kind of the talk of the town. In fact, I think we were together the weekend that that Dropped or no, that was maybe RE bench that they put out at that time. And then they circled back to their doubling time for agent task length a little while later. I was thinking of the curve. I know you know it, but basically they've gone back in history, looked at the task length that agents could do 50% of the time, plotted a straight line on a graph, determined that the doubling time is every 7 months. More recently there's been, hey, maybe that's even kind of sloping up a little bit. Maybe the doubling time now, if we are indeed in a different regime, looks like maybe 4 months. What's your thought on the increasingly infamous meter graph?

I don't have a very strong opinion on it. I think it's very dangerous. And I say that as I'm very AGI peeled. And so I'm always the one to actually talk about the exponential takeoff and all of that stuff. But I will acknowledge it is dangerous to draw conclusions and to drive these lines on a log graph when you have so few data points. I think in the case of AGI, you actually do have a lot of data points. You have like 60 years of Moore's Law. So you just know that compute is going to keep increasing. And in the case of AI, at this point, we've gone through, I don't know how many, like 5 to 10 orders of magnitude, and we see the scaling law just keeps working. So I think we have enough data points to draw that line on that log graph for AGI. I don't think we've got that for agents yet. Yeah.

So how would you say that that lines up with what you have seen as you've created 1,001 Lindys over the last year?

Oh, I share the empirical observation so far over the last 2 years, right? I have seen the same trend described by this line. Like, I don't dispute the past. It is the future projection that I have a question mark on. I just don't know if I can keep drawing this line. I don't know if it's going to be linear or exponential. But yeah, I mean, look, when we started this, and I mean, you know, we started pre-GPT-4, we started with GPT-3.5. In hindsight, we were, well, I wouldn't say too early because it was only 2 years too early, which I actually think is the right moment to start a startup. But definitely the company, the agents didn't work. They really didn't work. It was so dumb. 3.5 was profoundly dumb. Then GPT-4 came out and then things changed. And GPT-4 was too expensive and it wasn't as good as the new models. And now we have like Claude 3.7 Sonnet and we've got Gemini 2.5 Pro. And like these models are just like incredible. They're actually very, very fast, very smart, very cost effective, huge context windows. So I have seen all of that happen over the last 2 years. And yes, I have very high expectations for the next 2 years. I don't know if those, if they're going to be exponential. I don't know if they're going to be as strong as what happened over the last 2 years, but Yeah, I do expect agents to get better and better.

Of course, it's important to keep in mind that 50% versus a sort of 99%, that is a pretty big gulf. Where would you say, in terms of task length or complexity or however you think about it, people should aim if they want to put points on the board as a new user of Lindy? What's something that is sort of the maximum that can be practically valuable, maximum practical value, while being confident that you can get this to work? How do you guide people?

Yeah, it's a slight reframe around the question, but I think of it more as ambiguity of the task rather than task length. I'm calling it a reframe because fundamentally it's the same thing, right? Because basically, if you can describe something as a sequence of steps, Basically, that's a low ambiguity workflow. It's just like, I kind of understand this workflow and I can describe it in this sequence, right? Again, that's basically sort of the equivalent of saying it's a short task length because it's just a succession of short tasks length. But I think thinking of it in terms of this succession is important because that actually allows you to cover a much broader set of work tasks, right? Because many work tasks are actually— few work tasks are just in and of themselves a 30-second task. But actually a surprising amount of them is a succession of 2-minute-long tasks. So with that in mind, yeah, I think anything that you would feel comfortable giving to an intern with a Google Doc, and the Google Doc describes a succession of steps, and it's like, honestly, the length of the Google Doc doesn't even matter, right? Because all that matters is the maximum complexity of any of these steps. And I would say even that doesn't matter a whole lot because modern AI agent builders, and that is certainly the case of Lindy, have this concept of human in the loop. So you could totally just build your Lindy, basically turn that SOP, that Google Doc into a Lindy. So then you have a huge Lindy. And then if you detect that one step is particularly risky, you just toggle human in the loop on that step and now you insert yourself. And we're actually about to start something that's like, we call it IC or LHF in context reinforcement learning from human feedback. Such that if you toggle on ask for human confirmation on any step in your Lindy, she learns from your feedback little by little. And she actually learns really quickly. I mean, you've seen the same papers as I have about in-context learning. It is surprising, the in-context learning ability of these models.

Interesting. So that's a feature that basically helps people curate the gold standard examples by doing it bit by bit over time.

That's right. That's exactly right. You're thinking about it in the right way.

Yeah, I really like that because, boy, is it hard to get people to sit down and bang out some gold standard examples. I find I have had quite a few adventures on that front, actually, and it's been really eye-opening just how some people just, you cannot get them to staple their pants to the chair and do it. I don't know. It's very weird. That remains, I guess, my number one tip for performance optimisation. I mean, obviously, write good, clear instructions is key, but presumably most people are at least able to sit down and bang out a couple of paragraphs of instructions of what they want. The gold standard examples typically don't exist, I find, or they're so fragmented across context or the chain of thought. It was always kind of in their heads. All those problems really hold people back. So I really like this idea of just basically starting off in a human-in-the-loop paradigm, having people come in and review and/or fix, and then compiling those and building up longer prompts that drive performance by leaning on those examples. That's really good. Is that the number one driver? Few-shot? Prompting, is that still the biggest thing?

Big time. Absolutely. Yeah. Yeah. Much better than instructions.

Any push into fine-tuning? I mean, that would be obviously the next step at some point, right?

No.

Maybe at some point one day. I just think the models have become so good. I feel like fine-tuning is a little bit of a thing of the past, isn't it? We heard a lot more talk about it a year and a half ago than we do today. We used to have a fine-tuning team that we cut, frankly, because part of the issue was just the open source models did not deliver in the same way that we hoped for. And fine-tuning went from a world of, look, again, Lindy used to not work. And so we were like, ah, the models are not ready. We need to fine-tune them to have agentic behavior. And then the models worked. So now it just became a nice to have. I just don't think the juice is worth the squeeze for fine-tuning. For the vast majority of use cases.

Yeah. I mean, OpenAI just put out their reinforcement fine-tuning to, if not everybody, at least a lot of accounts this week. So it seems like they haven't given up on it yet. This has been one of the biggest divergences, though, between them and basically everybody else, right? Google has made sort of a token effort at most. Claude allegedly was going to allow you to fine-tune Haiku at one point. I still don't think I've ever been accepted into that program. And so basically OpenAI is kind of leaps and bounds ahead of anybody else in terms of their fine-tuning offering, but they must still be seeing something from it, right? To be pushing something like that all the way to production.

Well, first of all, it's a very big company. They have a lot of things on the stove. It's hard to infer too much. And also, look, I'm not saying fine-tuning is completely useless. I think, first of all, if you operate at scale, that's the first requirement, because there's a very high fixed cost of fine-tuning that you've got to amortize over a large volume. So if you operate at scale and if you have an important, a critical part of your workflow that you are looking to make faster, cheaper, more reliable, and if that part is sufficiently narrow on the task that it's trying to perform, so I'm looking at like RAG use cases very often have that in terms of like re-ranking and prioritizing and all of that stuff, like then you probably ought to fine-tune a small model. And insert it in that workflow. I think I have heard that Cursor and Windsurf have at least part of their workflows that's using a fine-tuned model, but I'm not sure.

Yeah, I think the point on narrowness is definitely really key. I mean, it's tempting in some cases to try to imagine creating like the fine-tuned model for our company that does everything for our company. And that, in my experience, is not the way to go. It's much more about just nailing down with clarity, what is the desired behaviour on something that really matters? A good example from the reinforcement fine-tuning docs that I was reading this weekend was just like from healthcare, some sort of doctor notes transcript of the appointment to a diagnostic, or I think it's even a billing code, which is like super gnarly stuff. And obviously accuracy really super important there. So That kind of thing often I think will work and hopefully will push the frontier of like what people can actually do in these various frameworks. But the narrowness piece, I think definitely resonates a lot with me. What would you say are like the most, you know, by, if you were to maybe weight by the actual number of tasks as opposed to the number of Lindies, like what's driving the bulk of the value through the system today?

I think if you look at it by task, it will almost, it's going to be ironic. Like if you look at it by task, you're going to look at like the least important use cases. Because almost by definition, if there is a Lindy, well, it's not the least important, but it's going to be like very high volume, very, very, very small tasks. So most likely if we do that, like it's going to be like an email task or like a Slack task, it's going to be one of these 2 things because those 2 things are such high volume. So we see people deploy Lindy to automate their email workflows. That's like a big use case of ours. So like email triaging, email drafting, if you receive a lot of proposals by email. Like, Lindy can look at the proposal and reject it proactively if it's not worth you looking at and all of that stuff. So we've got a variety of use cases here. That's probably going to be the biggest use case.

I mean, that does resonate with me because that is often where I tell people to start, like something simple, relatively low risk, high volume, put some points on the board. What if you reweighted by credits consumed?

If you reweight it by credits consumed, well, I think it's going to be one of two things. The first one really pertains more to our credit system, which is we use prospecting APIs for like lead generation, and those are very expensive. So we charge you a lot of credits for that. But I could show you, like, I have like a recruiter at Lindy that I talk to and I'm like, hey, find me 30 engineers working in San Francisco at this or that company. And then she uses these prospecting APIs to find these engineers. And it's 40 cents per engineer per lead. Right here, that's like $12. And then she's like, okay, I found them. And I'm like, okay, send them an email, all of them. And so right here, again, if it costs me 10 credits slash cents per outreach, that's going to cost me $3. That's one. I think the deep research use cases are quite big. And I'm using it as a portmanteau for a very broad category of use cases. Anytime you want your agent to review, consume a large amount of data and then do something with it, I think agents are, by the way, excellent at that. It's one of such killer use cases because it's just they're good at reading tokens fast. And if a human had to read these tokens, it would be very slow and very expensive. And then she can write a report about that. So one of my favorite use cases for Lindy is we have this Lindy that you can basically think of her as sitting at the interface between the company and the outside world. So she reviews, she is in every customer call that we have, every prospective customer call, every support ticket that we answer. And at the end of the day, she writes a report based on that interface. between the company and the rest of the world, which I think is a very important interface. This is like, hey, this is what's happening. This is what's happening in the sales pipeline. This is what customers are saying. This is the issues we're having in the support inbox, and so on and so forth. And that token goes there. It's like hundreds of thousands of tokens every time.

Is that the same one that I interact with when I talk to the chatbot on the site for help?

It's not exactly the same, but yes, that Lindy in question does also ingest Gotcha.

So when I talk to that, that sort of log becomes an input to the higher-level summarizer.

That's exactly right. And then if a lot of people talk about the same thing to that Lindy, that's gonna come up in the end of the digest. And so that's awesome. And by the way, it sends the digest in the general channel on Slack. So it's just such an awesome— it's like the heartbeat of the company, right? It's like you can think of it as ingest all the context, Broadcast it back, ingest, broadcast it back, right? And just does that every 24 hours. And as a result, the whole team is in sync. It's really powerful.

Hey, we'll continue our interview in a moment after a word from our sponsors. My compliments to the chef, I guess I'll say, on the onsite chatbot. It was actually helpful and was able to respond in a way that felt like I was actually talking to something intelligent. I mean, so many— it kind of strikes me today that, like, and I'm not really sure why this is, obviously, just inertia is a powerful force. The old Tyler Cowen, you are all the bottlenecks. But, man, chat still sucks on most sites, right? It's like, it's not good. When I went to yours, I was quite impressed that it was actually a natural conversation. It actually had relevant answers to the questions. And then at the end, I was like, can you forward this to the team? And it said, like, yes, okay, I've done that.

I've forwarded it on to the team.

And I was like, wow, that's pretty— it felt definitely much more like the future and kind of where I think a lot more people I would have expected to be at this time. Why aren't more people here? Aside from just the general slowness of life, I feel like people have tried, but they've They've often failed to make these things work as well as a few people have demonstrated that they can. What accounts for that in your mind?

Well, first of all, and I don't say that to peddle my shit, but that chat is a Lindy. And so we've just spent a long time, one, crafting the platform as builders of Lindy. And then as builders of this Lindy, as users of the platform, we've also invested a lot of time in that. Lindy. And we know how to build good Lindys because we built Lindy. And so it's a really good Lindy. It's big, it's got a lot of prompts, like the whole scaffolding makes a lot of sense. It injects the right context at the right time from the right source. Like it's just a complex Lindy that we've spent a long time crafting and it uses good models. I sometimes suspect that companies in a misguided effort to save money are using really bad models for these chatbots. And I think you should not. I think you should just like— You should have the best possible model. Well, not the best possible, because today there's like O1 or O3, and that's just going to be very expensive. But come on, give your customers the Gemini 2.5 Pro. It's not that expensive, and it just performs extremely well. So yeah, we just know how to build good agents. But I think, thank you, I will take the compliments and I will pass them to the chair, which is me. It's not me, it's the team. And I think that's an indicator of the current capabilities of these systems, which I agree with you. I think they're under-tapped. It's really crazy just what is possible today that is not yet really exploited by 99% of businesses.

So tell me a little bit more about context. You said, like, it injects the right context at the right times. That's along with, like, the difficulty of getting people to actually buckle down and write some gold standard examples. Generally speaking, the challenge of assembling context or accessing context, Also seems like a constant theme when I talk to people who are trying to implement stuff.

Yeah.

Aside from just kind of many iterations, what lessons would you say you've learned? Or, again, what tips would you give to new users about how to muster the right context at the right time?

I think it is a lot of iteration. I think you do enough writes that you end up building an intuition. And I think That intuition is on, there is a balance between using just similarity, like vector search, like BM2.5 and all of that stuff to search your knowledge base. And on the other hand, handcrafting exactly what to search and what and when and in what knowledge base. And the more you use these products, the more you understand where that balance lies. And so very concretely, we've got that Lindy chatbot that assists our customers. And the customers ask it all sorts of questions. And so we know that if they ask a question about billing, it's like refunds and how do credits work and all of that stuff. We've got a specific portion, a segment of our knowledge base that's specifically about billing. So we're going to have a branch there that's like, he's asking a billing question. Okay, now you consult this knowledge base and this is the kind of query that you draft for this knowledge base. I'll also say, by the way, another intuition you build is when to not even use a knowledge base at all. And when you are not very conscious about or worried about saving money, I think more and more, look, I hate to be the guy who says RAG is dead, but it's not dead, but it's limping. There's a lot of use cases that we have where we don't use knowledge bases anymore. We're just like, hey, these are the 5 or 10 pages we have about billing. It's not that much. You know, it's like a couple thousand tokens. Like, these are the tokens, just ingest them all at once in your context window. You know, we don't do it for the support bot because it does get expensive, at least for now. But yeah, anyway, it basically becomes this hybrid between a handcrafted pipeline, like a handcrafted RAG pipeline, and like a BM2.5, like vector similarity search.

So maybe people should be thinking like, how can I 80/20 or 90/10 this where I will actually create top N categories of situations I want to handle, branch into those, hand curate relevant context, whether it's the 5 or 10 pages about billing or what have you, and then kind of have one catch-all bucket at the end that's like, Okay, for that you can just search through this knowledge base, but then you maybe increasingly pull out of that and you minimize that bucket as you go. Is that what you would recommend in terms of the iteration cycle?

And in the end, it ends up happening very naturally because what ends up happening is you create your Lindy agent and you deploy it and then you monitor it every so often. You check in on it, you look at what it's done, and then you're like, ah, this was really dumb. This is not how you should answer this question. And so you go back, right? And you edit the Lindy, you edit the prompt, you add steps, you modify the knowledge base, you just tweak it around the edges and you rinse and repeat. And I find it funny that there is a sort of natural instinctive reluctance that people have to go through this loop. There's something about it. I think it's just not instinctive. But when you consider The time that you invest to onboard a new teammate, a human teammate, it's a lot. Like training a human takes weeks for a human to fully, fully really ramp up. So I actually think agents are easier than humans to onboard. It's just a less natural mode of interaction because a human, you can just go to them. You're like, hey, don't do this, do that moving forward. Like with an agent, you've got to know how to use this flow editor and you've got to build that intuition that I just mentioned. That's not always going to be the case, by the way. Soon we're going to announce something big that's going to make it a lot more natural to iterate and improve on your agents. But yeah, I would just say iteration.

That almost sounds like a memory module. That's been a space that I've been watching really closely. What's your take on sort of, I mean, there's been a lot of different kind of frameworks for memory, whether it's like graph databases. I did an episode on HippoRAG. There's HippoRAG 2 out now. Then of course there's more inherently neural structures, which could just be a vector database. But we've got an episode as well on Titans, which is like building an MLP into the thing and updating that MLP so that it can sort of retrieve from history. ChatGPT is doing its own thing. We don't know exactly what it's doing, but it's currently got a— it's at least got a mix of the explicit saved memories that you can go and read, and then some sort of more vague, nebulous— it'll check in with your chat history and they don't really tell you exactly how that's working under the hood. What paradigms for memory are you most excited about?

I think this is one of these things, like I read all those papers. I've seen like the HiPo and HiPo-2 papers and so forth. And it's very exciting. But like, I think this is one of these things where number one is the bitter lesson comes for us all. I think as models become better at having more context and at fully utilizing this context, I think like all of these systems become moot because you can just throw it all in the context window. And I think this can be just fine. And also, like, I'm a big believer in simplicity when it comes to these systems, because the more moving parts you introduce into the systems, the harder they are to reason about and to debug. There's this principle of engineering that I really like that states, you need to be twice as intelligent to debug a system as you do to design it in the first place. So if you are operating at full intelligence when you're designing the system, you're going to be unable to debug it. And I think that's the case of all of these fancy memory systems. It's like, you guys are operating at full intelligence here. I can't figure this out. I have to really sit down to understand the systems in the first place. I can't debug it. I don't think you can either. By the way, that's always the problem with these academic papers. It's like none of them is really building with that constraint in mind, which in my experience, when you're building systems that go into production, is actually the defining constraint that you need to keep in mind. So with that said, my understanding of what ChatGPT is doing, memory system there, is that it's actually the simplest system out there that also is operating at the greatest scale. And I don't think, again, to my point, that is a coincidence. And I think what they do is literally just like they take conversations, they determine whether there is a memory that's save-worthy in that conversation. If so, they use an LLM to distill down the memory into a short sentence or 2, and then they just inject all of that into the context window. They may go one step further, but honestly, I don't think so. They may go one step further, which is perhaps assign an importance score to the memory. And so you could imagine, like, hey, you've got so many tokens worth of budget in the context window for your past memories. And you're going to prioritize based on that priority score that you've defined before. And then you could imagine, again, you can go slightly more fancy. You could imagine a sort of decay with time. And so you could come up with a composite score between the priority score and the time score. And maybe the older the memory is, the lower the priority is. You just allocate fewer and fewer tokens to that memory. So maybe you save multiple length wheels of representations of each memory. That's the kind of thing I'm thinking about. But even that incarnation of that system, which by the way is purely conjecture, it's pretty simple. I think it's really simple and I think that's just how it works.

It's pretty similar to what O3 guessed when I asked it. It guessed that it was doing some sort of vector search. Well, it kind of went back and forth between distilled and then vector search or just chat history direct into vector search. But it did have a vector search component in its guess.

I would bet you, I'd bet you a lot of money that there is a vector search in there. If there was vector search, it wouldn't be able to retrieve when you say, what do you know about me? It wouldn't, it wouldn't be able to retrieve it. Vector search won't let you retrieve that unless it's a really fancy RAG pipeline, like a HyperBERT.

Could it be a custom retriever?

It's simple. It's simple. I think it's a very—

They do have a tend— I mean, certainly I think that's a good prior for all the things that the Leading companies do. They definitely have a strong bias toward doing the very simplest thing and just applying a lot of compute. So I think you're certainly right to use that as the jumping off point.

Yeah. Um, you know, look, the other thing is like, they're all building on shifting ground because like the entire underlying paradigm is changing every 3 months. And so the more complexity you bake into these systems, the more assumptions you bake into the systems. And so the more brittle they are to future Yeah, interesting.

How do you think that will impact the frontier lab versus API-powered developer as we go into the future? Right. There's of course been multiple rounds of the debates around who has moats, where does value accrue, et cetera, et cetera. It seems like, I mean, take OpenAI specifically, they're both going toward chips on the one hand and toward buying Windsurf on the other hand, right? And kind of trying to be a real full-stack vertically integrated provider. How do we escape total big tech victory, big tech black hole of value?

Yeah. I mean, I really think of Sam Altman as like, Bill Gates 2, basically, I think in the scope and the breadth and the nature of his ambition. He's very similar to Bill Gates. Like, if you study Microsoft's history, it's remarkable, right? Like, they started as this, like, basic compiler. And then they almost stumbled upon the operating system, but they didn't. It's not like it was just like, oh, pure luck. Like, it's Bill Gates. His modus operandi is very much like, we want to own the whole stack. So he was philosophically open. to the operating system as well as to the compiler, as well to the applications and so forth, right? And so he really saw it as his whole charter to own computers, right? Personal computing, like we own the whole thing, right? And so peripherals, yep, we're gonna do it. We're gonna do the mice and the keyboard. Operating system, absolutely. Modeling software and security software and application layer and yep, yep, yep, yep, yep, yep. We're gonna do it all. We're going to own the whole fucking thing. We're going to be like an index stock on computers. If you believe in computers, you got to believe in Microsoft. So I think that's what Sam Altman is going for. He's like, yep, that too. We're going to do the compute, we're going to do the API, we're going to do the applications, we're going to do the code, we're going to do it all. We're going to do it. Now, look, I mean, history doesn't repeat, but it does rhyme. There are old patterns in here. And in the end, Microsoft did very well for itself, but it's just too big for one single company to own it all. And certainly that's what's happening right now. There is this 800-pound gorilla, and then there's a lot of smaller players orbiting around it. And Cursor is doing very well. Replit is doing very well. Loveable is doing very well. We're doing quite well. The market is just ginormous. This is by far the biggest opportunity in the history of tech and software of computing. So yeah, I think it's gonna pan out exactly like that. You're gonna have a couple of very, very, very big players, and then you're going to have a thriving ecosystem around them.

How about some tasting notes on models? You said a little bit earlier, at least give your customers Gemini 2.5 Pro. I might say Gemini 2.5 Pro is my favorite model today. That Might be a little strong. I mean, I certainly wouldn't want to be without any of the leaders at this point. I do use, of course, Claude and O3 increasingly. I mean, I think I pretty much use all 3 of those on a daily basis. You give me your tasting notes first, especially with an eye toward what is working in the agentic context, and then we can maybe trade notes from there.

Yeah, I agree with everything you just said. I love Gemini 8.5 Pro. It's delightful to see Google finally win wake up. I wouldn't even say wake up because they've been aware of the threat. It's just the machine takes a very long time to get going. Yeah, I mean, I think you ought to be, you ought to be, you should be, what's the saying that like Gen Z says, like you need to be model maxing, you know, you should talk to all models. O3, I like a lot for like very beefy, like I use it as a thought partner as well. And it's quite good for that. Like, I think O3 is the first model that I've talked to, because I use models a lot as like that thought partner. And O3 is the first one that I've talked to that's really blown my mind. Well, I've been like, goddamn, like this is insightful. Like it's really, really opened my eyes to like some deep insights that I've really appreciated.

Do you do that just directly in ChatGPT or is there any other intermediate interface?

Just ChatGPT. Yeah. The memory system in ChatGPT also is killer.

It's so good. So the default though in Lindy is Claude. So how do you decide to make the define? I believe it's Claude 3.5, right? As opposed to—

3.7, I think. I thought we switched it to 3.7. We are considering switching to Gemini 8.5 Pro. We're looking deeply into it.

Okay, so unpack that a little bit more. I mean, I've been poking around different agent products lately, and I've noticed that there's starting to be a division between 3.5 and 3.7, for one thing. And it seems like 3.5 people are like, a little more reliable. We can kind of trust it, does what it's told. 3.7, a little overambitious sometimes, hard to wrangle. Amjad told me a couple of really funny stories about what 37 was doing in the context of their app building agent, specifically when they tried to get it to not edit a certain config file and the multiple ways that it still attempted to do so despite being told not to and despite actual barriers being put in its way. So I was interested to see that that seemed to be, although I might be I might be wrong, but I thought that was the default in Lindy. I guess maybe one way to put it is, how automated or well developed is your sort of eval machine at this point such that, are you going on a set of 1,000 tasks across a bunch of categories where you're able to say, oh, we know exactly how these things compare on a rubric, or how much room is there still for the Proverbial vibe check.

A lot of room, and more and more room. I think we've not invested as much as we should have into our eval suite. And so as a result, like today, we have limited trust in it. It is a signal that we look at, but I think that's also a function of the business. Like, people are using Lindy for so many use cases now, like more and more, and frankly, more than we foresaw initially, that we're very careful about changing the default model, because it's basically hot-swapping the brains of your AI employees. It's a big deal. It's like all of a sudden your entire AI headquarters is operating on a different brain. So we're very careful about it.

Yeah, that's an interesting challenge. Would you go back if I have a Lindy that's working and I just sort of accepted the default, whatever it was at the time, and you want to do an upgrade in general of the model? I could see a strong case for, hey, let's go back and Upgrade the model that everybody's using where they just accepted a default anyway. Like, let's give them whatever we currently think is the best. On the other hand, I could also imagine that could create a lot of chaos. And maybe the alternative would be like, freeze all that stuff and set the default to the new one for people going forward. But that sounds like a pretty hard decision to make because you want to bring people into the future. You don't want to have so many versions you have to maintain or worry about? How do you think about how much to change when somebody's not even aware that you might be making a change for them?

We take it seriously, for sure. We do do it all the time, though. So if you create a Lindy and you pick the default model, or you don't change the default model, it's not like the default model when you created the Lindy was Claude 3.5 Sunet, hence that Lindy is on Claude 3.5 Sunet. No, that Lindy is on the default model. And we change the default model all the time. And so when we change it, it's not like we have to go back. It's just like the Lindys that are using the default model use the new default model.

And so we've actually got, we call them like model labels.

So it's like we've got default, then we've got fastest, which currently is Gemini 2.5 Flash, 2.0 Flash perhaps. Then we've got most balanced, which right now is Claude 3.7 Summit. Then we've got smartest, which right now is O3. And then if you want, you can also just manually pin your Lindy on any one specific model. It's like, I know what I'm doing. I want O3. I want specifically O3. But most of the time when you want O3, you don't really want O3, do you? You really just want the smartest model possible. So we actually do recommend to use the model labels and then to trust us to do our job well, which we do. Like, we've done it all the time and only once did it go wrong. And that's caused us to upgrade our protocols here. It's the very first release of O3. And this is when we also updated our priors on the validity of our eval suite. Because I don't know if you remember when O3 first came out, it was very clearly just a reaction to the DeepSeek blowup, like that weekend. And O3 was not ready. It was simply not ready. It was not a good model. And I, you know, our evaluation suite was weird. Like overall, it showed a superior model, but like it showed a lot of variance. So we went ahead and swapped out the model and did not go well. Like our customers who are using that smartest model label, like reported issues. So we rolled it back same day. It was very fast. So yeah, I mean, we do it all the time.

Reminds me of the Sycophantocalypse episode that we've recently seen.

I think they took much too long to roll back this one. I think that should be part of the postmortem, right? Like there's always like a time to detection and time to mitigation. Like the detection was very fast. The mitigation was much too slow. I want to add one more thing about this idea of swapping out the models. Again, that's part of the value prop. Just imagine if you are still running on GPT-3.5, you shouldn't have to think about that. You should trust us to pick the best model. And sometimes we actually save you money. If and when we swap our default model from Claude 3.7 to Gemini 2.5 Pro, you're going to save money. Your agents are going to be more cost effective.

We've kind of touched on this a little bit, but maybe just to double-click on it for a second, see if you have any additional thoughts. You could put this in the context of building Lindy's or just other product builders who are building agents. I've recently seen, it seems like right now we're in the sort of proliferation of strategies phase still. And I recently did an episode with Andrew Lee of Shortwave who basically said, we just trust Claude. Basically, in Claude we trust and we He said, we do a very careful job with the caching because that's critical to make the whole thing economical for us. And they have the best cache hit savings rate in the game, although Gemini just got into that game in a meaningful way too. But aside from a very careful implementation of the Claude cache, he basically said, we just load the thing up with tools, let it go to town, and basically have really long episodes. No subagents, no handoffs back and forth. And that, he said, gives them the best results. Then on the flip side, you have the OpenAI Agents SDK, where there's a handoff from agent to agent is like one of the core abstractions in that toolkit. And I thought Harrison from LangChain also had an interesting point of view on this recently, where he was basically like a little bit more like the OpenAI side. He sort of said, There's kind of 2 kinds of agents. One is like the task-specific, dialed-in, highly curated context, and maybe you have a bunch of those. And then in front of that, you have a different kind of agent that's sort of your facade, the one that faces the outer world, the one that chooses which of those task-specific agents to call on for any given interaction that it might have. And that one maybe also can be a little longer running and have a more global sense of your history. Whereas the task-specific one, you don't want to distract with all that. You just kind of want to localize it, hopefully give it everything it needs to know, but not too much so that it becomes overwhelmed, distracted, whatever. Any thoughts on, is one of those right, wrong? It depends?

What do you think? I think it's all the above. I think there's just a spectrum of maturity of these different approaches. And today the most mature, and it's really being deployed pretty fast right now, is the single agent system that's using some tools. And sometimes put on some deterministic scaffolding and that just works. Then on the other side is many agent systems and that is still being defined and that does not work nearly as reliably. And then there's another approach, which I surmise is the one that Harrison from LangChain is talking about. And it's also the one I believe that OpenAI makes available through its recent SDK, which is somewhat in the middle because Nominally, it's a multi-agent system. You've got this passing of the baton from agent to agent in the workflow. But actually, when you do that, the agents share the same context. So it's really like, you can almost think of it at that point, to me, if you share the same context, you're really just one agent going through multiple states and multiple stages of your lifecycle. And at that point, it almost seems like a matter of terminology. Is it a multi-agent system? Is it just one agent going through multiple steps? And it's just one of the graph-based Agent systems, I don't know. But that is also, I would actually say it's closer to that side of the spectrum where it's also mature enough to be put into production.

How about a little lightning round on commercial solutions that you possibly use or possibly don't use because you maybe rolled your own before they came out or whatever? But I think one of the things people are always kind of looking for is like, What's a good solution for these different kinds of parts of the overall build-out? So let's imagine you're advising an enterprise, let's say, and they're trying to build some stuff. Data acquisition. I don't know if you guys do any data acquisition partnering. Who would you trust? Who would you look to?

Anybody in that category?

Scale, Search, Invisible are like the 3 main players right now. I suspect this is going to be an underwhelming exercise right now for you because we actually got started to be full, much of that ecosystem bloomed. So we had to build, unfortunately, a lot of our tooling. Like, I don't recommend people to do it. We had to do it out of necessity, and it is not good. I would rather we use stuff that's out there because it's better and cheaper.

Are there any, like, parts of what you've built that are kind of top of mind to replace with something commercial?

The evaluation suite is P0. Like, we had to build it initially ourselves. I hate it. It's not that good because it's not our job to build an evaluation suite. So right now we're looking into Braintrust and there's also like this new French startup. I really like them. They're called Basalt, B-A-S-A-L-T. They're doing a really good job so far. Okay.

Say the first one again too.

Braintrust.

Braintrust and Basalt, B-A-S-A-L-T.

That's right. Yeah. Yeah. Like, you're wrong.

So I assume you're not using anything like LangChain, LangGraph, any observability? Nothing like that.

Everything is in-house. Nothing like that. No.

Well, is there any—

I don't regret building in-house. I would do it again because I think it's too close for comfort to give it to an outside party.

Do you do your own guardrailing? Or is there any sort of, like, if I tell Lindy to do something bad, are you just relying on the foundation models to refuse? Or do you have any additional layers? How do you think about that?

We also built just like a feature, like you can just toggle, like, ask me for confirmation at any point in your Lindy. So we trust the users a lot on that. Yeah, I mean, if you don't want Lindy to send an email without your say-so, don't ask her to send an email without your say-so. If you want her to ask for confirmation, there's like one click. It's like you click on send email and then you toggle, like, ask for confirmation and it just works.

How about voice?

Some stuff with calling now as well, right?

We do voice. We use ElevenLabs for that. We use Deepgram for the transcription. We use Twilio for the phone infrastructure. Yeah, we don't use any higher level, like I know there's like VAPI and like Blend and like, I forgot the other players, but like there's a couple of players there. We just rolled out our own because we again really did care about meeting a lot of the flexibility that we needed. Because that's the beauty of Lindy. You can really create your agent. And every time we looked into these solutions, which we did, they were too opinionated, too high level to be useful for us.

So for ElevenLabs, you're using their voice models for synthesis, but you're not using— because they have kind of call scaffolding type stuff as well at this point, but you have your own thing. We're not using that scaffolding.

Well, we really care about the model agnosticity of Lindy. So in any of your Lindys, and even like in any steps of your Lindys, you can override the model that this Lindy is using. We really care about that. Like if we used ElevenLabs like full-blown scaffolding, you wouldn't be able anymore to define what model you want to use.

Yeah, that makes sense. Any other providers in any category that you would shout out?

Providers? No, no, we're very close to the middle here.

Yeah.

I'm kind of, from my corner of the world, I am kind of bearish LLM apps and agent apps as a category. I don't view a nearly big enough pain point and I don't view a big enough market. I think the market is going to end up being concentrated by couple dozen players or something like that.

I could be wrong.

I hope I'm wrong. And insofar as there is a pain point, I view it as too closely related to what a sentry is already doing, for example.

Are there any of those sorts of things where you have seen an AI-first or an AI evolution? I recently got pitched something that was like, oh, it's an AI-first sentry. And I was like, oh yeah, that's interesting. I've been out of that game myself for a little while, so I don't know, maybe a century now is like an AI-first century. But have you seen or have you adopted any products in your technology stack that you would say are notably next-gen in their application of AI to these classic product infra problems?

Obligatory notes about Lindy. We'll just set this one aside once and for all. But I use Lindy all day, obviously, every day, and it's a life changer. That said aside, I really like WhisperFlow. Like, I use it all day, every day. It's a life changer. Like, it's basically replaced my keyboard. So for those who don't know, WhisperFlow is this software on Mac. They also released an iOS app recently that you dictate to your Mac, and it's just next level in the quality of its dictation. And like, it also tweaks what you said ever so slightly to like match more closely what you would have typed if you typed, because people speak differently than they type. So WhisperFlow is incredible. I have built my own, actually that's Swyx who built this and then I tweaked it. It's like, it's a shortcut on iOS using the Shortcuts app that taps into the Whisper API. And I mapped it to my action button on my iPhone. And that's so good. I can just, I can just, I use my phone. I just like, it's a button and I press it on the side and I can just dictate. And even though I have a French accent, as you can probably hear, it's subtle, but you know, it's there. It's flawless. It is just perfect.

Head nod.

I know, right? I'm basically American. It's flawless. It's really good. What else have I— I was thinking, I have really been disappointed by the incumbents here. So very disappointed. I think there are so many apps that are basically begging for LLMs. And okay, so the Kindle and the Books app on Like, obviously there's no LLM. Like, it's just so obvious. I'm sure there's like some IP reason why there's no LLM here, but okay. Social media. I don't understand. Like, I'm part of all these group chats, and I'm sure you are as well, that are like much too active for me. I can't keep track of them. It's just like there's way too much going on. Like, where are all the LLMs? Like, why isn't there an LLM in there that like summarizes the group chat so far? Like Twitter. Like, why isn't there an LLM? Like, I just tweeted yesterday something that went viral, and there's all these people with very low reading comprehension in your mentions that say something that simply is not what you said. They're attacking a point that you simply didn't make. Why doesn't Twitter have a thing here that says, like, hey, before you send the tweet, you can still send the tweet, but maybe there should be a little message here that's like, hey, this is not what he said, right? Also, like, when you have a tweet go viral, That's an experience everyone's had who has a modest following on Twitter. You get the same points back and back and back and back again. And so it doesn't matter how many times you addressed the point, people don't read the mentions, which they can't be blamed for. Why doesn't Twitter do that? Like, hey, you're making a point that was made and addressed 20 times by the author in the mentions, right? And so now maybe you can respond to the answer he made, and maybe the answer to the answer was also answered too, right? That's my point. So no, I have been very disappointed at the slowness of adoption here in what I perceive to be just obvious opportunities.

Yeah, I agree, broadly speaking. Gamma comes to mind for me as one notable exception. I think they've done a really nice job of just having a super high shipping velocity and trying every conceivable AI feature almost. And they just released a big update that I actually haven't used yet, but I suspect that they've kind of consolidated a little bit because they had an AI at sort of literally every touchpoint in the product. So much so that I compiled them into a slide at one point. It was like, here's all the ways you can integrate AI into your existing product. Maybe a little bit much, and it's really worked for them. I mean, they've got one of those cursor-like growth curves recently. Okay, so last little stretch here. You are, as we've covered in previous episodes, concerned about the big picture AI safety. What have you seen, if anything, from latest models in the wild in terms of bad behaviour? We've got the trend, obviously, of jailbreaks are down, but these sort of higher order bad behaviours seem to be on the rise, whether you want to call those deception or scheming, or I think recently with O3, it's been termed hallucinations. But I've been trying to draw a distinction between a hallucination of the old kind where it would sort of fill in a small detail that wasn't real versus some of these, what I would call lies from O3, where it's like, that was not like a small detail. I gave you some Guidance on what kind of Airbnb I might like. And you like made shit up outright. That was actually my first experience of O3. And we've been, you know, O3 and I have been like very gradually rebuilding trust since that first, you know, loss of trust interaction. Have you seen any of that in the wild or any like odd stories to tell or anything that's got your kind of hackles up at all?

Yeah, I think mostly, yeah, I mean, well, on Trackful, I think the worst case scenario, frankly. I think things are getting more concerning, not less. So open source is delivering. So the one thing here that will not be on track for the worst case scenario is Meta is not doing well in open source. Something's happening. I don't know what, but obviously DeepSeek is crushing it and they're on the curve. So open source is delivering and DeepSeek is a Chinese company. And I think we cannot let China win this race, period. And I think they're catching up. That's number one. Number 2 or 3, lying through its teeth. It's insane how much that model likes to lie. It'll tell you things like, sometimes you talk to it and you're like, it says something incredible. And then you're like, do you have a source about this? It's like, oh yeah, check out this paper. And then you're like, no way, the paper is not— this is not at all what the paper says. And then it's like, oh yeah, look, I must confess, I heard it in a conversation in the corridor of this seminar. And like, what are you talking about? So that's another cause for concern. It's just lying. Like a lot, which is weird. The sycophantic debacle in GPT-4o, I think, was really bad. Look, if there is one cause for hope throughout it all, it's that we are doing really good progress on interpretability. I think the work that Anthropic is doing here is really good, but they're not the only ones doing really good work here. So that's good. But no, overall I remain very concerned.

Are you seeing instances like object level in the Lindy platform? Like, are users coming to you and saying, hey, I selected smartest and that meant O3, and now I got crazy shit?

No, not yet. Knock on wood.

What do you make of that? I mean, I kind of expected that answer.

That's a good question. I will say that is one thing that makes me update my priors a little bit. If you told me in 2019 or 2020, if you'd given me access to a computer and that computer has Gemini 2.5 Pro on it or Claude 3.7 Sonnet or Claude 3, and that's all I can do, it's like a glimpse into 2025, and then you'd ask me what's going to happen in the world while these models exist, I would have predicted all hell to break loose and I would have been wrong. So I don't know. I don't know what's going on. I don't know if it's just like a case of slow diffusion of innovation. I suspect that's what it is. It just takes a little bit longer for people to really exploit these systems. Or I don't know if there is something deeper about the world that I'm missing here.

Yeah, I'm kind of confused by that. I mean, the most flagrant example I've seen from real life was when Sakana Published their CUDA engineer and then came back a couple of days later and was like, we got reward hacked, basically. I mean, that was a pretty notable one from a company that can do some serious work.

I think the concerning thing is lots of the doomer concerns are based on peculiar ideas of reinforcement learning. Reinforcement learning really likes to reward hack. If there is an easier way for it to get to its reward, Even if it's basically cheating, it doesn't care about cheating. It doesn't understand the concept of cheating. It just wants the reward. And so that's why a lot of doomers were concerned about the monomaniacal properties of the systems and so forth. And that at first did not happen because at first it was just supervised fine-tuning and all of that. And now actually more and more of these models were back in reinforcement learning. And now all the researchers at this Frontier Labs talk about and think about is how to scale reinforcement learning for reasoning large language models. So that is what obviously is giving rise to the reasoning abilities of the O-class of models like O1 and O3. And even Claude, much of the improvements in the latest few generations of Claude is because they have beefed up their reinforcement learning part of their training pipeline, in particular for code. Claude is really good for code and so is Gemini 2.5 Pro. It is because they have a Part in their training pipeline that is dedicated to reinforcement learning for code. Now, if you look at what's happening with Claude 3.7 Sonnet, you can actually see the reward hacking. You can actually see it like, hey, can you please fix this unit test that's failing for me? And it's like, yes, no problem, assert true, which is basically like, just like removes the unit test, right? Or like, or actually removes the unit test. Or like, hey, the code doesn't transpile, the TypeScript doesn't pass because there's a type issue. It's like, oh, no problem, type any. So basically removes the types. It's like, hey, this is not what I asked for you to do. I've seen it many times myself. It's like, hey, I'm vibe coding and I'm like, hey, there was an issue with this component. It's like, no problem. And then just removes the component. So it's for wallhacking, just plain and simple. So again, I think that should not just A couple of points in the direction of like, is the doomier concerns, or at least somewhat warranted?

Do you have a point of view on how close we are to needing things like proof of personhood and various other kind of schemes to sort of say, whose agent is this?

Yeah, I think we're pretty close. I actually think there's like a big business opportunity. I was having dinner with a friend of mine a couple of days ago. And he had this business idea, which, like, I'm not betraying his trust. Like, he's got his hands full. Like, I think he would be glad for someone else to do it. But there's like a very big opportunity here for— he wanted to build like a USB stick that would be like a YubiKey, you know, and it would have like a mic in it. It would have like a camera, it would have like a fingerprint reader. And then so like it would allow you to jump on a Zoom call. And on the receiving end of the Zoom call, you would also need a piece of software. And what this would do is it would correlate the actual sound waves that are captured both by your computer's microphone and the microphone into that USB stick. And it would correlate a bunch of these things to be like, hey, most likely, it's not going to be fully foolproof, right? But most likely this is a human on the other side of the line. Man, I think if you did that, you could sell it to a bank or a massive airline. There's a lot of people here that just really care about identity verification. And you could probably grow into a pretty sizable revenue pretty quickly.

What, if any, questions are burning in your mind around agent dynamics? By which I mean, I think everybody has the first simplest mental model is like, the world is the world. I'll deploy an agent here and then I'll be efficient and it'll be great. Obviously, if everybody's doing that and we start to have agents negotiating with agents, or my agent talks to your agent, what have you, Negotiations between agents. Seems like that's a very dynamic system that we don't have great models for. I recently did an episode on the study of Claude learning to cooperate and kind of pay it forward to itself. Flip side of that, of course, would be if it starts to collude with itself. But do you have any— if you could put out a request for research or sort of the biggest questions that you have about what the giga-agent future might look like? What are the big questions you'd like to see answered?

I mean, the question that is most top of mind for me because of the nature of what we're working on is the relative importance of the scaffolding and the model over time. Because what we're doing is we're building the scaffolding. And so is the scaffolding going to grow in importance or is it going to shrink in importance is one of the top questions on my mind. So far it seems to me like it's going to grow in importance, at least in absolute importance, because models and AI are going to become more and more absolutely important. In relative importance, I'm not sure yet. I'm making my mind on this. Yeah.

That's a good transition to what I think is my last question, which is, what does Lindy look like in an AGI or an early ASI world, if you can extrapolate that far into the future. Somebody might say, well, a superintelligence, what does it need scaffolding for? And you're like, as you said earlier, very AGI-pilled. So I'm sure this is something you're thinking about pretty actively every day. Yeah. Are we in wait-and-see mode, or do you have a vision for how You can sort of be a channel by which people access AI that might be legitimately more capable than they are.

Yeah, no, we definitely think of it all the time. I think it's basically going to be— I think the drop-in replacement human worker is coming, but I think that speaks more almost to the user interface than it does to the underlying paradigm. I do think AIs are going to have voices. I think they're going to have perhaps faces, and you're going to be able to just talk to them and ask them to do stuff. And then they're going to be able to do it very reliably. Now, that doesn't mean that the underlying paradigm is just like an end-to-end agent and just like one very big model with a very big prompt. I think in order to be convinced that that happens, I think the 2 areas of research that I really watch closely are new attention systems, in particular attention systems that may cause attention to be way, way, way cheaper, and in particular to not be polynomial. And I think everything I've seen so far resembles more a hack than a fundamental innovation that really causes attention to be much cheaper. So that's the first thing. If you can get infinite attention, then you truly get immediately a drop in human worker. 2, dynamic compute. So models that will decide at inference time which of their weights to activate. So gone may be the days of you have all of these different classes and sizes of models. Maybe you just have one very big model and you can pass it a parameter. of how smart you want it to be, or maybe decides how smart it needs to be depending on the task. So that's also coming. There's a lot of activity in that research area. If both of these things happen maximally well, I think there is a stronger case to be made for the end-to-end agent versus the scaffolding agent. Even then, I still think that there might be room for scaffolding for other reasons. I think scaffolding is always going to buy you something.

Yeah.

It's always going to buy you some extra reliability, some extra speed. It's going to buy you some things. But if these things don't happen, I am then very bullish on the value of the scaffolding. So yeah, in this world, I'm thinking of it as you chat with your AI employee and then something happens. And I think, again, much of that is going to depend on the LLM and the model paradigm that we're running on at that moment. And that's the end of that black box. And again, we have a lot of ideas of how that black box is going to work. But at the end of that black box, what you want to see happen happens.

It's almost like when the unhovellings become the hovellings again, when the model becomes more capable, maybe no longer needs the scaffolding. Instead, the humans need the guardrails, and maybe the scaffolding sort of does future duty as guardrail when it's more about limiting what the model can do as opposed to maximizing what it can do. It's an interesting paradigm. All right, we're out of time. Anything else you want to share before we break?

No, this was great. Yeah.

Thanks for doing it. Flo Crivello, CEO of Lindy. Thanks for being part of the Cognitive Revolution.

Thank you so much, Nathan.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.