I'd Hire an AI Agent Over a Junior Engineer | John Berryman, Github Engineer

90 or 95% of AI projects don't make it to production. Demos work, productions fail. You've been building production AI since Copilot. Where are we actually in this adoption curve?

Big companies, the board says you need some AI, so they go into it halfhearted and they don't have the core competency. When it fails, they've got enough budget to absorb and write it off. The startup can evaluate on their customers, which is the true evaluation. My experience with it has moved from being just assistant to morphing into a manager and architect watching a really smart, fast developer and making sure they stay in line. I am the architect, I'm the product, and I am the manager of naive but much smarter developers. To become a mature person at any thought worker position, you need to stump your toes into the furniture, figuring out the lay of the land and making all the mistakes. That's how you become a good architect. And these agents don't stump their toes.

What does this mean in the future? Do you think human capability is going to go down?

I think we're all doomed.

This is the end of this conversation today. Welcome back to 2SetAI. Today I'm talking John Berryman, early engineer on GitHub Copilot and O'Reilly author of Prompt Engineering for LLMs. Do you have the book with you? It's behind you.

I have, yeah, I have the picture on the wall. I've got both of them, I actually, there you are. There's what I've written.

Yeah. And now founder of Arcturus Labs. John, thanks for joining.

Yeah, thanks for letting me be here.

Let me ask you the tough question first.

Okay.

Okay. The official tough question is, I think we've all seen the stats for like 90 or 95% of AI projects don't make it to production, right? Demos work, productions fail. I mean, you've been building production AI since Copilot. So where are— where do you think are we actually in this adoption curve?

It's an interesting question. Big companies, the board says you need some AI, so they kind of go into it halfhearted. And they don't really necessarily have the core competency themselves. They're like just racing to put something together. When it fails, they've got enough budget to absorb the failure and they just like write it off for another day. The smaller companies, uh, the startups especially, uh, it's easier for them in some ways because they can effectively evaluate on their customers, which is the true evaluation. They have it, they have easier customers if something goes terribly wrong. It's not a huge embarrassment like it is at a big company. Uh, so this makes it easier for them to keep going. But of course, all small companies, you know, only some of them last. And as a whole, we're trying to figure out what this technology does. And every, you know, every half year we're finding out some big mind-blowing thing that trivializes what you could have done a year before, right? So I think that's kind of the struggle. That's part of the struggle that we're having is just everything is getting so amazing quickly that it's hard to keep up and figure out which product is worth even keeping.

Yeah, that's so true. It's hard to tell. For the big companies, they have access to the customers too, right? They could do their evals as well.

A big company has an incentive not to absolutely embarrass themselves.

Oh, it's for the optics.

Yeah, absolutely. Like, uh, everyone remembers when Microsoft made the neo-Nazi agent like 10 years ago. That's kind of like gone down in infamy. No one, like, no, no company wants to do that again. So when, uh, so in order for them to productionize stuff, they need to get like really rock hard evals. And evals are actually a hard thing to do. So that, that probably is a limiting factor, uh, for, for some companies, I'm sure.

Most people treat AI like magic black box, right? I mean, most people's experience is from ChatGPT or Claude, the chatting interface, and you put in a prompt, you say something and something comes out, and if it doesn't work, you just try again, right? So how do you work with AI?

What is my relationship with AI? Yeah, it's evolving, kind of like what I said, like we're discovering how this new creature can be used. When I was working at GitHub, we developed the first, like, code completion, you know. It was, it was one of the early successful products in this kind of genre. And my, my experience with AI at that point was, oh cool, it can save me, you know, it can save me 10 seconds of work generally. Sometimes I'll hit something that would have been searching on Stack Overflow and it saves me maybe an hour because that, that can sometimes be annoying, but as the technology's gotten smarter, as we have figured out how to do more interesting things with it, then we're seeing like, you know, Claude Code and, and, uh, Cursor and things where I can say, make me this big generic thing and it will go off and do something. So my experience with it has moved from being just assisted as a developer to kind of morphing into, um, manager and architect. Watching a really smart, fast developer and making sure they stay in line. So that, that's how kind of like development work goes with me. Now, as far as consulting work, that's been interesting too, because not only am I using AI to build the things, the things use AI themselves to do things. So early customers were like, my earliest customer was code completion. Figures, right? Some of the earlier stuff was like extracting structured data from like very unstructured text. But we're seeing that things are becoming more complex and in some ways simpler to make. Like when I was doing the structured data extraction workflows were very much, you know, a human has to think through every piece of this and it's always going to have the same very fixed, we got to put the agent on rails because it does crazy things. So, you know, every step along the way is a very, very tightly prescribed set of instructions that you have to like double and triple check. But the agency is becoming a lot more flexible, more dependable. It's certainly not perfect. But you can do, you can do things like say, hey, dear agent, here's just a big set of tools and here's a really well-defined set of instructions about what I want you to do. When you get this information and it's easier to build very, very unusual things that were just absolutely not possible 5 years ago. No chance. And the amount of time that we can let the agent kind of go and do their own thing. A recent client had a really cool application that they were working on and I was fortunate to help them kind of build up their foundation and it, it is therapist trainer application. So they're not replacing a therapist because I prefer we don't do things like that. Therapists need to be human. But he was going into places that didn't have like the educational institutions to deal with training therapists and he was making a simulation. So imagine like war zones around the world, you know, places with extreme poverty, but places that need therapists, right? And so what we had the agent do was, uh, it was a, it was a Google agent and it was able, we would place the human therapist trainee into a setting where they're speaking with a client patient, basically that's an avatar on the screen that would speak with them. There was also a coach in the room that was watching and making sure that they follow a protocol. There's a protocol on the screen that they need to adhere to. And there is an evaluator that's kind of listening to everything. And, you know, providing ideas for like how you can improve things and, you know, stay back. All this stuff was live and it's, you know, we're used to, it was like, here's an email, pull this structured information out of it. And it would like usually get it right, but sometimes not. And you have to really check. Now it's, it's like, here, we're going to put you in this immersive experience and you just kind of go with the flow, make the client say whatever it wants to. And we've got that hooked up to really cool, like vocal, you can hear it. Make the coach pop in if, if, you know, the coach is reading this document, make the evaluator. It's just amazing how we can coordinate this stuff and the primitives that are available to coordinate it is so much easier than it used to be. Like the, the, the idea of an agent is getting, I think, finally a little bit better defined at this point.

Hmm. It feels like it's, it's immersion experience for the, for the therapist, right? They're, they're like, they're in the game, but they have the coaches and evaluators.

Yeah, I mean, if you, if you were training as a therapist, imagine implementing, implementing this with real humans. That would be a feat. You'd have to have an actor for the client. You'd have to have a trained professional for the, the coach. You'd have to have probably another professional as evaluator. And it's not perfect for sure, but you know, I was building the thing. And was still quite amazed with how intelligent these systems are. And, and they're just getting more so and more so. That client, for example, one thing that they've come— got in the works, the protocol involves like drawing stuff on a whiteboard. And we decided, uh, just the, the technology is just not there. We can't quite do that yet. Uh, don't know how to do that, but like, it's always working. Do I—

what does the drawing on a whiteboard mean?

Well, so there's different protocols that we might walk the therapist through during training. But in this family protocols, an important thing is just to make sure there is an understanding of the ideas. So the therapist will like draw a triangle and say, you know, these are thoughts and feelings and actions. And you know what, this situation happens, what is the first thought that comes to your mind? Write it on the board. And of course you can't have an AI do that, except about a month into it, we're like, oh wait, this technology is really close. And we were looking at, uh, Nano Banana. And by the end of my, like, 3-month tenure there, uh, I, I think they decided to start adopting some of that technology because, like, in the span of 3 months, it went from basically impossible to, wow, it seems to just work. We can have this avatar drawing on a whiteboard with, with the, uh, the therapist. The, the startup I was working with, the CEO is a Johns Hopkins, uh, therapist. So very, you know, kind of up there. But her whole thing is making sure that it's evidence-based. So everything's in development at this point, but by the time it's finished, I'm confident in looking at the strength of these models that they'll be able to say, listen to the therapist. And look through the protocol and make sure that, you know, they're on the right track. And they will be able to, to, to show that the therapist is being trained to administer these protocols appropriately.

I see. Yeah. So, so I, when you, when you mention about this drawing thing, like, I, I just, I just want to say that I recently interviewed Teodraws, the CEO, and I think their canvas is amazing. And I think their Canvas could be a good candidate to try it out. Thank you. Yeah.

Yeah.

I'll, I'll definitely share a clip about, about it. I feel like this is, oh, this, anybody who's using a Canvas could use something like that. It's like, as I said, they're infra for Canvas.

Really good. Well, that's cool. I am the architect, I'm the product, and I am the manager of small team of developers that are naive, but much smarter than I ever was.

Right. Speaking of like junior engineers, right? The real human ones, not AI. And I think last time you mentioned something like they're using AI like they're wizards casting spells. And if it works, right? It's like me as an average human using the chat. Like if it works, it's great. But if it doesn't work, like, you know, sometimes it gets stuck. Yeah. How do you, how do you think about like, what's the difference between, between them and someone like you who has been building AI?

Oh, it gets tricky, doesn't it? I've seen some junior developers that kind of like we were talking about last time, they, they talk to the agents just as if they're wizards casting spells and they will be educated in a very, very different way that I was. And sometimes maybe it's good, but the, like, The world hasn't kind of shaken out how that's going to be right now, because sometimes the junior human developers are too naive. They expect it to work, and then when it doesn't work, they don't know exactly how to go through and debug stuff. There was actually an Anthropic post recently that showed if you have AI-assisted development, that you're just not as good at debugging and thinking through the problem on one level. But at the same time, the other end of the spectrum is, you know, the old codgers— look, I'm getting there— that, like, I've developed everything with my hands on the keyboard. Um, and I think going into the future, if you depend too much on that, if you over-index on that, then you're just going to miss the whole new world. So there's, there is some sort of happy medium where probably it's you talking to the agent enough to get the point across, the agent being mindful enough to say, you know, here's some ambiguities, let's work through them. And then when there's a problem, the agent being mindful enough to know how to direct you to like the problem and how it, you know, the options for resolving it rather than just mindlessly like, I'll try a different thing, I'll try a different thing. And then you just got this crazy code base that doesn't make any sense.

So, so if today you want to hire an assistant, engineer assistant. One option is AI, completely AI agents, and the other one is a junior engineer, a fresh graduate engineer assisted with AI. So which one is better today?

Oh, a pure AI agent versus a junior assisted by AI?

Right. Yeah. A junior human plus AI versus AI. Which one is better? Based on your experience?

Well, I'm afraid I'm going to say that for me, I would, I would be more comfortable with the agent, but I don't think that should be like the standard rule. For a very junior developer, if you're telling the developer who's then going to use an agent, it's just like another, another link in the chain in the telephone game.

It's like your manager and you hire someone and someone hired AI agents and then another layer for that.

This is, this is probably, this is one of the problems that the new world is going to cause us to solve. Like, a junior developer, if I tell them something, let's see, in a 10-year-old, uh, 10-year-ago world, if I tell a junior developer to solve something, then I expect them to work on it, bump into problems, but ultimately be able to get through it, uh, but in today's world, it's kind of weird because the, if I tell a junior developer to work on something, then they're telling a junior developer who is actually an AI to solve it. And there's like a double chain, right? I almost, the way we're ultimately going to solve this is our junior developers of the future are not going to really think as much in for loops and if-then statements and stuff. They're going to be architects, product managers, managers. Of the AIs and the really good juniors are the ones that are going to be able to orchestrate these things and, and have the agents accurately do their bidding. But it's still tricky at the point where they have to go through the code and like read stuff. And I haven't resolved that in my head exactly how we train that up correctly. Does that make sense?

Yeah, totally. It's a, it's a dilemma, isn't it? I never thought about that. You just, you just gave me the, the, the, the idea of like, oh, you know, this is like a reporting chain. And you, you kind of need this new manager to be really good at using AI. You have to be a good manager of AI for me to use you.

And this is like the weird paradox that is hitting all of industry right now. In, in order to become a mature person at any thought worker position, you need to have kind of the stumping your toes into the furniture, figuring out the lay of the land and making all the mistakes. That's how you become a good architect, product manager, manager. And we're finding out that these agents don't stump their toes so much. They, they know all the syntax for every language ever written. They've seen all the examples. So for the really, for the junior level stuff, they're fine. And so it's awkwardly easy for a senior to harness an AI. Right, right.

What does this mean in the future? Do you think that technology and human overall capability of developing technology is going to go down?

I think we're all doomed.

Then this is the end of this conversation today. Right.

Thank you for having me today.

Most people, I mean, the traditional way of setting up vector database, chunking, embedding, pip install, and then just turn it on, one line, and then it will work. What's broken about this approach?

I feel like we're getting through a period of time where everyone viewed RAG as like this black box thing. It's really strange. Like, okay, so I come from a search background. That was the— my first career was search, and then I went into AI stuff. And so it's always been a puzzle to me why people choose to cede over control and all the details I know are hard details about search and just leave that inside this black box called RAG. The better approach, the thing that I think is slowly starting to win out, is to not think of RAG as one atomic black box thing and instead think of it as all its constituent pieces, right? So RAG is an agent. And it's search. And that's, that's all there is at this level. But let's peel back the onion a little bit further. An agent, all an agent is, is a for loop around a large language model call with a, with a context that, that keeps growing, but that's not too hard to implement and some tools. Now, if the tool is itself a search tool, ta-da, you've got, you've got RAG. And then let's break it down just a little bit further because there's the search side. If RAG isn't working, it's not because RAG is broken. It's because, well, you know, something along the way is messed up. So I'm going to look through all of my common searches, and either this search is a weird meaning right now, because it might be a user conversation, or it might be literally users typing in. But I'm going to look at all those common searches, and by hand for, you know, the top 50 searches or whatever, I'm going to go into this RAG agent, I'm gonna look at the conversation and try to figure out where it got it wrong. Is the agent itself interpreting the user information incorrectly? Okay, well, I can fix that. It's prompt engineering, context engineering. Whenever the agent decides to make a query, is the query well-formed? Is it— does that make sense? Okay, we can fix that. Whenever the results come back, are they sensible results? So we've completely decoupled it into an agent AI problem and a search problem as distinct problems. And, you know, if it's a search problem, we can, we can tune the relevance. It doesn't have to be like just throw everything into semantic index. You can still use lexical search and there's actually a lot of really beneficial qualities for that too. Yeah, but now since everything is, you know, a pipeline of indexing and searching and, you know, the interpretation and context management, there's all these pieces that could go wrong, but now I can pinpoint them. Fix them. And that, that's why it's— I think this idea of RAG as a black box is silly.

What's the best approach for RAG today? If you were to build a RAG system from scratch right now, how would you do it?

Uh, the best approach for RAG, uh, is to build it from scratch, is to build it according to whatever your needs are. So that is hard to answer in the abstract because the needs of every place are going to be a bit different. But again, I wouldn't think about it as I'm, as just I'm building a RAG. I would think about what's the needs of, what's the information need of the user? That's what we always talked about when I was doing search relevance. It's like, you know, what is the need and how do we figure out the need? Well, is your application like a conversation? That changes how you're going to do the prompt engineering side of it to glean the pieces out. What is the user need? Are they looking for something really— let's see, how do I say this— is the user looking for something where words matching exactly is important, like they're looking for the title of a film, then I would use lexical search. Are they looking for like just generally the idea of something, then maybe I would look— I would still do vector search, semantic search. But all these questions You know, how do you build RAG search? Well, you figure out, you know, all those steps in the pipeline and think through your user's experience and figure out how to answer those to best benefit your user and the company, obviously. I've been looking into if you can wrap an agent around traditional lexical search and actually have almost a free relevance boost by just you know, wrapping an agent around lexical search. That's an interesting space that I'm investigating right now. But you can also have like agents that don't even use a search engine at all. I've explored this as well with my— I've got a roaming RAG blog post where basically you say, here's the table of contents for your book. Which chapter you want to go to? Here's, you know, the sections, which section you want. You can have agents navigate that. And when you look around, that's become really powerful. Anthropic Skills, that's a RAG application. But the form factor is, is these agents, we've got agents now that are really good at navigating file systems, reading files and managing their own context with files. So, okay, let's just make a search application that looks up all, all these skills that we have loaded, knows how to traverse the trajectory, the directories, and infers from the information it reads and the names of the files, like you know, what's going to be relevant to look at. And that's still a RAG application. It's just no index. It's just, you know, traversing a file system. Very strange, very different, but very useful in that type of situation.

I really like the, I really like the Anthropic experience now, like with the add-on of skills and also like projects. So you have, you preserve a project level memory as well, and you kind of know the cross chat thread within a project together with the skill, and I find that very, very useful. You worked in search for so many years. I mean, are you still using any of the traditional search technologies? You mentioned about lexical search.

Yeah, lexical search is kind of the oldie but goodie in my books. One of the side effects of the RAG revolution, that original RAG paper required vector search. Basically the LLM and the vector search were fine-tuned together, which is a really cool idea. And vector search has some amazing qualities to it. Like you can look for an idea as an idea because I'll say gorilla suit and it'll find documents with monkey costume because it's the same idea. Whereas traditional lexical search, it takes the text, chops it up by the words, kind of— it does some modification to the words. So like walking and walker and walked, they all kind of compact down to symbol. But you know, you type in gorilla suit, it's not going to match monkey costume. But people got so enamored with semantic search that they kind of forgot all the good bits about lexical search and they had to learn some of the pain points of semantic search. Some of the pain points of semantic search is, you know, you got all these really cool demos based on out-of-the-box, like embedding models. But if your domain is not like the generic everywhere domain, then you're really going to need to fine-tune the embeddings to be more appropriate for your domain. Otherwise, like, it's just not— you're not going to get the out-of-the-box demo with your dataset. Another thing that is, I think is confusing to me is the data structure, navigable. The data structure that they use for semantic search is really cool. The way that you can super quickly find nearby neighbors in this, you know, 2,000 or whatever dimensional space, but they don't really easily admit the notion of other filters. Like, I want to find the documents that most closely match my description, but have a price in this range, have, you know, you know, color equals red, blah, blah, blah. And that's something that lexical search is just super duper duper good at. You can have an arbitrary number of fields. It's, you know, the more fields you add, it doesn't, you know, make it slower. If anything, maybe it makes it faster. It just constrains the space down, uh, more and more tightly. And in some ways, it's a lot easier to work with them. The other kind of pain point is with semantic search, you don't have exact match. Uh, I mean, the cool part of semantic search is you don't need exact match. The bad part is you don't have access to it at all. So if I wanted to look up a specific person's name, that name is going to be mapped to an embedding space, which it doesn't have any it's not going to be mapped sensibly. It's just not going to work.

So, so is your take for today's search engines, uh, it should be a combination of these approaches, like semantic together with the traditional approach?

Is like, I perhaps to a fault, I always work from first principles. So like if your domain, your problem needs to match on ideas,, and if your business can support probably hiring, you know, a small ML team that, that knows how to do the fine-tuning stuff, then semantic search is probably going to be a good choice. If you're doing something that requires exact match, lexical, and there's no reason you can't combine them. Like, um, the— you could, you could even provide a tool that the agent understands, and you tell the agent if they're looking for an exact string match, then you, you, you know, put it in this field. And if if they're looking for an idea, but you could, why not both?

Yeah, that makes sense. I mean, we've been talking about these and how this information retrieval works in the search space. How about let's flip it around. I mean, what about the other side? This is a new challenge and is, I think, emerging challenge that everyone, every company wants to do is they want to be able to, you know, get found by AI. Right. It used to be like there's the SEO of the space and then now like there's this new term called generative engine optimization, right? They want to be, I want to get discovered on AI. What do you have any thoughts on this? Like what's your take on this new AI discoverability, AI search optimization hype?

I am completely unaware of it.

Oh, you are not aware of it? Okay. I have to explain to you. Yeah. People, because I think it's a consumer behavioral change today as well. People are searching. Like the other day I was walking on a street downtown Castro and then, you know, there was like high school or middle school kids, they were walking by and they were talking to each other. One kid said something like, oh, let's go to Walgreens. And then the other kid said, oh, let me ChatGPT what's on sale. Like, so they're doing search right there, like, you know, on ChatGPT on their phone. So I think this is a whole, like, you know, new— remember, like when you were searching for something, are you still using Google?

So the world's about to get much weirder. And this, so I'm going to take this in a different direction than I think you pointed me, but you'll see how they're related.

I want to know.

When I used to do search, I never did SEO type work. So that's why this, you know, that first part of the question, it's like, I don't know about this SEO discoverability by Google. It was always like building specific implementations for, you know, clients. But how can my website get found on ChatGPT or, you know, Gemini? I don't think my website, I don't think any website is going to exist in 5 or 10 years. I have no idea what the heck is. But I think, and you touched on it in your discussion there, the kids are like, let's see what's on sale. By looking through ChatGPT. I think in the 5 to 10 year range, everything that's like a landing page website is going to go away and in its place, that website is going to have data that is accessible by search tools and the ability to modify stuff. So at an e-commerce site, it's not a website place, it's going to have the ability to purchase things. That's like and you'll have the ability to search for things, but you won't have any UI. Instead, ChatGPT or Anthropic or anybody is going to provide a conversational interface just like the kids had, or live coding is going to be so easy for, for the trivial apps and for UIs that basically you can say, here's what I want and here's how I want you to present it to me, and your own Amazon.com that has all the facets that you want and all the stuff that you care about is going to be presented to you and you only. And so that's what's going to happen.

You are going, you know, 10 years ahead of me already, but it was, it was actually something I was thinking as well. So based on what we discussed so far, everybody's company landing page will become this huge lengthy LLM text file that nobody will look at.

Yeah, totally a possibility. Yeah.

Oh man.

Yeah. LLMs.txt and a list of tools or something. Yeah.

Basically. Do you think something will be missing there? Because, well, you know, still today's landing pages, we still have like checkout function, right? But I guess OpenAI can have checkout, so you don't need that.

Well, it's standardized. It'll be like the tools for checking out will be like custom, will, will be, standardized so the models are trained on how to do checkouts right.

And right, so what's needed on today's landing page that's gonna last into the future?

Nothing.

Is it really nothing? Reviews? Reviews?

I mean, it's still important right now, but I just think it's going away. I think you should keep doing the normal thing right now because internet authority, you know, get other web pages to refer to you, that's still going to be useful. I, so maybe this is kind of something that would be interesting. The way that we know that the internet, that the web pages are useful in internet, you know, PageRank by Google is other sites point to the pages in the future when there is no landing page and nothing to point to.

What's going to happen to report that? Like, you know, how do these huge LLMs.txt files differentiate from each other? That becomes a problem. I guess.

Yes, we'll solve it. I don't know how, but it might, it might be like the web crawler notices new things and provides that statistically some portion of the time to test if it's a good service.

Yeah, good service, good product. There has to be another scoring system other than like the PageRank, the backlinks. Maybe the, you know, the ChatGPT of the, or the, the AIs will will do that because think about they are not releasing any information around search volume. So if people are searching for like Christmas gifts, like, of course, let's say Christmas trees, and on ChatGPT, you know, ChatGPT is not revealing that information or that data to the public, but how many people are doing that and what are the service providers or product, you know, the e-commerce shops, whatever they are servicing to the users. So maybe it becomes another scoring system later when a lot of people, a lot more people actually click on it and then decide to check out from a store, then they have a higher score.

Right. All right. That's where I would go with it. I think the PageRank, all you have is the backlinks, which is kind of a signal. And then Google also looks at whenever one like clicks on a link, they can figure out if they've dwelled there a long time, and that's kind of a signal, but maybe these completely abandoned, they still don't completely know. At this point, you're going to have someone chatting to an agent that's smart enough to read the conversation and say, we nailed it with that one. Plus point for amazon.com or whatever, did a good job that time, or minus one.

Okay, so company websites will be LOM.txt and also maybe a database. So, you know, for search engines as well.

And LLMs.txt and an API.

An API for their data.

Maybe an MCP if MCPs are still a thing.

Good point. Good point. Okay. So I think we're coming to our second doomsday prediction and third one. One is landing pages are doomed and third one is Google going to die.

No, no, no, no.

Google's not. No, we have this, this conversation is turning so This totally is a—

I think Google is in a pretty good situation right now as, as a, uh, like Google search is going to change. Like already they've got their AI is very prominent in their search. And so it's going to become, it's still going to be search, but like I go to ChatGPT or Google expecting to interact with the AI, but before I actually expect to interact with like, you know, the 10 blue links. That's going to go away. But as far as a company, Google, they were sleeping on this AI revolution a little bit. But remember, they came up with a technology, kind of their labs, or, you know, Attention Is All You Need that came out of Google. They've got some incredible minds there. And when, when the internet is complaining, oh no, we've run out of alt text to train on, guess who's got all the videos? All the appropriate videos. But guess who's got all the, all the videos that you could train on? That's YouTube. It's Google. Yeah, yeah, they're in the right position if they just keep, keep nailing it. Yeah, they need in their world models. Have you seen their world models?

Uh, not yet.

Uh, okay, we've got text models, we got image models, we got video models. World models is really weird because it will, in, in near real time, they're not super fast yet, but in near real time, it'll generate images of a video of you walking through a world. And they're thinking that that's going to replace like video games. I think we're a bit away for that, but things are accelerating, so who knows.

Yeah, I don't know, I don't know. I feel like I'm talking with so many interesting people and I'm learning, I'm trying to connect the dots. I recently interviewed, uh, CEO of a company called Prismax, and they're the visual production company behind like the world's largest musical festivals like Tomorrowland in Belgium. So, so yeah, they also have shows at Sphere in Las Vegas. So he told me about like the— what's the— what's that immersion experience is like with the— like when you're creating such experience. And he's Well, you know, spoiler alert, he said that the current status of AI is not meeting the requirements, so it's not good enough yet, but we'll see.

It will. It'll get there.

Yeah, I don't know. Yeah, we'll see. That's another interesting aspect. Can you tell me, can you share a little bit? Because we talked about this AI search thing anyway, so we still have to face today's, you know, world where we do have companies do use landing pages. Can you share your thoughts on how AI is doing the search? Like when people type in ChatGPT or Claude searching for something, can you break down the steps? How does AI answer the question as, you know, and do the search?

It's a RAG application.

Tell me about it.

Yeah. Pip install LangChain.

Is that what they did? Guy shamelessly using LangChain.

I mean, but there's not, there's not too much to it. They, Google has the ability to go from a keyword search to the whole web and probably, well, I know they have other parameters that you can put in there to limit it to like only the site, date range of stuff that is created, but it's just a search and it's, it's a tool. And so if you tell any model that you've got this tool and it has the power of Google and OpenAI has made their own version of this and And, you know, everybody has their own version of this. They've indexed the internet, but it's just a search tool. And you say, go find the most relevant thing. And then once you find it, you paste it into the context, at least the relevant pieces. And you say, do you know, learn from that, tell a user whatever they were asking about. And it's, it's a lot easier than traditional search. In traditional search, this is my struggle when I was a search consultant, you got this text box. And you're supposed to infer from, you know, the 5 words tops that someone types in there, everything that's in their head. You can't. But if it's a conversation, then you've got the full context of them talking about what they want. You show them an answer and the context continues where they say, that's not quite what I'm thinking of. That's, that's huge. You can, you can glean so much better information and make more targeted searches, get exactly what the user wants. Hmm.

Good point. So if the user, you know, type in a more concrete question, a more concrete query into the chat interface, and then OpenAI, let's say, they will try to understand what is the intent of the query. And then essentially, maybe because if they use, let's say, Google search engine behind the scenes, the next step, they probably will somehow turn that into a keyword search and then use a Google SERP, you know, to do the search. Let's say narrow down because the point of narrowing down, right? So they still need a search engine to narrow the information down to, let's say, instead of the whole entire web, they're narrowing down to 50 answers, maybe top 50, and then they're putting everything in the context window and then synthesizing the answer. That's our guess, right?

It's what they're doing. It's, it's not complicated. I mean, I, we're talking about a simple version of it, but I don't think it gets that much more complicated. Probably if I was them, probably they are fine-tuning to make sure, like they can do reinforcement learning to try different search strategies and they can learn their own index and how it works better so that they just hit the right query to begin with., but I mean, it's still at the end of the day, it's a search tool. It's a large language model in a loop, small pieces wound together by traditional, like software. It's yeah, that's the cool thing about all this. It's like, it's amazing what you can do. But if this is like 5 years ago, if you wanted to do anything like with ML and AI, then you needed to hire a bunch of PhDs. Now you realize that AI is just an HTTP request. Text goes in, text come out. It's kind of magic, whatever happens in the middle, but you don't have to worry about that. And you can— it's just like you got a weird, smart, yet not quite all there human in a box. Strange.

Man, the fourth prediction, the PhDs are dead as well. The PhD titles.

Is that a dystopian topic?

I know, I know, just kidding. Okay, so, so I mean, with all these experiences, you've, you've done Copilot, you've worked on RAG and search applications. Can you share what are you working on now?

Oh, sure. How much can I share though?

Whatever you can share. I think you already shared a project.

Well, I shared one project that was, that was a previous project with the Avatar Service Trainer. That was a really neat one. I'm taking a stab right now at model fine-tuning. We've got a really interesting application where basically we can look at the traces out of a model, out of conversation, and we can train smaller models to mimic what we're seeing in those traces for this tight domain. So that seems plausible that you take a smaller agent and, and do something that the bigger agent was doing well. And we can also do a type of fine-tuning, reinforcement learning fine-tuning, to look at how that trace worked out simulate other possibilities that we would try to do and over time get better. Ideally have a small, small model that's doing significantly better than the original expensive model in, you know, this fairly tight domain. So really cool product idea. And it's, it's been fun to get my, my hands dirty with all the Unsloth and OpenPipe and stuff like that. It's a neat domain.

Is this for— you said this is for a domain use case?

This is just another product that this company is working on.

Okay, okay, okay, cool. How do you feel? So you're— this is one of your consulting projects, right? Okay, cool. So you're now— when did you leave Copilot? Sorry, GitHub?

Uh, May of 2024. So it's been It's been about 2 years I've been out of the cage.

Almost out of the cage. Congratulations.

Yeah.

How do you, how do you feel? How do you feel? What's the difference between like building for a company, building AI, AI, you know, applications for a company versus building your own company?

It's interesting, right? I mean, there's a lot of comfort in being in a company working on a project, knowing what the next day is going to be like. But at the same time, it's a little bit risky. The most comfortable, risk-free thing is actually risky in the long run because we don't know what, what, how the world is going to change with this AI. And if you are inside of a company working on like just one fixed project, then, then I at least was missing my education of like, well, what's happening in this subfield, in this subfield, this subfield. And it was weird becoming a consultant. I, I got out of GitHub. I had written a book with another, uh, with one of the research founders of Copilot. And GitHub had really taught me a lot about the fundamental understanding of this new technology. We had built everything around it because nothing existed at the time. We were one of the first, right? Yay. But I, because of that, I'd missed out on LangChain, for better or for worse, and all these other technologies that were you know, competing for a new understanding, all the different paradigms that people were trying to wrap around this technology. And that was— that, that wasn't great. I had spectacular fundamentals, but I'd missed some of the, the kind of neat things that had happened in the ecosystem. And when I jumped out as a consultant, it was the fire hose all over again, which is good. I like that. But I got to learn about all these different technologies, some of them have gone by the wayside. Like, you know, as we've talked about, like the world is figuring out what these agents are really good at and what, you know, how they should be shaped. But yeah, being a consultant's been a lot of fun. Every month or two is a completely new world. There's not one project I've done that has any similarity yet in the past year, year and a half really where I've been going, but doesn't have any similarity with any other project that I've done. That's just really spectacular.

Mm-hmm.

Yeah. So, but it's, it's got its ups and downs too. Like, you know, there'll be periods, feast or famine. There's sometimes where I'm so busy that it's annoying and there's sometimes where I'm not busy and I get nervous, but that's just, that's just part of this field.

So what was the hard, what's been the hardest thing in building this?

Building the consultancy or?

Yeah. Building your business, your own business.

Hmm. The hardest thing is something that I'm still not good at yet. And that's, that's marketing. You're helping me, actually.

I am. I am.

Thank you for this platform.

Anyone who needs help on, like, I'm looking for AI solution, reach out to John.

Thank you for sponsoring Arcturus Labs today. That's Arcturus Labs, by the way. ArcturusLabs.com. But yeah, I'm good with, I think I'm good with people. And so coming into podcasts and doing things like this is a lot of fun for me. And it gets my name out there, you know, different sources. People get to hear from me from different sources, but I still don't really have like a good fundamental understanding for like marketing and exactly— I don't do it scientifically. I do it with a shotgun.

So you think there must be some scientific way of doing this?

That's what I think. And you're going to help, right? That's—

I will. I will. I'm actually, uh, I'm actually interviewing like some marketing experts as well. So I'll share like if there are insights, I'll definitely share on LinkedIn. And I'll share with you personally as well. Like, I'm learning too. Like, I feel the same way as you. Like, this is a challenge for all technical founders. I have no clue how to have a steady pipeline versus, like, like agents. I know the agents working, right? Versus my marketing pipeline, my sales pipeline is so unpredictable.

And since everything is shifting ground, like, uh, I wanna— I like writing blog posts. I do it Very slowly, unfortunately. But you know, that used to be a really great way. I think probably it's still a pretty decent way, but now we're competing with AI slop everywhere. Like the volume of stuff that could be read is huge. And then, and then maybe I can augment myself by making, you know, I'll just like talk to my computer and make it write a blog post that sounds like me. Should I do that? Yeah.

Yeah. You know what? This is organic at least. So this is real. And you're not talking to an AI, I'm not talking to an AI. So this is the real John Berryman, this is the real Angelina Yang. So this is the best. Anyway, for the near term, how do you think people's career paths are going to change?

I think we are at the fork in the road between utopia and dystopia. I think if you're close to the end of the career, you might be lucky and be able to ride it out. If you're in the middle part of your career, you better be adapting pretty quick so you don't end up in trouble, you know, 10 years left to work and not, not employable. I think it's important to understand how these AIs work and can augment you because you're going to be competing with people that are augmented by AI. It's just a tool. And there's people that are fretting about like, well, it's going to take away my job. And you like developers saying this, or like the creatives are saying this, but there is a group within those fields that are thinking ahead. They're thinking, I'm not going to be able to do the things that people were taught to do in my field for so long, but there's so many things, more things I can do. And if you have that mindset, the world's your oyster. Then you can, you can do whatever you want to. Love startups. Uh, that's why when I'm consulting, I try to, to pull myself close to startups. They got fun ideas. But I would not build a startup today that looked anything like a startup from 2022 when this all kicked off, because that's the infrastructure, the interfaces, everything is something that you could almost vibe code in like a weekend for those things. But thinking ahead, you got to think about, well, what, what are people going to, going to be doing? How's this technology going to change things? I want to build that, something that is, was utterly unimaginable and unreachable in 2022. Hyper-personalization is going to be the way of the future, but I still think there's going to be really interesting things out there. They're just kind of, they feel like they're in the meta space right now. They're something that probably wields AIs better. That's— I think there's still interesting problems, but they're the abstraction level above what our minds are used to. You'll see it.

We'll see, we'll see. And, and I mean, last question, you have kids, right? What, what are you telling them they should grow up to do now, and how would you teach them how to farm?

The ho-ho pops. I actually do have a farm in West Tennessee. My grandmother had a nice farm. So if the dystopia side wins out, then I still, we've got a tractor. I'll have to find something.

You have a fallback.

Yeah, we have a fallback. No, I, what do I tell my kids? I have an 8-year-old and an 11-year-old, and I'm trying to practice what I'm preaching. When I became a consultant, I wanted them to see me visibly struggling and failing and being confused and getting up and trying again and succeeding. And just, you know, and they've seen it all, and I've made it— made them very aware of it because I want them to understand it's okay to try, struggle, fail, and just once you keep going, it— you won't fail for forever. And failure is not— failure is not really a thing like you thought it was. And then the other thing is just like with the technology and stuff, I make sure that they're very aware of it. My whole family is. My wife actually vibe-coded a Revolutionary War education game. We homeschool our kids. So we're weird. And my son, I've got him set up vibe coding. He's got a website, which is his arcade. And basically all it is is I've got cursor set up to say, hey, you're working with an 8-year-old boy. He's going to want to do nothing but make games, use a, you know, one index.html, and whenever he says that's good, then publish it to GitHub so that it's served up on the internet. It's GitHub Pages, so you can go to his website. It automatically publishes whenever he's happy with one of his games. He's got like 100.

Yep. Wow, I need to teach my son to, to like try it out as well. Like, you know, you gotta be more custom to, to these tools that's totally different from how we grew up. Thank you so much, Sean. You can always come back to share something. Thanks.

This has been a lot of fun. Nice chatting with you.