SolveIt: The Thinking Developer's Environment w/Jeremy Howard & Johno Whitaker

Welcome everyone. We are here with the legends, Jeremy Howard and John Ovidiker. This is a highly anticipated talk. I think everyone by this point is pretty turned on to the idea of notebooks and developing in a certain way, but I don't think people have seen the next level that you're going to show today that goes beyond what even people's imaginations. We can go ahead and kick it off.

All right. Hi there, gang. I'm Jeremy and we've got Johno here as well. We're from Answer AI. You might know of me from fast.ai. fast.ai is part of Answer AI now. So it's kind of one and the same thing. And yeah, we've got something to show you today, which is, I won't say the culmination of, but the current state of well over 10 years of work. But the particular thing we're going to show you, we've been working on full-time for the last year. I've shared a couple of things on the Discord. Maybe I'll just share my screen so we can talk through them together. Because I think the main thing I want to talk to you about today is more about a technique and a mindset than it is a software platform. But we've built a software platform to try to make that mindset the natural way to work. So our co-founder Eric, who many of you would know from The Lean Startup or from The Long-Term Stock Exchange, he's been writing his new book with SolveIT a lot, like not to draft the words, but to do fact-checking and to check for issues and to help with segues and stuff like that. And yeah, he pointed out that what we call the Solve-It method, which is what we're going to describe today and hopefully demonstrate today, is very much about working in smaller batches. As you know, if you're coding a line or two at a time, or if you're writing a paragraph at a time, And so this is by no means an invention of, of us. In fact, something that became very clear as we did research for this course and methodology with Eric is that what we've done is we've kind of reinvented something that many people have invented before, which we've known for a long time that this approach of doing things in small batches lets people and organizations operate more efficiently. Interestingly, we've now discovered by doing this a lot over the last year that it also makes the LLMs much more intelligent and works with us much better, which has been a fascinating discovery. So as Eric points out, vibe coding, you know, gets it wrong in both ways. It doesn't let us take advantage of the smaller batches thing that people in organizations do well and doesn't allow the LLM to take advantage of these small batches either. However, you know, warning, as Eric points out, doing it the Solve-It way feels like a lot more work. So for his writing, it's going slowly paragraph by paragraph. At any point, you can just tell an AI, do it for me. And it feels, you know, you get this dopamine hit. You get this rush, it feels great. So that's why I've written some things to think about today, which is like, if you're working on some code or solving some problem, why are you doing it, right? What in your ideal world, what, what would you get out of this process? What are your personal long-term goals? You know, so you maybe you want your skills to improve, right? And that's going to require study and deliberate practice. Maybe you want to build something meaningful over a longer period of time. I sure hope these are your goals. If they're not, then maybe this isn't for you. Yeah, so for me, I want most of my work to involve creating something substantive. So the more I work on a project, I want that work to be getting easier and easier the more I work on a project. I want to see myself going faster and faster the more I work on a project. I want to see tasks getting easier and easier. So that means that I need to be building up this, what I describe as a concise, thoughtful, well-designed system that I and the people I'm working with fully understand. And this is the opposite of what I often see in software development in general and in AI-helped software development in particular, is things tend to get getting more and more clunky and people find them kind of saying like, I'm getting less and less confident personally. So like, even if I'm doing a small one-off script, you know, where VibeCoding can totally just do it for me. Yeah, you know, you can do that. But I'm almost, almost always wanting to build into something bigger eventually, or to learn something. From the process, right? So I don't, even for that, I don't kind of do vibe coding very much, you know. If it's non-trivial to write the thing, me or the AI, then I'm going to want to make something that makes it more trivial next time. That might seem weird when you're just getting the AI to do it for you anyway, potentially. But the thing is that, you know, think about, for example, you wanted to create a browser-based 3D game. And you didn't have access to 3D.js, the brand, the LLM didn't. LLMs can't create anything useful without access to something like 3D.js. Like the AI is only as powerful as the foundations you give it as well, just like people are. So the more and more powerful, strong foundations you create, the more stuff you can build other people can't build. So that as we build these, you know, foundations, then either me or an AI can build on that to create something more sophisticated, right? And then you get this amazing feedback loop where you repeat it, you get to build deeper and deeper stuff. So the stuff we're going to show you today and what we call the Solve-IT platform, the entire thing is basically written top to bottom in Answer AI technologies. And we've built these technologies on top of other Answer AI technologies that are built on top of other Answer AI technologies. And the more we do it, the faster we're getting. Which is, that's where you want to be, you know, both as a human being and as an organization. And, you know, something Hamel Husain and I've talked about a lot over the past few years is that I'm always investing in my long term. You know, I spend at least half an hour a day doing deliberate practice of something that I'm not good at yet or learning something new. I have done for like 30 years. So when I'm working, that's how I'm thinking, is investing in myself as well. So yeah, overall, I guess there's this idea that Eric Reese talks about a lot as well, which is, you know, trying to— rather than incurring increasing debt, How do you work in a way that your work gets better and better, you know, whether as for an organization or for a human? And so that's important to understand that there is no methodology of software engineering. So software engineering is a study of creating non-trivial systems that can continually improve over time. There is no methodology that works with bad coding, you know, we've got decades of study of this that, you know, we need to think about how can we create things that continuously improve over time. So yeah, we want to talk today a lot about the dangers of focusing on short-term gains at the expense of long-term health. And it's really hard because it always looks so attractive to just get an AI to do something for you. But we're not seeing it pay off. And you can see that because, so for example, actually Eric Ries is a good example of this because Eric Ries is kind of the most, you know, one of the most or the most well-known startup guy in the world. Every startup ends up, you know, every interesting startup ends up on his desk early on. Gets asked for advice about it or help finding investors for it, whatever, he has not seen any which have been built, you know, through vibe coding. So, you know, it's been around for well over a year at this point. So if people were creating huge, amazing systems and dramatically improving their capabilities using vibe coding, we would see it and we haven't seen it, right? So it's, it's a lot of smoke and mirrors there, and it is because of this dopamine hit we get when we pull the lever, you know, the Claude Code lever, and hopefully some great code comes out. So we have what we're describing as an antidote to this problem, which is a system where rather than the computer being the agent, it's agentic in that you're the agent, you're the human of the agent, and the whole system is designed to make you more competent every day that you use it. This is also great for your mental health, right, because you feel and see yourself improving every hour. Rather than getting more and more buried, you know, by mountains of code, more and more concepts you don't understand, you know, knowing that so far that approach has not been successful in building sustainable businesses. So that's a key insight, you know, and we've tried that a lot at Answer AI. We've invested heavily in really pushing agentic AI to the limit, studying everybody else who's doing it, using every technology, building our own. And we've discovered some small limited areas where that approach can be useful. But overall, we've discovered that many of us have felt like, wow, this is going to change the way I work, this is going to change the world, maybe I'm not going to be needed anymore, and then discovered over the months of like, okay, that was, that was imaginary. That's not what happened. I haven't actually sped up. I haven't actually created 1,000 times more things than I did last year. So that's the background. Now we started working on this over a year ago and we did it, you know, partly because we were worried about the direction things were going, but also just because we were excited about the opportunity to take something that we had already been working on for years. Which is the idea of highly interactive, highly exploratory approaches to, to building stuff with code. So these two things have basically come together. So the platform we built, we call Solve-IT, which is based on a book by a guy called Polya called How to Solve It. Now Polya's book is from, I think, from the 1940s. It's an old classic math book about how to solve difficult problems, how to solve problems that you couldn't solve normally, you know, by creating better processes. And interestingly, the process that he developed is basically the same process as what's known in dogfighting as the OODA loop, O-O-D-A. It's the same as the methodology that Toyota developed for rapidly iterating on their assembly lines, and it's the same as the approach we independently developed for exploratory coding approaches. It's interesting, you know, it's also very much the same as Eric Ries' Lean Startup approach. So we've all kind of ended up at the same place, which is creating this loop that we do, where you start out by, you know, really thinking carefully. And AI can help you with each of these steps too, right? But you've got to be, you've got to be human first, right? So understand the problem, read it carefully. So one thing to think about here, if For anybody here who hasn't studied and isn't familiar with deliberate practice, I would start there, right? Because deliberate practice is the thing that like lots and lots of performance specialists, people who are expert sports coaches, music coaches, so forth, have studied this and have identified that deliberate practice is the most important thing to becoming better at something. Deliberate practice means working on the stuff that's hard and doing it the hard way, like working hard at it, failing lots. So you can't expect to feel like you're in zen, you can't expect to think like things are easy, right? If you're feeling like that, then you're not in deliberate practice mode. So, you know, you can't ask the AI to read the problem for you and tell it what it says, right? Read it, then you restate it and then you can say to the AI, hey, does that look right? Did I miss anything? And it might say, oh yeah, Jeremy, you did actually. Did you not notice it said that this is line by line rather than paragraph by paragraph or something? Oh wow, okay, I'll do it again. Right? So, you know, what do you know? What needs to be known? Again, write it down. And this is all stuff you can do, as you'll see, in Solve-IT. To help you. So when you understand the problem, you can then think about multiple strategies. And again, after you've thought of multiple strategies, you can ask the AI, are there more? You know, can you think of any additional ones? You know, try and come up with analogies to other problems you've solved. Now, just talking about problem solving, you might think, oh, that doesn't— that's not me, I just do CRUD UI things. Or I just do, you know, landing pages, you know, put whatever, like whatever you're doing, you are kind of, you're solving a problem, right? You're trying to get something done that didn't exist before. And if you're going to do it more than once, ideally you'd like to come up with a way of doing it faster and faster every time you do it, better and better every time you do it. So that's the problem you're solving, you know. So then, you know, you implement your strategy systematically and you be thorough and patient. And again, this is so different to the kind of let's use Claude Code approach, right? Just carefully execute each step without rushing, like taking your time. It might be like 10 times slower than if you just got Claude Code to do it for you, but maybe it's like 10 minutes versus 100 minutes. If you think about how much you get out of the 100 minutes of carefully executing each step without rushing, you'll realize that when you do that day after day, you build up this incredibly powerful set of skills, of libraries, of code. SolveIT will very naturally document each part of the process, as you'll see, makes it easier. And then this is the piece which a lot of people skip, which makes it the loop, look back, right? And again, you can have SolveIT help you with this. Jono, before I just jump into a quick show, is there anything I should add or anything you want to add about the kind of background here?

Yeah, I think what you're saying is really important. The most common failure mode that students fall into in our course is actually trying to go way too far way too fast with AI, and they try to like throw all their traces at an LLM. They say like, hey, why are you teaching us this? Why can't we just use an LLM? Just like, just like let the LLM tell us what's wrong with their application. And we constantly have to say like, no, that's, you know, a recipe for disaster. So it doesn't— it applies on everything, like not just coding and stuff like that where you might show today. It's like any place where you might use AI. This mindset that you describe is super important.

John, do you have anything?

Uh, not much to add at this stage, I guess, other than like we did— we kind of framed it as the loop is like this one pattern. Um, I tend to think of it as also applying just all the way down, right? Like fractally, I want to make a plan for the whole problem that I'm solving But then also every single step, right? I want to extract this part from the response from the API. That in itself you can apply that same loop to of like, what am I trying to do? Carefully, you know, implementing it, reviewing the outputs, reflecting, is that actually doing what I wanted? And looping and looping and looping from the 1-week scale to the like 10-second scale.

Yeah, definitely. And so something interesting over the last year and a bit has been we've tried using this approach for dozens of different kinds of things, and it's, it's worked across a lot more than just software, you know, but we're trying to give you a taste of some of it today. So I'll give you a quick—

and I get there's one more thing that I think is really huge is the whole literate programming angle. Like if you're going to learn, if you're coding, this is the frustration I have, like you don't want to be at arm's length from the code, like you know, static IDEs where your code is there kind of discourages you from playing with your code, honestly. Like, especially if LLM is generating it, you're like, oh well, like, what am I going to do? I'm going to like copy and paste it into some other thing. Um, you know, and the things that you've built, Jeremy, kind of like before this like LLM stuff, um, you know, is like helps you play with code in a more natural way. So like, I think it's like the fusion of these two things, like this problem-solving approach plus the literate programming. And the literate programming, I think you just have to see it, but it's something to keep in mind.

I mean, I can show you a quick example of that actually. That's a good point, right? So as we're kind of saying, this idea has been around a lot longer. We've been working on it for over 10 years now, a lot longer than generative AI has been a thing. This is a library I created called Claudette, which is, you know, I think it's by far the best library for interacting with Anthropic Claude API. And you'll see this page is the source code. This is literally the source code to the library. And if you read it, you'll also find that it also teaches you how Anthropic's SDK works. So, you know, can you explain that a little bit more?

I think that can break people's brains. Like, what do you mean this page is the source code?

Sure. So, so, you know, here's some, here's, you know, a description of how to use Anthropic's SDK. Here's a little tip about it. Here's some code that I actually ran, right? And so I haven't built anything yet, right? And then I start saying like, oh, you know, the output from Claude is really hard to read. So let's write some functions to make it easier. So first I create a function called find_block and here is the definition of the function that I wrote. So here's some source code to an actual function. Which I then show how to use, right? And then we write a function called contents, which we then show how to use, right? And then we create a special function called repr_markdown, which is what Jupyter uses to create a Markdown representation, in this case of an Anthropic message. And so by the time we've done that, now when I say, okay, here's a message, it's automatically formatting it for me. So I've written the source code and showed how to use it, provided tests, but it's all in one place, you know. So another example is usage.

So just to clarify, people that may not be familiar, what Jeremy's talking about is like he has authored this document. There's not a separate web page that he made or separate documentation that he made. He actually like wrote all of this in one place and it became documentation and source code.

Yeah, the documentation is exported from it and the module is exported from it and the tests are written in it. It's all in one place. Yeah. So for example, we want like usage input output cache read write. So, you know, be available more easily. So we create again a TinyTotal function. To do that. And we start adding some properties to it, like total, get the total amount. And again, as soon as we create everything, we immediately show it in use. And then we create a representation of it. So now we can just type it and we can immediately see. So yeah, we kind of build things up slowly, right? And none of this is about AI, right? But then when you add AI to it as well, you get this really beautiful loop going on. So if you— so we've got this thing called Solve-IT, which everybody in this course now has access to if you wish. But a lot of these ideas you can use outside of Solve-IT as well. Solve-IT's just unusual in it. Really makes it easy to do. So if you subscribe to Solve It, you get access to, I don't know what this is, probably 25+30 hours of lessons, including 2 from Eric Ries about building startups using this approach, exploring APIs with this approach, building out websites with this approach, writing with a guy called Hamel Husain, whoever that is, and so forth. And then you can start, you know, you can then click connect and go into the system. And the very first lesson that we— you did in that course was to basically just show— quite a few lessons basically going right back to the basics, kind of algorithm— algorithms and data structures. So I had my— she was then 8-year-old— try it out. So her name's Claire, and so this is her attempt at doing Lesson 1 from the course. So Lesson 1 talks about the Advent of Code puzzle number 1. Advent of Code is a really fun set of puzzles each year, and so you can see here she just was asking So this is what Solve-It looks like. She's asking Solve-It, like, and Solve-It sees everything you see, right? So she pasted in the problem to solve and then she asked Solve-It, can I, can you see it? And so she's got here, actually it's interesting, she's got all 3 examples of the kind of things we have. You can have a note, you can have an AI prompt, which gives you an answer, or you can have code. And then she's asking it, how does that look today? Because she wasn't sure how to get the sample into it. So, okay, she doesn't know how to get the lines out of this. So it tells her, she tries it. And so you can see it's really encouraging her to write the code, not to do lots of for you all at once. So she's like, okay, help me with this first line. How do I go about it? Again, it's giving you small bits of code. To kind of try and she's flittering around doing it in forward and reverse and printing things out and she's getting very excited because she hadn't done much Python coding at all at this point. So you can put things into folders and stuff. So I've created a folder here for the lesson. So basically you can use it just like ChatGPT or Claude or anything. If you want to use it in that way, you just turn off this default code. We'll talk about why you shouldn't do that too often, but you can. You can then just go to prompt and you can say things like, hi there, can you please teach me how to plot a sine wave? And then what you should do is you can click submit, but what I suggest you do if you want it to be just like kind of ChatGPT is hold down the Option or Alt key and press Enter. And that sends off this prompt and puts you down here ready for the next one. But you'll see it doesn't go and do it for you, right? So it's really trying to use a good approach here. This is a lot more than just a system prompt, by the way. So, yeah, I know the basic idea of a sine wave and I've done some web programming with Python, so I know how to import stuff, for instance. So I'll do the same again. I'll hold down Option and press Enter. And anytime you can, you know, ask for more help. So I'm not sure, can you show me the imports I need? Now here I've got some code, right? And so the idea is that we make it super easy to use that code. So if you just hit the W key, it pops that code here, right? And then I can immediately hit Alt+Enter or Command+Enter. Command+Enter stays where I am or Alt+Enter creates a new line, and then I can keep going. So create another prompt, and I'm not sure. And so again, it can see the code you write, right? So we can work interactively. So I'm going to leave it there and hand it over to Johno to show how we can use this kind of approach to build evals which you can really trust and develop over time and feel very confident about.

Cool, thanks Jeremy. All right, so I just wanted to reiterate that we're going to be showing the tool and the process today, and the more important of the two is the process. And so you could do all of this in something like Google Colab or modal notebooks or Jupyter notebooks, and a lot of them have some sort of AI, but you'll see as we go that we've really tried to tailor SolveIT to be even more suited to this kind of workflow. So for the demo, I thought, I coded this up yesterday, it would be fun to quiz state-of-the-art models, you know, to give them some sort of eval and to give them an eval that I care about. And so I thought I'd take some of my observations on a platform called iNaturalist, which is a way for people around the world to share photographs or records of organisms in the wild so that people can find out things like distribution or patterns of behaviour and so on. I thought it'd be fun to feed some of these images to a language model, these visual language models, multimodal models, and see how well they do at identifying them. And so over in SolveIT, we're going to start with plotting what our sort of main approach is and then flow through the steps of turning this somewhat vague idea into a concrete evaluation that we can apply to some different language models. And then along the way, seeing how that evolves and yeah, what the process looks like in this very stepwise iterative way. Okay, so Paul here is setting the plan. I wanna test these state-of-the-art models on a hard task, identifying organisms from photos at species level. So I need to find a way of getting some pictures from iNaturalist. And then turn that into a multiple choice question that I can give. And my rough goal is to have like 50 questions just to do a small scale, you know, can these models do it at all, roughly how good are they, and how good am I, what's this human baseline. And so you'll see I've got these different sections here that I've used just to keep things organized, but we're going to start by exploring, can we pull these, this data from the, the API, is there an API, and working through that piece by piece. So I've started out asking Solvit, can you search and show me how to get, say, 100 of my observations from iNaturalist's API? This is my user ID, and I'd like only the research-grade ones. These are the ones where other experts have come in and confirmed that, you know, the ID that I suggested or that the automatic system suggested, someone else who's really good at identification has come in and agreed with that observation, you know, tag. And so it's gone and searched and said, yep, based on my search results, you can get these observations. This is the API that they have. You can get up to 200 observations in a single request, and here's some code. And you can see—

John, do you wanna just zoom in one or two levels?

Oh yeah, yeah, good idea. So it's gone and done, it saved me having to leave the interface, right? I could also have just gone and searched and tried to find some example, but it's given me exactly what I want. I wanted to get 100 observations of my observations at research quality grade, This seems great. And so straight away we can try that out, right? So I'm setting in the parameters that we want using HTTPX, getting the responses, and then checking straight away, like, did I actually get some results back? And what does the result look like? So if I go result 0, I wanna inspect the very first item there. And we get back this giant chunk of JSON that has what time it was observed, whereabouts it was observed, whether or not the identifications agree, et cetera, et cetera, et cetera, all the information we could possibly need to have this stuff that we can work with. And so there's a couple of pieces I'll come back to the fact that this code looks different to this and how we can steer or solve it in a little bit. But first test was, okay, I wanna be feeding these images, can I get the image? And so here we go, let's plot the image of the first result. And the way, even for a line of code like this, it's very rare that I'll write a whole thing like that in one go, right? You'd probably start by saying, let's, sorry, the Zoom interface is kind of in the way. Let's look at the photos part of that giant big result that we got, right? And we see that there's 2 photos. Okay, so let's take the first one. And then, okay, cool. There's the URL that I want and it has the URL square.jpg, but I know from past work with this API that they have, you know, small, medium, large, or square. And because this is dynamic, because this is live, I'm not having to like rerun the whole script and redownload all 100 observations from the API or anything like that. It's just a live variable that I can work with. And so I could call this photo, you know, URL, is equal to results photo. Here we go. Um, you— and you can see like the dynamic autocomplete. This is an object that's live in my environment, right? I can see what, um, attributes this specific thing has. I could put that in a new variable. Um, and if I ever did Ctrl+Shift+V or, um, opened up the little contents page here, I can see I have these, um variables, I can see where they're defined. There's the photo URL, and then we could go on and load it and feed it into a language model and so on. Okay, so we're just starting to explore.

So you can jump to a definition of anything just right there on the right-hand side?

Yeah, yeah. Where did I get those results from? And also for headings, right? If I wanted to go back to the top heading or the different sections that I've done, just a nice way to keep it a little bit organized.

Cool.

Okay. By the way, you should check out this plant. They call them ghost pipes or different plants. No chlorophyll, just taps into the fungal network under the trees. So I was very excited to find that this weekend.

Oh my God.

So that's your picture?

Yeah. Yeah. So I went, I specified I just want ones from my user. And so it's just ones that I'm familiar with, right? So I'll hopefully be able to spot errors a little bit more easily because it's stuff that I know. And yeah, that one's from this weekend we went backpacking up in Washington. Okay, so we're starting out on our plan, right? We said we'd get some observations, we'd come up with ways to turn that into a multiple choice, and then we'd feed that to an LLM and see how well it does. So for the next step, I need to get the other options if I'm putting this in a multiple choice. And I wanna make it hard for the model, so I want other similar species, right? Closely related species, possibly in the same genus. And so again, I went to solve it. I don't know the iNaturalist API, you know, by heart. I'd like to get these distractors in the same genus. Can you show me how to do this for one observation using the TaxAPI? And so it's gone and searched and given me back the same sort of format of like, here's how I could do it. Now you'll notice like my preferred way of doing it isn't to make a big dictionary on multiple lines and so on and be all verbose. And so what I could have done if I cared about that, is taken the code that it gave me versus the code that I actually ended up with. We can go and edit the LLM response here. So I'll edit the initial response for the first time I asked it. Here's a simple example to get you started. I could replace its example with mine, right? And trim it down a little bit just to let it know We're happy with, you know, more concise, simple examples. And now that I've shown it this format, if I re-ask this question, right, so I can click this little star button to rerun that AI, my guess is, I haven't done this before, but my guess is it'll follow that more concise syntax that just happens to be the style that I prefer.

Sometimes it does. It helps to like, often you have to ask it once and also have it show it once. So just to explain what the reason this is really important, and it's really weird, almost nobody else does this. That did work nice. Is that, you know, a lot of you might be aware behind the scenes language models are what's called autoregressive, which is to say they look at what the most likely next word is. And so when they have examples in their own responses that are formatted the way you want them to be formatted, that's the strongest possible thing for it to make it be more like that in the future. This is why people often fall into this huge trap where a language model makes a dumb mistake and then they respond by saying like, that is a dumb mistake. Here are all the reasons it's dumb. Now make it better. And it gets it wrong again. You're like, ah, that's so, so, so bad. Make it better. Every time you try to get a language model to fix a mistake, you make it more likely to make more mistakes of that kind in the future because autoregressively it's more likely the next word in a dialogue where there's been lots of mistakes is another mistake. So it's much better to go back and edit a response to make it correct or to say just once, "Oh, that wasn't right. Can you show me how to do it better this way?" and then edit its response to that to make it correct so that there's now like the autoregressive flow of the dialogue is one of behaving the way you want it to and it's going to make it and the more you do that the better the dialogue will get.

Cool, okay, so I'm going to speed up a little bit and work through the code that I ran but there's a few things I'll pull out. So we're—

Yeah, actually a quick question for the HiNaturalist API, did you pull in those docs explicitly? Did it just go out and search and those? Like, how did that happen?

No, so you can pull them in explicitly, and I'll show an example of that for FastHTML. Um, but SolveIT also has, uh, like web search and reading URLs by default. So I could give it a link to the API docs and say, could you go read this page and then show me how to do this? Or it can just go and search. And so in this case, I think a combination of the API being somewhat old and stable, and then also its ability to then go and do a web search, read the page, come back You can see, I think in one of the early ones, it'll give little citations and you can see the results of the tool use reading this URL.

You deleted some of the citations earlier.

Oh yeah, that's true. That's true, yeah. Cool, okay. So our quest is to go and find related species that we can use for our multiple choice. And so what I've done is taken the code that it gave me to get related species in the same genus, right? So this genus ID is, the sort of higher level group that the species that I'm interested in falls under. And so then I can go and say, let me pick 4 random distractor ones from the list that I've got here. And then straight away it's like, wait, hang on a minute, right? This is not 4 different random options that I could use for my different multiple choice. This one is the plant that we're interested in and there's a repeat. right? And so me being a fallible human, I'd use random.choices when instead I probably wanted random.sample, right? So random.choices, I just here inspected the signature. It says it takes chosen_with_replacement. Ah, okay. What do I want if I want random.choices without replacement, i.e., unique choices? That's right, random.sample. So I have the AI there to kind of, A, you know, maybe check my expectations. Like I could probably have said, At this point here, let's go add below. How are we looking? I'm guessing it would be smart enough to notice because it can see the code and the outputs and the objective to say, oh, you know, there's something, you're making a mistake here. You're getting duplicates in your random selection and some species might have common names. Would you like me to help? So the AI is also pretty good at that, but I didn't even need that, right? Because I could see straight away visually inspecting, is this output what I expected? Know there's duplicates and so we can work around that. We use random.sample_state.

And this is kind of our strongest, like, theme in, in the SolveIT course is after every 1 to 2 lines of code, output the result. Now that does mean if you're working on 100,000 samples, that's not going to work. So what we always do is we always go like x samples and [0] then we do everything end-to-end on x, print, print, print, print, print. And in this environment, you don't even have to say print, as you can see from Johno, you just evaluate, the last line will always print out. And then once you've got that working, you know, all those 5 or 6 lines and 5 or 6 separate messages, then you can create a function to put it together, which I guess is what Johno is going to do now.

Right, yeah, so once we've ironed out the kinks, we're working just with one example, right, one that we know, we're looking at the outputs, Once we've done all of those pieces together and said, okay, I'm finally at the stage that I actually wanted where the distractors are all unique, they're not the original one, then I can copy and paste that code or that we have, you know, a merge and duplicate all of those shortcuts into a single function. And then I can test that function right away. I am now getting, I tried to run several different results. Oops, should actually run the code. I'm now getting the distractions that I need for this example that I can then turn into a multiple choice, right? And so we're building up this functionality that we want, we're doing it very iteratively piece by piece, and we already found a bug or two, right? Looking at those outputs and saying, hang on a minute, that's the genus that we're— that's the species that we're interested in, it shouldn't be in the distractors, or hey, wait a minute, there's a duplicate, I'm using random.choices instead of random.sample. And so it's very, very slow and gradual building it up together. Okay, so that's the process. I'm still leaning on the AI for things like, you know, once I've done this for 50 questions, I'm making lots of API calls to iNaturalist. I know they're a nonprofit, I don't wanna be hammering them. So I've got a nice time.sleep saying, hey, can you save that to JSON and load it again? Like that kind of code, I'm very happy to just quickly have the AI save me the busyness of typing. But if it was my first time ever, I'd probably be saying, okay, how should I save this? What would be a good format, et cetera, et cetera. But yeah, just plug it in.

Because otherwise, if it's using a library function you're not familiar with, you also won't know if it's using it the wrong way. You won't need, you know, it, no. So we kind of have a bit of a rule of thumb at Answer AI and also in the course, which is anytime an AI shows you a function, a library, or an API you're not very familiar with, you should stop and you should go and read the documentation for it. You should then go back to SolveIt and say, oh, I want to learn all about this. And then you should try running a few different inputs, see the outputs, get SolveIt to help you understand the details. And only then should you allow yourself to move on. I think Jono's probably used json.load once or twice before, so he's probably fine.

Yeah. Just one time.

Cool.

Okay, so the next section is now asking these questions to the AI.

And so you're opening up a collapsed header there, I see.

Yeah, yeah. So I've got, I can use left and right arrows to expand and contract, or there's the little toggle thing there.

And it expands and contracts it to the next occurrence of a heading level of the same or higher level.

Yeah, exactly. So if I wanted, I could make it so that everything up to this point, getting all the observations, is all collapsed, and then this is now like the piece that I'm working on. And I might like at the start of this then show, or even just like remind myself what I'm doing. This is the type of data that I have so far, and I want to turn this into something that I'm sending to the language model. I want to give it this picture these options and the correct option all jumbled together, and then I want to get back a response that forces it to choose one of those options. And so I know there's function calling or structured outputs and JSON mode, and all of these APIs have kind of shifted over time. Another great use to say, could you search and get the up-to-date one? Those of you who've worked with using LLMs to write code that uses other LLMs, you'll know if you don't have kind of web search or grounding, you often end up with like outdated versions of the OpenAI API. Very frustrating if you're trying to use like, oh, the new Gemini one, right? So having the search is kind of nice. And then here, this is a slightly different usage of the AI in that this is less like, I'm not going to use this code, all of it right away. I just want to see the equivalent of the docs example, right? And so I could also have just gone to this URL myself. I just want to see the general structure of how do, how does one send data to the API and get back a structured output. But it's very nice to have SolveIT go read those docs and produce it for me with the context of I'm wanting to send an image and I want to, you know, produce my data.

The citations are really nice, it allows you to build trust in what is going on.

Yeah, yeah, exactly. But I don't— I also don't need to build trust that way, right? Because I can try it. So I can look at my question, I can shuffle them up and look at the options, and then I can build my prompt and print out the prompt straight away. And this is very, I know you guys have been doing all sorts of fun and fancy evals and this is like the, whatever the lowest tier of prompt engineering is, but it's still really nice to be able to just look and read, okay, identify the species in this image, choose the correct answer. There's my 5 options. And I know in this case it's a California turtle shell, 'cause I saw in the question there that that was the correct one. Oh no, it's a Mexican turtle shell. Nope, it is a California turtle shell. You know, so you can like slowly build up, okay, can I send this to the thing? What does the response look like? And before I'm pulling out the thing, I'm like, okay, let's look at, you know, maybe the response, you know, the first thing there. And then the multiple choice answer. And then there's the content. So you can explore and dig into all these pieces and slowly pull out, this is what I actually wanted is the model choice species and look at the question and are they the same? And only once I'm happy with all of that, this now comes back to the code that's almost like what it gave me initially. But I've now run this, right? I've now looked at my random options. I've now built my prompt and printed it out. I've now called this and looked at what the response actually looks like, and I'm content that this brings me back out the actual answer. And then straight away I can run that function and see this is the model answer, this is the correct answer. Does the model answer equal the correct answer? Yes. You know, so we've done all the pieces, we've tried them all out, and then we put them all together. And so finally we're ready to send them all through the model, and we can get a score that we're at least at a first blush, like, okay, I'm somewhat comfortable that the prompt is getting formatted the way I hoped. The model is able to get, you know, better than chance, so maybe it's doing something. Ideally, then next steps would be like looking at all the mistakes, right? So printing out, because I've got the guesses and the questions, printing out which ones did it make a mistake on, inspecting the data manually, we'll look at next.

And remind me, are the note blocks also produced by Solvit, or is that you authoring that?

No, that's mostly me authoring that. You can, we have primitives for like giving Solvit tools that can add messages. So you can say things like add more explanation to my dialogue and it can go back and add extra notes. But no, these are all me making notes to myself, but also notes for this lesson. I can share this dialogue if people are interested, but it helps the more information you can pack. Like if you're having thoughts or thinking, oh, the next thing I should try is looking at this, I should inspect someone's, If you're having those kinds of thoughts, putting them in notes for yourself for later, but also for the language model later when you say, can you help me with the next step? It knows what the next step is, right? Because it sees my plan at the beginning, it sees my headings, it knows what we're working on. So the more information you pack into this shared dialogue, the more likely it is that the other participants in the dialogue, i.e., the computer and the AI and future you, are all going to be able to contribute usefully.

So, Jono, we could try something here just to show an example of a really easy way to do this. Maybe, so one thing to mention is A really important theme of SolveIT is the language model only sees stuff above the current message. So, Jono, can you do me a favor? Can you copy the contents of that note and then create a new message above it and then paste in the contents of the note, change it to a note, and then delete the everything except for the first hyphen of the list. Not the last, sorry, leave the, just undo that, just leave the very last line in as well. So just get rid of all the list except for the first hyphen. Yeah, and put a space. Okay, now Jono can press Command+Shift+Full Stop and it will look at the entire dialogue to autofill his list.

Nice.

There you go, right, and it sees everything above it in the current message and below it in the current message, plus all the previous messages, but none of the next messages. Another thing he could do is delete all of that and delete the hyphen as well. And now create, write anything. So just put a space and then select it. Now if you've selected something first and you press Command+Shift+full stop, it lets you, it puts up a prompt. So try that. And now you could type, create a list of the bugs we've seen as a markdown list or something like this.

So we have some of the features—

So you can do a note cell, code cell.

Yeah, anywhere. Exactly.

Yeah.

You can even put it in a prompt response to like auto-generate the prompt response you wish the prompt had created to get it in the right mindset.

Yeah, yeah. So we, we, we could maybe show like we have all the normal things like ghost text or like this completion, but they're hidden by default. Like they take a little while to show up, they don't always show up. They are also by default in learning mode.

You can press Option full stop to auto.

This is the like minimal normal ghost text that I can accept. But if you're in learning mode, which we try and use as a default and encourage, you don't get that an issue proactive. So I could use a keyboard shortcut to trigger it, but if I don't, I'm encouraged to type the code myself and think a little bit about it myself.

And this is from like a lot of experience we've all had, you know, is most of us at this point have independently discovered that when we're using stuff like Cursor and relying on tab completion and stuff, it's damaging our ability to do high-quality work over an extended period of time. So we're like, and it's, it's, it's just not enough to like, again, there's a lot of psychological research about this, which is if you, it's not enough to expect to rely on willpower to like not hit Tab lots of times because every time you do, it's using up your energy, you know. So instead we try to make it that the natural, easy, comfortable flow is the one that is going to produce high-quality work and high-quality learnings and, you know, a positive feedback loop where you and the code get better and better.

Maybe a fun way to show it is I thought I'd try— so look at your data, right? The Gospel of Hamel Husain. We should definitely make a little widget to try this eval ourselves and to see if it's it's reasonable and check if there's any bugs.

By the way, there's a tier of gospels. I think I probably may have learned it from Jeremy.

So, um, but yeah, so I did one version where I wrote that myself using FastHTML, which is our library. I think we probably don't have time to go into it, but it's one of these pieces that we built to make things easy for ourselves. And you should definitely go check it out if you're interested in developing web apps, and it plays really nicely in SolveIT. Um, And so yeah, I'd done it, but you can see I've hidden my version from the AI. I thought it'd be fun to see like, what if I really lean on SolveIT? Like, can I push it more into the, I'm not looking at the code, I'm not understanding it, I'm just, you know, closer to vibe coding because this is like an ancillary task on an already somewhat frivolous eval. And so I load up the FastHTML docs. We have a concise guide with like, this is the way we like to write FastHTML. I say I have fasthtml imported. Can you make routes and pass them to me?

Let's just mention fasthtml is a web framework we've created for creating web applications. It's really good. Everything you're seeing here is written in fasthtml, by the way. And in fact, everything you see here is written in fasthtml using notebooks, by the way. And increasingly we're using SolveIt to write SolveIt. Itself. So if you're wondering, can you create complete production systems, the answer is very much yes.

Yeah, and so I'm kind of very much angling for like, could you write this for me, solve it? And it's very much pushing back and saying like, hey, we should build—

scroll up just for one moment to that where you have the guide. Oh yeah, I just want to like this, like one of my favorite features, like right here. Where you, you know, use that variable in a prompt is magical. It's not— that is not available in any other tool that I'm aware of. Like, you can create this and inject it into a prompt later, like something from your code.

Yeah. Um, so we have this system where we can, uh, we can give Solver tools with, with that syntax, and we can also give it, um, the contents of a variable with the syntax. And then it's able to read that as text and answer questions about it. It's a very, very useful way to pull in context.

And to be clear, it's not interpolating it into the string. It's a much more careful approach here where I don't need to provide the full detail, but basically it's setting up a system where it has access to what it calls its context, which can include these like variables that it can then back-reference. And so it's done in a way that works extremely well with how caching works in, you know, KV caches work in Language Models, for example, with how attention works and so forth. So these features are all like built with a strong eye to taking advantage of the, you know, what language models are good at.

Yeah, so this is not going to be a FastHTML tutorial, but you can see I've whipped up a quick little widget here that I can use to take the test myself.

That's crazy, right? You've just, you've just written a web app in a message.

Yeah.

And it's important to know as well, this is not like HiPy widgets or something. Just scroll up to see the web app. It's a real web app. Just scroll up to the code maybe.

Oh, sure, yeah.

Yeah, it's a real web app, right?

We'll dwell on, but I think the important stuff out of this is going to be much more the discussion that isn't like fast HTML tutorial. But yeah, I guess still fun to see. Yeah, we can, we've got all of the primitives of HTML, right? Like headings and paragraphs and images and so on. We also have htmx, which gives you the ability to have the button when it's clicked, send a POST request to some route. And then FastHTML lets us define those routes. So for example, this answer one here, when it receives a choice from someone clicking on the button, it's going to go and check if the choice is correct. And if so, it's gonna update the score. It will move on to the next question and then show that question. And you can see like this web app is running, we're in the same environment. So because I've clicked the button, you know, 5 times and I got all 5 correct, I can see, I can see what it's doing. I could mess with those variables. I could set the, the current question back to 0 if I wanted to restart. So it's, it's like all the same live environment. I'm not compiling this to a web app and then saving that to disk and then running that on a separate server and then interacting there. It's all just the same live environment. Sorry, I'm getting distracted by my own quiz.

And you can see here like how This is a great example of this positive feedback loop in action, right? You know, we, we can write SolveIT in SolveIT now. We have all these benefits for ourselves as we build SolveIT. SolveIT's written in FastHTML, which is something that we created to make it easier to write web apps. So FastHTML uses Monster UI, which is the CSS framework we developed to create easier web apps that, like, it's like layer upon layer upon layer of us continually building stuff to make it easier for us to build stuff. I feel like that's the mindset you want to be in of like, how do we just keep it making it so our work, you know, gets more and more fun and faster and faster and easier and easier.

In this web app that Jono showed, I just want to point out that this is not something that just works inside SolveIT. This is a real web framework. You could extract this code and, you know, run it anywhere. It just happens that you can also run it in this live environment. And I like Instead of running a separate web server, like, you know, in two windows or whatever, you're just kind of like rendering it right there and playing with the live app.

Yeah. So we have some like long-running apps that are an instance of SolveIT that slowly builds up the app iteratively and then serves it and then it just sits there running. Um, and that then lets us access it. You know, we just have the URL shared and we have one, I think that does, you know, various Discord tricks via the Discord API and I think you wanted to show _question, by the way, John O. Oh yeah, I mean, this isn't a root and I also—

Oh, it's not a root.

I don't have all the headers and things, but yeah.

Well, one thing you can also do is maybe you can show is if you can click on the right-hand side, little pop-out button in the prompt answer and the— sorry, in the output. Yeah.

Yeah.

So you can turn anything into a usable application by clicking that button. You can then send that URL to anybody. To use that application independently. That's now a full working web app.

Yeah, and so it was fun, it was fun going through the quiz. Things like this Argyllospora, I mean, I couldn't identify that, I had to send it to some specialists and I noticed the model got that one wrong as well. And going through the ones that it made mistakes on, they all seemed like particularly tricky ones, right? The torrent sculpin, the only reason I knew that was a torrent sculpin and not a spoonhead sculpin is 'cause I saw it in a freshwater stream and it didn't have that information. So I found it very useful from a like, is my eval any good point of view, like checking the mistakes and doing it myself and also very gratifying to beat the model at least on my own observations.

I can show another quick demo if you like.

Yeah, yeah, I think I'm pretty much done with mine.

Oh.

The one novel thing is like, you're not doing any context switching at all. Like, you're writing code, you can write prose, you can bring your code stuff into your prose, you can then do a web app, you can play with your web app. There's no other coding environment that allows you to do all of these things, just mix and mash them in one without—

yeah. And it's not even just a coding environment, you know. Like we say, Eric Reese has been writing his new book in it, um, we write a lot of blog posts in it. We do, okay, yeah, all kinds of stuff. We have a law firm, as you know, Hamel Husain, who you'll be pleased to know has recently started doing legal contract drafting in it. Wow. Yeah, yeah, I'll show another example that I really enjoy using this for, which is to use kind of SaaS APIs, online APIs, stuff like that. One which is particularly difficult to use in the past, although they've been improving recently, is the Cloudflare API. So I want to show an example of, of that. So maybe I'll just kind of skip over a few things a little bit so we can like, I want to kind of show something that you might otherwise do in something like Claude Code to show how this kind of more interactive approach can, can be really helpful. So you can pop an exclamation mark on the front of things and use, you know, bash commands. And then often, like, for things that I don't really want to run again, that's already cloned, I'll tend to just comment them out. So command slash comments out. If you want to start adding stuff to the end of a dialogue, you can just hit the forward slash key and you'll see that starts there, right? There's a whole lot of these keyboard shortcuts, like lots and lots, and pretty much all of them are worth learning. So if I want to you know, start using this, I can start exploring it a bit myself, right? And so you can see there's little suggestions appear for paths and symbols. So if I hit Tab, that's going to fill out that suggestion for me. And the nice thing is not only am I seeing that, but the, the model does as well, right? So each of these has a shortcut: J, K, L with Command+Shift or Control+Shift. So I can say, oh cool, there's a README, right? So let's have a look at it. flare-python slash Oops, I accidentally was in prompt mode there, so it's going to get confused. So I'll go ahead and change that to code. All right. Now, in this case, there's quite a lot to read. Probably not a bad idea to read it, but if, you know, if I'm in a hurry, which I am now, you can just press O to collapse the output. You see how it's made that smaller and I to collapse the input and I could just have it read it. For me. Can you tell me whereabouts the documentation of this library is? Yeah, okay, great. So it's told me that, so it saved me a bit of time there. Okay, great. Can you please use some RG to search that API file to find Anything that might be useful for my project, which is that I want to create a simple function to create and delete CNAME records. Actually, show me how to use Since this API changes a lot, let's only do things after we've searched the documentation and examples to see exactly what the current version does. This is the problem I've had with this Cloudflare API is that it changes a lot. It's really poorly documented. And so in fact, if you have a look, you'll see it says, yeah, usesapi.md. And it's like, okay, this is our documentation and it's just insane. That's a quarter of the way through. And none of that's even docs. It's just a list of stuff, right? So this is pretty difficult. So Oh, I forgot to say, Arjun, show me how to use.

What do you think their intention was? They were trying to do an lms.txt and did a bad job?

No, this has been around for a long time. They just never really got around to documenting it, you know. So yeah, so here I can now hit, I can either hit this button, or I can just hit W and that's going to pull out any code blocks and turn them into cells here, messages here. You can see it knows this is a bash one, so it's automatically put an exclamation mark on the front of it for us. And, you know, it hasn't done a great job here. I think I'd say Maybe we should search for a wider variety of things to understand how to fully utilize the DNS API and maybe also show some context around the results. Can you show me how to do that and suggest some good approaches to doing this really rigorously? So RG is a really nice kind of grep-based tool. And so, cool. So here's quite a few examples. Okay, that's great. Can you put each of those 4 examples into a separate fenced block for me? So it doesn't know necessarily the fenced blocks are what I really want. Oh, I've got to click here. That looks great. Can you put each of those 4 examples into separate fenced blocks? So you kind of have to show it how to make your life easier. So once it's done that, this is really cool. If I hit W, right, I've now got 4 things I can run and then I can see how they go, right? So, okay, there's that one.

And you can also see, yeah, you can go like you hit W and it'll generate, it'll grab the code and put it in.

Grab the code and create them. Yeah, and I can see how many tokens is in each output as well. So you're not going to end up like overwhelming your context without knowing about it. And so then Yeah, I mean, some of them are going to fix, like putting in anything with a capital A is obviously stupid. And then minus i there is pretty stupid as well. So it's good. You can kind of catch it doing dumb things. And so I tend to say at this point, like, okay, Great. Do you feel like we have all the information we need at this point, or is there anything else we need to look up? So yeah, I'm kind of going faster here, but, you know, I'd be inclined to be reading this carefully. And so if you didn't know about -C, that's a really useful trick to know. Which is that it prints out a bit, a few lines before and after each search result, which is really great. Great, so it thinks it's found all the information that it needs at this point and we can, you know, start working through it. So that's one approach which I really like. There's another approach I also really like, which is to write your own tools. So in this case, we were— because the, um, what we see in the dialogue and what the LLM sees in the dialogue is the same thing. It's totally transparent at all times. Um, but if you want to be a little bit more agentic, it's really interesting to see with the SolveIT approach, how easy it is to create your own complete agent framework to do exactly what you want, no more and no less. So here I just have this little function that I really like using called run_command. It's going to use subprocess.run, it's just the thing that runs arbitrary binaries in Python, and it's going to run a particular command and then you just pass in the entire argument to run with that command. Right. So with that, I can create a function called RG, which is now a Python function, right? arg_str colon str. Run the RG command with the args in arg. Master, no need to backslash, escape, return, run command. And you know, here's a place where I maybe I just hit Option full stop, you know, just to type in the rest of that for me. Oops, press the wrong button. Right there we go. Now I'll show you what's really interesting about this. So we can then try it out, right? So rg -i -c 3 Bay records, for example. Just make sure it works. Oh, and we've got to say where to do it. So we're going to do that in cloudflare-python. Oh, I should probably want cloudflare-python/api.md. api.md. Okay, cool. That seems to be working. So here's what's really fun now. I can, so we can kind of, what I tend to often do with this kind of thing is I'll often like paste in things like Polyas' problem solving strategy, you know, or, and paste in my restating the problem in my words, you know. So here I might let it add a note and say like, today we are going to try to create a function, two functions one which generates a CNAME record and one which can delete any arbitrary record. We are going to be ensuring that we check the Cloudflare API documentation for everything to make sure that we are using the most recent version and the appropriate parameters. So I'll add that as a note. And then we can say, can you please use RG to search the documentation for me for everything that might be helpful for completing today's project? You can also look for examples as well as checking the API MD file. After you've found what we think— you think we need, create a complete detailed list and documentation of everything we need to know so that you and I can refer to that in the future as we build out these functions. Go ahead and do all the research required now using the RG command that I provided. Okay, so something like that. Now, how is it going to do this? Well, here's the trick. You can turn any function into a tool by typing ampersand and putting backticks around it. And as soon as you do that, you now allow SolveIT to call that agentically, however it likes. And in fact, we can do even more, which is there's another trick you guys may or may not know, which is I'm going to say make a copy of this. I'm going to call this one said. This one said particularly useful for reading a section of a file. So I said something that can, for example, read a bunch of line numbers. If you find sections that might be useful, you can also use ampersand backtick set to read those sections. Okay. So I've given it two tools, something to search, the Cloudflare Python. I should maybe just mention that only look in Cloudflare Python. All right, so it's now going to go away and do it. So you might be thinking like, oh, is this like MCP or what's going on here? It's like, no, you don't need MCP, you don't need Claude TextEdit API, you don't need anything. Oh, that might be too big. It's fine. Try to keep amount of info you find limited so we don't overload the context window, overload your context window. Let's try that again. Yeah, this is literally all you need is just to write functions and any Python function you can write is now a tool that your LLM can call as well as you calling it.

I mean, this is really amazing. And again, there's no indirection. Like you just have your code and you don't have to like now go somewhere else to plug it in.

Yeah, you don't have to create any scaffolding or schemas or servers or anything. Yeah, it just, that's it.

You can define the tools in one dialogue and import them into another. I don't think we need to go too much into the exporting and importing, but it's pretty easy to just share like, hey everyone, here's a gist with some tools that I wrote for this. And then very easy to then import those and use them and feed them in as a list to say, hey, use any of these that you want. So it makes it really fun to prototype. It's kind of MCP-like things, but just in a much more explicit, Much quicker and easier.

And you don't, you're not relying on other people, you know, as long as you can write Python. So here's, here's what it did, right? So we can see in the details, it went RG-C2 CNAME. And then based on the result, it did RG DELETE. And you see it's using this context thing we learned about. And it's quite nice, like to check these details to see how is it doing things, to see if there's stuff you didn't know how to do before. And then it went like, okay, what else can we do? So it went RG def delete and then it's gone like, okay, check in tests and it found this example, found this example. And so now it's, and remember it, the LLM, in our prompts, we'll only see what we see. And actually, it doesn't see anything inside these detail sections, right? And so you'll see actually this dot dot dot, they're actually truncated here, right? So that's why it's important when you use this approach to tell it to like create the summary, right? So now we can say, great, can you please write a function for us which can Actually, let's not do that. Okay, let's start by importing Cloudflare and checking with some simple function that we can access it. So again, it's like really nice to do things in small steps. So I'll hit Alt+Enter. And now I'm going to mention this default code idea, which is it's good to keep this on because the idea is we actually want to write as much code ourselves as possible. So if you say default code, then after everything, regardless of whether you do a prompt or whether you do a note or whatever, it's always going to put you in code next, right? Okay, I have some credentials, so let's go ahead and create the object and and do some simple tests to make sure it's working. Great. So now I'll press W. And actually, rather than passing in the token, what you can do is we've got this thing in Solver, called secrets, and you can just dump them all here, anything you need.

I see that you have a lot of different API keys. Can you switch your model to Gemini, Groq, whatever?

How does it work?

No, these are for me to use that through code so that we, when you're doing stuff with prompts using our API key, Should show this one as well. You can also use this special one, max think tokens, to turn on access to Claude Opus 4.1. If you do that, you'll get an extra brain button. If you press that, it's going to use reasoning mode. So I put in 4096 to say use up to 4096 tokens of reasoning. Don't make it too high, otherwise it'll just crash on you. But 2048 or 4096 is pretty good. So in this case, I've already got the Cloudflare API in my secrets that automatically puts it in as an environment variable. So, oh, once you go back in, you'll have to rerun cells. Cool. So it's got all of our zones there. So we can now say, okay, that seems to be working. Can you show us how to create a function which can create CNAME record. And then let's test it out by creating a test record in the fast.ai domain, which we own. Oh, need to rerun those. So you can always rerun everything just by clicking Run All. And so this is the message that you'll get if you try to run a prompt and some of the things that are ampersand backticked and not available. So now that's available, I can rerun it. So say like, yeah, I think this API would be more ergonomic if instead we passed in a zone name and you could look it up for us. So again, it's really encouraging us to do it ourselves, which normally I would suggest doing is a good idea, but for now I'll just say like, sure, you go ahead and implement that for me please, since I'm in a bit of a hurry today. Okay, so we have a lot of really helpful keyboard shortcuts. So one thing I think is important in this style is to not use up lots of vertical space. You want to be able to see as much in a dialog as you can. The normal approach is to have these really, really tall things which don't take advantage of our widescreen monitors. So one useful thing is Ctrl+J, which joins the current line and the next line or the selected lines. So stuff like this can be quite helpful. This is actually a really crappy implementation. So if I was doing it with a bit more time, I would tell it to look up the docs to check whether there's a way to get a zone by name without doing this for loop, which you actually can. And you can see it's created a separate one for the test. Oh cool, that seems to have worked. Great. Okay, let's finish up now. Just show me quickly the line of code necessary to now delete this CNAME record.

And this is exactly like— I get to watch Jeremy code on stuff. This is pretty much exactly the process for anything we're going to be using. If we're going to be creating records with Cloudflare, then we're going to be using some nice nifty, you know, create CNAME record function that's built in a notebook somewhere. It's not like, oh, this is just for the demo. This is generally how coding happens.

Let's try it. Great. Yeah, and so this is something where we could even kind of grab this here and say Command+Shift+Full Stop or Command+Shift+Period, as Americans would say. Replace this with something that uses the API to look up the name directly rather than using a loop. Cool, much better. Yeah, so let me just have a quick look. Yeah, so the other modes we've got here, learning mode is the one we suggest you use, but if you want to, you know, just be super fast, if you go into concise mode, it's just going to try and get the work done for you as soon as quickly as possible with very concise targeted code. And then standard basically turns off, it just gives you kind of more of a standard plain Claude prompt. I never use that, I don't think it's very good. You can do everything with the keyboard, so you can press up and down to select things, and then like all the different buttons here, you can see what they all do. But so for example, there's a button to copy the input. There's another button to copy the output. If you have a look in here, you'll see that we have keyboard shortcuts for those. For things that can apply to input or output, we tend to put them next to each other. The one on the left, comma, is for the input, and the one on the right, full stop, is for the output. So for example, with this selected, if I press full stop, that will copy the output, and then I can paste it there, for example. And we've got quite a lot of this designed to be like super ergonomic. So if, if I've got something that I'm typing in already, then copying will immediately refocus it on the bottom. So this way it's really easy for me to kind of go through and say, yeah, I want that. Okay, I want that. Okay, I want that. You know, so there's lots of these little things we do to try and make everything like super fast. Also, you'll see you can maximize the editor, but like again, we've got all these little things to try to encourage you into the right mindset, which is we don't want you writing this amount of code, right? We want you writing this amount of code, and once you start writing, it'll put newlines there for you, but we're really trying to get your brain into the idea of like small batches, as Eric Reese would say. So it's going to feel really weird at first because it's really different to what you're used to, but try to go with it, you know. Yeah, so as I mentioned, we've got this brain button here that's going to take longer. It's going to use Opus 4.1. And do reasoning mode. You shouldn't need to use it very often, right? Ideally, you're using your brain to think. If you're clicking this button, it's a bit of a bad sign.

And is the, right now, the people that, you know, sign up for SolveIT, are they going to charge for tokens? Is it kind of flat fee?

Right now, we've provided this for free to our whole community that's signed up for this a year ago for the whole year, because our number one priority is to have people using it and giving us feedback. And so we've got this very super dynamic community of people trying things. And so, for example, every Advent of Code day has a complete discussion of people trying to solve it for both 2023 and 2024. And you'll see like a lot of people just talking about their experience of like, what's it feel like to use these techniques? We have office hours a few times a week and also constant kind of discussions of new features being added to the platform itself, a special channel for people who are quite new to coding.

Um, um, it is a really dynamic, uh, community. There's people talking about stuff that's not even SolveIT, just interesting projects they're working on.

Oh, we have a chit-chat here. Yeah, yeah, yeah.

Someone asked, um, what our evals look like, you know, for SolveIT. Do we have any sort of fancy automated evals? And the answer is actually that the evals is kind of like us in this community, um, seeing what works. I think, you know, that's maybe heresy in current company, but the small batch size of trying individual things, seeing how people respond to features or to changes, getting that like kind of like fast, immediate, obvious feedback is a hugely valuable and useful signal.

It's one of the reasons why we did the course. Yeah, there's no better A/B for like, hey, does this work well for writing? And then be like, hey, Eric Ries, why don't you try using this to write a book? Does it work well for web development? Okay, let's try to create a large new web application in it. Does it work well for blog posts? Okay, let's write all of our blog posts in it. It's a good eval.

Jeremy, what would you say? There's going to be a lot of people watching this. What kind of people should try SolveIT? You're a programmer, you might be watching this and be like, I don't know about this, or I'm used to this way of programming. You might have all these preconceived notions about programming. Can you just talk a little bit about it?

It seems to be best, honestly, for people who are really new to programming or to people that have over 20 years experience. The kind of the 3 to 20 period is often people who think they know programming well but haven't realized they don't yet. The people with over 20 years experience realize how much they still want to learn new things and, you know, tend to be like, oh my God, this fixes so many of the problems I've developed, discovered over the last 20 years. People with less than 3 years experience are still in that mindset of like, I've got so much to learn, I should try new things. But it's like, it's such a different way of working that for people with a medium amount of experience, it can feel almost confronting to be like, wait, this is not about CI and CD and PyTest and projects with 20 files with a separate function in each one. And like, it's a whole new way of thinking about not just AI-based coding, but coding in general. And I mean, you know, Hamel, we've done a lot of work together over the years and from time to time people have tried to work with us pre-AI, but in this much more exploratory-based approach, and they kind of bounce off because they're like, this isn't how you code. So I think it can help everybody, but either you need to be very junior or very senior or very open-minded, you know. I think the open-mindedness maybe more people are becoming more open-minded as they're realizing that the promises of the last year or two haven't really worked out. And so maybe there are more people now who are wondering, like, are there other approaches? And the answer is yes, this is, this is another approach, and it's working great for us. I do think you need to be part of the community though to benefit from it. If you, you know, to share your failures as well as your successes. We've had a lot of people on the Solver Community talk about their failures and ask for help to get past them, and a lot of people have got past those failures. We've now got like literally hundreds of testimonials, people saying this has been one of the more life-changing experiences they've ever had, which is not what we expected, you know, we would have been happier if people were like, oh yeah, this is quite good, I suppose.

But yeah, it's hard, right? It's not like, it's not like life-changing if you open the app. It's, it's life-changing if you put in the practice. Despite, like, despite the fact that, for example, I could prompt the same eval that I showed, and I did, and an agent in an IDE happily went off and did it all. But I think if you're pushing through that, like, avoid the shiny new thing and put in the hours actually typing the code, actually understanding it, stopping when you reach something, like Jeremy said, that you don't understand. It's the people who've gone through that, like the people who stuck around and stayed active and tried those things and put it into practice, they're the ones who come back and say, wow, that deliberate practice, that time spent, that's the bit that made the difference rather than the like shiny first impression.

Is it fair to say that what you showed today is just the tip of the iceberg? Because I think I— there's like a lot more that you can do.

Oh yeah.

Including, and it might not be obvious, but, you know, putting things into production.

And also you can, you can paste, you can paste a, you can paste a screenshot into it and SolveIT, the LLM can see the screenshot. There's actually a shortcut key which lets you, so you can create web components and if you want help getting the CSS right, SolveIT can see the rendered component in your browser. Yeah, you can create web applications, you can, yeah, I mean, it's, that's why we have like 30 hours of lessons and that only covers SolveIT as it was a year ago. So we have a new course coming up hopefully in a couple of months that will go further. But yeah, it's, it's a whole territory, you know, it's a whole, it's a whole new different way of thinking about software engineering.

It's not just a playground, like we can write real software with this.

Yeah, absolutely.

Okay, great. I encourage everyone to try this. I've experienced some flavors of this for a long time and I think it's a great way of programming, and I encourage everyone to explore it. Thank you for coming and showing us. You have made a lot of progress in just 6 months, or however much time that I've last seen it.

Thanks everybody for joining.

Thank you.

Yeah, everybody.