Does vibe coding actually work?

Okay, so everyone's talking about vibe coding and how you don't have to look at code anymore to program. And I decided, okay, I'll just give it a go myself on something non-trivial. So my test is to try to replicate some of the, some of the features from SolveIT, which is this really cool notebook product developed by Jeremy Howard and Answer AI. Try to replicate it in open source Jupyter to see, you know, if I, if I can, like how good this vibe coding is without looking at code. And so SolveIT, they have this like blog post that lists all of the features or like a good subset of the features. You know, it's about a month old, so there's probably some stuff that's not in here. But just to give you an overview, it's kind of this like really cool notebook that has, you can see it looks almost exactly like a Jupyter Notebook, but I have things like prompt cells. So instead of, in addition to like markdown cells and code cells, you also have prompt cells where you can talk to AI and the AI will respond. Let me zoom in a little bit more so you can see that. You know, you can ask questions and things like that. You can see here there's different kinds of cells, code, note, prompt. There's raw cell, for example. There's quite a lot of functionality here. You can see from the table of contents, my favorite piece of functionality is how this notebook system handles tools and variables. And so what you can do is you can define a function in a cell like normal, like you would do in a notebook, except anytime you have a function you can just tell the LLM to leverage that function as a tool. So you don't have to like wire it up to your LLM or anything like that. You just say, hey, use this code over here, uh, and call it. So this is a demo where like two functions are defined here and it's, and the prompt is, can you please generate the ID for the current user? You can use the tools and then has a special syntax where you give it the name of the tools. And then the AI will, will call those tools. So that's really cool. And you can do the same thing for variables. And the idea is like the notebook is kind of like this nice scratchpad. And you can also edit the outputs. You can edit the inputs and the outputs just like AI Studio and Gemini. You know how you can edit the outputs as well, which is really useful. You can do that here in a notebook. So it's, it's really cool. Um, however, like making this happen is, is a bit tricky and non-trivial if you're trying to do it like by hand and you have to know quite a lot. And it's something that I think AI wasn't able to fully do, uh, before. So what I did is I took a day yesterday, um, and try to vibe code this. And here's the GitHub repo, ai-jupyter. It's a JupyterLab extension. You know, you can just kind of like install it if you want and you can try it yourself. But let me show you real quick like what it does. So just to zoom in a little bit, here is a notebook and you can see here At the top, I'm importing the random library and computing this value for a. Now I haven't printed out a, so there's no way for the LLM to know what a is, except in this extension, the AI can see the kernel and it can get the value of a and saying, okay, the value of a is 34. If I run a normal code cell, So down here, if I run it, you can see the value is indeed 34. So that's really powerful because now you can compose your code and your prompts. They're not separate. They're like, it's like one integrated surface area. Same thing for functions. So I have this function here, xq7, which I imported. And the reason I imported it and the reason it has such an obtuse name is because I don't want to have the LLM cheat. I don't want the LM to know what it is. And what I want to instruct the LM to do is to use the tool to compute the result of 10 and 15. So I don't want it to like hallucinate. I want it to actually like have an awareness of what the tool is. So I'm going to tell the AI to do that. And it's going to call the tools. And it's going to tell me the result is 50. Now let me zoom out a little bit and let me verify if I actually do call that tool. Is it right? Yes, it's 50. And if we look at the code by using the double question mark thing in Jupyter, let me zoom in a little bit. You can see that's exactly what it does, x plus y times 2. And you know, it's the AI is able to look at the docstring of the function and figure out how to use it. Et cetera. So that's kind of that functionality. Let's look at the sort of the other kind of functionality, which is, hey, like using it as kind of more like a chat interface with like prompt cells. So like, so here we, I'm just going to have it write a poem about something. So I'm going to say, okay, poem, write a poem about snow. And it's going to generate a response. Interesting that it wanted to put that in a use the code block, but okay, that's fine. By the way, I can do like convert to cells here. And then it will convert that to cells, which is interesting. And then I'm going to say, okay, continue the poem you wrote using the same word for the first word of every corresponding sentence. Just to test if the AI is able to see all the cells above it, which is one of my biggest pain points of other AI integrations with notebooks. It's not able to do that reliably. So you can see, okay, you know, the words silent, then the, then children. Silent, the children is here. So, okay, sorry. They have the original poem and then they had the continuation one. So you could see like the continuation is this like silent, the, and then children. So it's like continue, you know, it's continuing. So that's really interesting. Like this would be very difficult to do. So this already is very powerful. I know what I showed here is like super simple looking, but even with just this composing these things, it's very powerful when you're trying to play with AI, like code and AI together, it kind of fuses it all into one environment in a very interesting way. Now, I don't, so the coding agent I use for this is AMP. So AMP is really interesting. Um, you know, at the moment AMP is basically using Opus 4.5. It's made by the team that created Sourcegraph. Um, I really like it. I'm not going to get too much into AMP. You can read it. The one thing I like about AMP is it saves all the threads. Um, you can save it in the cloud and share it publicly, which, which I'll probably do in a bit. One thing I'll say is like, even though I can do this, make this, do I think, I don't think it obviates this product of SolveIT. You can see like SolveIT has a lot of features. They have a lot of like, they have like sharing and, you know, all these other things that you might want. And yeah, certainly, you know, you could, you might be able to build that out with AI, but there's a lot going on. Besides just code, there's support, there's maintenance, stuff like that. Do I wanna maintain the tool? Do I want to constantly, you know, upgrade things, test things? Probably not. I would, you know, still gladly pay for SolveIT if I was trying to, you know, if I need to use something like this. So I don't think it kills Sass, but It is interesting that, yes, it seems like AI coding or this, like, vibe engineering works. I will say that I babysit it heavily the entire time I did it. And I had it constantly write tests. And I pointed it at context. So I gave it the blog post as a spec and then also pointed AMP at the Answer AI public GitHub repos. And then also had it look through a bunch of other Jupyter extensions and plugins and things like that for patterns and had it constantly refactor, rewrite, clean up code and constantly write tests and add to tests. And also, I didn't look at any of the code directly, but I looked at the diffs as they were streaming by. And I could see like fishy things sometimes and I would stop it and say, hey, why are you doing that? You're going down the wrong path, et cetera. So I was definitely babysitting it mostly for like, you know, the 8 hours that this took. However, I'm fairly impressed that this works. This would not have worked before on prior models. I tried it many times. It did not, would not get this far. So it is quite interesting. You know, I didn't do anything too crazy. I had it plan its steps first, and I had it then, after planning its steps, I made some very minimal changes and had it then execute those steps. And then along the way, I just had it iterate and kind of babysit the iteration and sort of use the UI a bit. One thing I can say that really made a big difference is skills. So I gave the AI its own skills. And there might be in here. Oh yeah, there might be in this toolbox. So in this toolbox, you'll see there's these Jupyter testing skills that I gave it. You know, there's these things I discovered like Galata, which helped. But these are like specific testing tools that really sped up the iteration cycle to narrow down like how you test this extension. And that gave— I felt like that helped unblock this whole process. Again, you know, I'm really impressed. I was a skeptic before I did this. But I can clearly see that it works. I've been using this extension for a few days now, yesterday and today, and it works. And Some of my friends have been using it already to create things. So it's really interesting. I will say as a warning, this is not something that I'm trying to maintain as an open source project or for the community or anything like that. This is an experiment for me to explore the limits of vibe coding. And so, you know, if you're going to use this, use this at your own discretion, you know, feel free to edit it. I'm not going to maintain it necessarily. This is a tool like for myself. But yeah, just sharing. I think there's something to it. I think there's definitely something to this vibe engineering that it can get this far just by guidance. So just sharing. Hope that's interesting. Thank you.