Build Your Own Eval Tools With Notebooks!

tons of times.

Yeah, I can share my entire screen.

Like the main thing, you need to have this URL, like this backup URL and the stream link basically, and that you can get from YouTube. So if you're on the YouTube thing— oh, we're already live.

Oh wait, I think I lost you. Okay.

Well, I think we are already live, but that's not—

yeah, we are live. Yeah.

Okay, so hi everyone.

I mean, if anyone's watching, uh, I don't know.

Yeah, it's like the people, I mean, people are, people are early. Like Dutch people tend to be early.

You know, you should, I wish you could stream directly to Twitter, like in the Spaces. If you could like stream directly into there.

Uh, you want to stream to all the places having one single button, just stream everywhere.

Yeah, yeah, yeah, yeah. Is there, is there a thing? Do you know?

I've never really tried it. I've only done YouTube myself. Like, the one thing is that all these platforms are slightly different. So like, if you have Twitch, then there's like this sponsor thing that you can hook into, and then people can pay you as you stream. And that doesn't work the same way on YouTube, doesn't work the same way on Twitter. So then all these experiences are kind of different. And like, there's a middle ground, but then you go for the middle road, and it's like a master of none kind of a feeling is what I'm what I've been told. I see.

I'm going to turn on my light real quick, one second.

So the auto camera is following Hamel Husain turn on the light. It's funny how the camera is also following you as you walk to the door and then to the light.

Yeah, I mean, it's all this AI stuff, right? Like, yeah, it is, uh, trying to be fancy.

Um, but how's— because you're, you're not in Silicon Valley, right? You're in Oregon, if I recall correctly.

No, I'm in Oregon. I'm in Silicon Forest, they like to call it. But, uh, there's not really any silicon or anything going on here. There's not much going on here in terms of tech. Um, but yeah, it's cool, it's peaceful, you know.

I can also imagine it being somewhat of a relief. Like, one of my So I live in this, this is kind of like a working class neighborhood kind of where I live. And there's this one guy who's single and whenever it's like nice weather, he has a bench and he just puts a crate of beer there and the entire street just kind of goes for a pint basically. But that guy's a painter and he has never been on LinkedIn in his life. And part of me does kind of go like, oh my God. I know. But also like when he's hanging out with his buddies, like they're painters and plumbers and stuff and they're cool people, but they know nothing about tech. Like the guy bought his first laptop 2 years ago when he became independent. Just to give like an overview of the crowd. And I gotta say, oh man, I love being able to unplug from tech, like just hanging out with those guys for a bit. Something about that feels very healthy.

So where do you live again?

So, you know Amsterdam, right? Yeah. Between Amsterdam and the beach, there's a little city called Haarlem, which is also, by the way, why Harlem in New York is called Harlem. It's basically named after that city. It's 300,000 people, it's close to the beach. It's a cool place to raise kids, but you're still close to Amsterdam. So if people visit, you can always say hi. I think that's kind of the sweet spot for me.

And you have kids?

I've got a little rascal named Kai. He's turning 3 soon, which is a really cool age, but it's— oh my God, he touches everything and things can break now because he's kind of big.

Yeah. I'm gonna actually like be right back. Sure. Use the restroom real quick. I'll be right back, one second.

Sure. I'm gonna show off my gear to people who are tuning into the livestream. Fancy keyboards and all that. Actually, let's see if I can actually go to that livestream now, 'cause then I might see people comment. That might actually be kind of interesting. Oh no, that's different. That's vanishing gradients. Oh yeah, build your own eval tool. Okay, I'm in the YouTube. Yay, that's kind of funny. So hi everyone, feel free to leave a comment if you want to say hi and stuff. We definitely started early, like Hamel just figured out the stream button and that's why we're kind of early. But we're going to start in like just a bit. Um, while Hamel Husain Husain Husain is away, people can ask me questions about the setup that I've got here, but we're going to start in like 5 minutes.

Uh, all right. Hello. I've been looking forward to this, by the way. Like, this is a different beast because I don't know, like, when I thought about this, uh, this talk, I was like, oh, this sounds like so much fun. Like, I'm gonna have fun right now. Which is not always like, sometimes it's, there's different variations of fun, but like this idea of like hooking up a, I mean, thing to a notebook just sounds like so much fun.

So besides the whole like, oh my God, Vincent found a way to get engineers to expense their notebook, like their, their gamepads and like that, that's awesome. Like that, that's definitely a part of it. But like part of it is also, and that's also why I wanna start with this scene, like I've had RSI issues and some of that was prompted because I was annotating. A lot of mouse work basically, and I really injured my wrist in doing so. And part of me does think this gamepad, as far as ergonomic goes, it's pretty brilliant if you think about it. Like it really fits in the hand, like the shape of your hand. So not annotating with something like this also feels kind of crazy if you think about it. So that's also like, there's also like a serious reason why I investigated this, to be honest. But yeah, the fun bit is definitely also part of it.

And So I brought my controller as well. I have a PS5. I don't know if that'll work. I have a USB cable, USB-C cable. It's kind of long, but whatever.

Yeah, so I mean, you can, um, yeah, you should be able to play around live. Uh, that should be doable. Uh, one thing you do want to do is plug it in, I think, and just check like your Mac settings if it can detect that Bluetooth device. That's sometimes an issue. I think in general, if you hook it up, then the Bluetooth is no longer a theme, but I don't own a PS5, so like it's sometimes a fair amount.

Oh, I own it, but I don't ever turn it on because I don't have time for it. Uh, I don't even know why I own— I have it. It was like I played one game and then that was it.

It's, uh, I mean, the annotation game is a fun one in general, but, uh, we are going to start soon, so I got us OBS in front of me. Yes.

And, um, do you have a hard stop? Um, I don't—

oh, I don't. So like, I can go on for a bit longer if you—

I can go on.

Oh, sweet. Okay, then I can, uh, I can show off more stuff.

Yeah, yeah, please do. Please show off all the things. Uh, okay, I'm gonna turn on my controller. I've never tried to— ooh, it says allow accessory to connect. Okay, allow.

Let's go for yes.

Okay, and now what do I do?

So, if you go into Google and type Mopad GitHub and then coning, like my alias—

Mopad GitHub. Oh, wait. It's under your—

Yep. So, it's this repository that you want to go to. Because Marimo is able to deploy to WASM, I can also host a Marimo notebook with the widget on GitHub Pages. So, you can Go to a page like this, hold on your gamepad, move things around, and events are being detected. That you should be able to do live right now if you wanted to. We're going to come back to this.

GitHub.

Yes. My alias is koaning, K-O-A-N-I-N-G. If you go to github.com—

Your screen is very low resolution for some reason.

It could be because you're looking at the webcam. If it's an issue, I can also just share my screen, by the way.

Yeah, I can't see the text. I can barely see the text on your screen.

Okay, that's not good. So I'm gonna hit share screen.

Open—

oh, it's one of those open system settings kind of a thing. Okay, fine. Zoom. I'm gonna have— yeah, I'm gonna quit and reopen. Be right back, one moment.

No problem. I made some of my friends co-hosts just in case. I always do that, so don't be alarmed, Jono or Isaac, that you're now co-hosts. It's just, I just do it in case my connection drops, uh, the meeting doesn't end.

Um, okay, so, um, we no longer have—

there's something. Uh-oh. Oh no, no, you don't have to prepare anything. It's just—

I'm the guy who has to prepare something for this meeting. Okay, great. Uh, requesting bypass. Okay, allow. Okay, so I think, um, if I'm not mistaken, you should see a higher resolution screen now.

That's better, yes.

Okay, now, okay, so shame that the OBS virtual camera doesn't work. That's a bit of a bummer because we can't do a super fancy thing, but, um, as long as people can see what I'm seeing I would argue everything is still good. Is there a way for me to see what everyone else is seeing?

How are you switching between screens? So are you doing this in a video sense? Are you doing this in like, like just with your OS? Like, it seems like you're doing some video fancy stuff right now.

So I do have a bit of a setup at the house, but it's also partially because I've got 3 screens. I've got, I've got a drawing tablet pen thing. And like, I can actually— let me start by showing that actually. And then we can get started.

Yeah, this is super interesting. Yeah, I know it's a diversion, but this is interesting. I think probably for everybody.

Yeah, hopefully. I mean, it's a thing where— okay, let's— I'm gonna hit video. And then I'm gonna hit this screen that I've got on top. So I'm also going to stop sharing my screen. And stop share. Okay, so hopefully people can see this is my desk. My desk is a bit of a mess. Yes, it's a mess, if we can get over that. What you see in the middle here is a split ergonomic keyboard. This is one of those keyboards that has like a key well attached, and that means that your hand can nicely fall in. It also means that your thumb has more than one button to touch, which is super duper useful. I can map one of these keys to be like my Mac whisper or whatever, but you can fully program these keyboards. I love these things. I have a YouTube channel where I review these, but both my hands are split. This keyboard comes with a trackball over here, which is nice. But in the middle, I've got probably the most important thing that I'll use today, which is the drawing tablet. This is a $40 Huion drawing tablet. It's the cheapest thing you will ever find. But I've got an app called Screen Brush that allows me to draw over something that is on display. So I can hit a shortcut, then Screen Brush turns on, and I can draw over something that is on screen. And I'm just gonna now start sharing my actual screen again. So I'm going to go to desktop number 2 over here. This is the screen that we will also be using in a bit. But I can now say like, oh, by the way, look at this. I'm pointing an arrow at something. And oh, by the way, here's an important detail that you shouldn't be missing. And especially during COVID I was just always so surprised that I was the only one in my company usually that made this investment. But especially if you're working asynchronously, yeah, you can write something up. But what you can also do is like make a 2-minute recording where you hover over the thing that's important. And also for educational videos, just this way of working is just so much more clear. Please, people, buy these things.

Like, I'm gonna buy it. I don't have one.

It's a $4, like, Mac app you gotta pay for. It is the best value you will ever get. And I would love for more people to do proper education on the internet, which involves buying a little bit of gear.

Okay, what is it? So, uh, I—

you can buy a Wacom. The brand is not necessarily too important. The main thing that's important is this one app. I can actually draw it in display now. So this is what it looks like. There's just basically this overlay thing, and I can just say like, oh, let's draw over something. You can also draw boxes. I can also hit F, and then what I draw just is on screen forever. So if I want to do things like whiteboarding, do a little bit of diagramming, Whenever I livestream, I do this all the time. Um, like, please make this like $5 investment and all your videos will be better and the world will be better for it.

Okay. I'll collect the links, Amazon links. Uh, I mean, not affiliate or anything, just the Amazon links for everyone's benefit and like put it in the notes of this talk. Uh, so, so don't worry.

Uh, there's a, there's also a calm code course I made like 5 years ago about how I make these videos and that people can also just follow that. That also has all the links in it.

Oh, okay, you have all the— using the same gear from that video, the Comcode one?

It's, I mean, it's— I used Wacom tablets before then, but it's just like in data science, the tools don't matter that much. Eventually, it's just a tool. So buy a brand, whatever you like, and then just focus more on adding context to your videos. That's the important bit. Anyway, I do want to make sure, like, am I— is my face on display now as well?

No. No, I see your desk and your screen.

All right, so then I have to, I think, select this one, and now my face is back, right? Yeah, good. Okay, um, people can also see this, uh, see this notebook over here, I think, right? Yeah, good. Then, um, unless you want to do something of an introduction, then what I would like to do is maybe tell a story or two.

Let's go right into the story, right?

So, um, I started in data science like I don't know, 10, 15 years ago, maybe. It's been a while. And nowadays when people talk about evals, people say like, look at your data, look at your data, look at your data. It's a good lesson. People should definitely do it. But the thing that always struck me that's interesting about it is that I had been doing that for ages. 15 years ago, it was also a problem. And the best way to explain the problem in my mind is to also talk about this one dataset that I tended to use for teaching. This is the ChickWeight dataset. For those of you that remember, this came with the R language. It's like one of those datasets that R came packaged with. And the dataset has weight, it has time, it has a specific chicken. So different chickens are eating a specific diet over time, so to say, right? So this chicken had this diet, and as time moved forward, this chicken started gaining some weight. Like that's basically the story. And there's lots and lots of these chickens, right? So as I scroll over here, you can see lots of these chickens basically. And note, by the way, what we're looking at here is a Marimo notebook. Whenever you output a DataFrame, we get a nice interactive widget with lots of bells and whistles. There's a button over here, we just added it. So what I could do is I could explore, let's say the diet column, there's all sorts of descriptive statistics. There's like pretty charts and stuff, especially if you've got a time series, this is very nice. But of course, we're not gonna like stare at a DataFrame like this. What you typically would do is you would make a chart. So if you wanted to maybe summarize the data a bit, you would do something that looks a little bit like this. You would basically say, like, look, for every chicken over time, we've got this little dot. So we've got time on the x-axis over here, we've got weight on the y-axis, and then for all of these different diets, we can also make like an average line. So we can see that there's this diet 3 over here that seems to be doing quite well compared to some of these other diets. And this is just doing summary statistics. You could also apply TensorFlow or scikit-learn if you want to do something fancy and predictive. But what a lot of people would do in data science is they would sort of look at this and go like, okay, diet 3 is best, business case closed. And we have a chart here that shows us that that's accurate. And I would argue that, you know, this is a common practice, but what I've just done is not looking at your data. What I've just done is staring at your data. Because the number one thing that people always forgot to do whenever they were looking at a chart is that you shouldn't just look at it and then move on. What you should do is you should actually force yourself to look at the damn thing for like 5 minutes. Because 9 times out of 10, something weird is happening, but you have to look for it. In this case, because we're using Altair, we can actually do a cool thing. We can add interactive on the chart object itself. When we do that, we actually get a chart that we can move around and we get a chart that we can zoom into. When you do that, suddenly something interesting happens because it seems like there are some chickens over here. That are actually losing weight. And, you know, presumably this is like a dataset with chickens from like a feedlot, so that shouldn't really be happening. And if you squint your eyes, you can actually see another instance of that here. There's a— I don't know exactly, you can't see right now if this chicken eventually lost weight or if it was a different one, but you do see that there's a point over here that's lower than anything that happened before. So there are also chickens that are losing weight. And again, that's just kind of weird. That should maybe not happen. Like, that's kind of a smell there, right? And whenever you smell something, you should chase it. But one lesson here is you can only chase a smell if you take the time to sniff. And if you took this effort, like, if you noticed this one phenomenon, you could make a different chart, which is this one, and then you could outline all these different chickens. And then you would definitely notice that some chickens actually died prematurely. So there's one chicken over here that died prematurely, there's another one here, another one here, and like 5 chickens out of this entire group actually died before, let's say, the final date.

Hmm.

Given that, what does this line actually represent? I mean, yeah, you're going to calculate the mean of all the chickens over here, but you're going to not count the 5 chickens that died prematurely, therefore this line doesn't matter. And the only way you've got— and I want to stress this— the only way you've gotten that conclusion right is if you actually looked at the data and you didn't stare at it. And like, that's a lesson I think that we should remind ourselves of also in this day and age of LLMs. Like, it's very easy to come up with a flashy demo, but you got to be very critical. And maybe what we got to do is come up with tools that help us. So back in the day when I was teaching this for the first time, we had static charts. One thing that's really cool about Altair is nowadays we've got these more interactive charts, right? So I can actually pan and zoom in and stuff. But interactivity is only part of the story. And what I want to showcase today is how reactivity is actually the cherry on top that you might want as well. Before moving on to that, though, I just want to make sure— Hamel Husain, any questions about this?

Yeah, I mean, I could—

I see you thinking a bit.

Just out of curiosity, like, um, is there any intuition behind the design decision to put the plot above the code? That's interesting, right?

Like, oh, right. I mean, that's a setting. So to be clear, there's a setting. Okay, figure this. Well, one thing that's kind of unique about Marimo, uh, and I'll— let me explain that first. So we're now going to go to part 2 because that's, that's the bit where I can maybe explain that. So these are 3 cells, and like, if you were doing stuff inside of Jupyter, let's say, right, like you would run this cell, you would run the next one, you would run the next one, like Shift+Enter, you know, that shortcut. And you could then also maybe do something like, I don't know, a minus b plus c. You could have a cell like this. But the thing in Marimo is we can say, well, the order of the cells don't really matter.

Can I stop you for one second? Because you did something that is on my mind. You have these, like, notebooks in, like, a continuous canvas or something. What is going on?

Oh, I've got columns.

Oh, you have columns.

Yes.

Okay. So, you can do that. Is that a— so, that's like a Marimo thing? That's like a unique—

It's one of the many Marimo things. But yes, that's the setting you can add. So, I don't want to press the button now because the whole screen will move. But you can have like very wide cells or very thin cells. Some people like the linear thing. I really like to have columns because then one column is typically like boilerplate setup. And then the next column is like slightly more interesting stuff. And then, you know, you can have levels of abstraction. Kind of logically, right? But this also kind of falls into the category of, well, the order of cells doesn't necessarily matter. I could move this into another column and the code would still run. And the reason why is because we do a little trick. So imagine that you're the notebook and you have a cell over here that says, I wanna calculate a minus b plus c. Then I know that I need these variables. And that means I gotta start looking for cells that have those variables. And I'm just gonna attach some names here. So this is cell 1, this is cell 2, Roman numerals. This is cell 3, right? 4. What I could do is I could say, look, this is kind of like a DAG. I have cell 4, and in order for that to run, I need this cell to be around, I need this other cell to be around, and I need this other cell to be around. And basically, before anything runs inside of Maremo, there's this pass over all the code where we try to figure out in what order the cells have to run. Because if we know it up front, then we can also do some clever things. We can say things like, oh, let's say the DAG was a bit more complex. Let's say there's a node over here as well, maybe another one over here. Then we could do things like, hey, if I make a change to this one cell over here, then these cells over here don't have to run again. We can kind of be a bit picky about that, and we can save some compute. And one interesting consequence of this is that it's also fully reactive. So I can change this one cell over here, and then the rest can just automatically update without me manually having to run the cells. Now, to be clear, you don't always want that, because if we're dealing with a big compute thing like giant PyTorch loop or big DataFrame operation, then you actually want to have manual control. So what you can do is at the bottom over here, we have this on_cell_change autorun. You can change that to become lazy. And then if you were to make a change to a cell and maybe run that, then you can see the color of the cell changes to indicate that this cell needs to be manually fired off because it's dependent on a cell that just changed. And this way you can manually change all the cells if you want to.

What does the on module change mean?

On module change?

Yeah.

Oh, if I have like a Python file, locally that's like a module that I'm building or like a package that I'm working on and I'm importing that inside of this notebook.

It watches the import?

Yeah.

Got it.

Then sometimes you want that, sometimes you don't. But we basically give you control here. The main use case I got to say for this really is the whole, oh, I have this one cell, it's like a big heavy compute thing. I don't want that to automatically trigger all the time and maybe for some workloads it's actually a lot better to just, do that stuff manually. That's mainly what the setting is for. But one cheeky thing that's kind of interesting at this point, like if we understand that we, we do this forward pass over all the cells and we try to figure out what order things have to run, there is this other thing that we could do. We could also introduce some UI elements. So I could say something like, oh, there's a slider here. So it's like starting value and let's set steps of 1, something like that, right? I can introduce a slider so that thing can slide around. I can now say, oh, that value b over there, that's equal to the value of that slider. Now as I move the slider around, we can also see that this value over here updates. If you squint your eyes, you'll notice something over here change. When I hover over, you can see how long each cell took to run. But as I move the slider around, you can also see the order of things. When I trigger this cell, this cell will update, afterwards this cell will update. So this whole reactive thing is basically happening on your behalf. And this is kind of special, right? Like the fact that these cells can run automatically, that is definitely not something that I was used to from the Jupyter way of thinking about notebooks. But this is, if you think about what makes Marimo special, a lot of it really does boil down to this. The fact that we do this forward pass, that we figure out the DAG, there's a, It's kind of a nifty UI here. It's like explore dependencies. And like, there's lots of cells in this notebook, so it's kind of big, right? But you actually have a map of like all the cells and what stuff depends on the cells and where we do the import. So you could also explore that if you wanted to.

I see.

I have a lot of questions, but I don't want to take you up on it. So this might be a good time to ask them, actually, because I'm going to go into an example after. OK. Because I did tell you, tweet this one time, that there's a consequence that you're not allowed to do this. So you have some library, mohtml. You can't do this. And I remember you tweeting about this, right?

Yeah, yeah, that one, that one caught me pretty— but I think that's an okay compromise. I mean, you know, well, it makes sense.

There's a reason for it, namely that this a is a global, and that makes it easy for us to figure out like, oh, everything's kind of global, it's explicitly defined, and therefore we can define the order of the cells. But there's also like another thing that trips people. You also can't do this. Hmm. Because what's the value of a in this cell? Do we have to assume that all of these updates happened? Or is this meant to be running after? So in-place variable updates is something that we actually prefer you not do, because it makes it very hard for us to figure out what the order of the cells are. There is a benefit to this though, and this is something that definitely bit me a few times inside of Jupyter. What would happen if I were to remove this a variable declaration, right? If I were now to run this cell, an error is gonna happen immediately because the representation of the cell has to also be equal to the representation in memory. And that's the thing with Jupyter that can catch a lot of people by surprise, if you accidentally delete a cell, that doesn't mean that the variable has been removed from memory, if that makes sense. So this is also an enforcement mechanism for consistency, if that makes sense. I kind of interrupted you because usually these two facts are the one thing that sort of confuse people the most, but now that I'm done talking, let her rip. What are your questions?

Yeah, tell me more about the AI coding feature. Can it see all the cells? Can it see the markdown cells? Can it see everything? What does it see? Does it see only the cells before it? Does it see after it? Can you tell us like more experience?

Yeah, so we have all the variables, right? So these are defined. So these are like variables. We have them in sight. We also have these data sources. I've only gotten data frames here, but you can like hook up Postgres or Iceberg or Redshift, that one's new. That one was added last week. But data's like, so we've got variables, but then data frames or tables are kind of like special citizens. And one thing you can do is you can also chat with AI. So I could do something like, hey, what are the columns in, I think I called it chick data frame, yeah? So it's an in-memory data frame, it can just detect that. And this gives us a way to say, oh, there's a data frame there, there's a schema attached to it. Let's add that to the prompt. And this is using Claude for Sonnet. And you know, it can see, like it's able to inspect some of that schema information basically. And on our end, what we do is we basically make the prompt a bit bigger. We're able to inject like some knowledge that might be useful. That's something that we can do here. Like a couple of output things happen here. You can add that to the notebook or you can copy it. There is also a Copilot feature you can turn on if you wanted to. And another thing that's also good to know, if you go to, there's an AI section over here where you can add your keys and all that. But one thing you can also do is like add custom rules. And basically, I prefer polars over pandas these days. And whenever I make charts, I really like to do that with Altair. And this is just going to be appended to every prompt that you send out to an LLM.

And does the Copilot in the notebook see all the cells in the markdown? Like if you wanted to just stay in the notebook and not go to chat, let's say.

I would have to do a little bit of a range check. Yeah, okay, I don't feel like doing this right now. So I think the tab completion thing that we've got, that only looks in the current— I don't know for sure, actually. I don't know if the— so we have two settings there. So one thing I could do is Shift+Command+E. There is this Generate with AI element that we've got here. And this one definitely works only on this one cell, I I think, but I could be wrong on that. I would have to check. I need to check the code for that, to be honest. I do know for sure that this one actually looks at the entire notebook and you're able to explicitly pass on schemas. Got it. I don't know for sure about the tab. Like, this is like GitHub Copilot tab completion stuff that you can get within a cell. I just don't know for sure if we— how much we look outside of the cell. That's something I prefer to be right than wrong. So I'm not going to answer that question.

No worries. No worries. I understand. Um, can you tell me more about the storage format of the Maremo notebook? Like, how is it stored?

And then it is a Python file. Okay, so, uh, one benefit of having a Python file is the Git story is just kind of a bit nicer. Like, it's a Python file, so you don't have to worry about the JSON stuff. Another thing is that you've got the cool UV benefits. So anything, any benefit that happens to like normal Python files, we also get. So you can add this I think it's like some— it's a PEP, so pip can also deal with this, but you can add this basically to the start of your file. You can say what version of Python it needs and what dependencies we want, and this makes the notebook pretty much self-contained. Like, you can, you can run this in a sandbox command. You do marimo run sandbox or marimo edit sandbox, and you can just install the dependencies right on the fly. The thing that's interesting, if you look at the cells, um, you can actually see that this is a cell that needs a couple of variables And you can also see sometimes that a cell returns a variable. And this is the, basically, the reactive graph thing in action as well. So, whenever you make an edit and we save it, we have a decorated function where, oh, this cell needs pollers around, oh, and this cell emits a chicken data frame. And every time we decorate one of these functions, we check like what's the input signature, what's the output signature, and that way we can also construct the order in which all these functions have to run. Another benefit of having it in a Python file like this is it's kind of like a FastAPI app, right? So we decorate everything and there's like, we've also got SQL cells by the way and Markdown cells and like we've got a lot of stuff. But at the end, you are able to do if name equals __main__, app.run. So you could also choose to run this notebook by saying, look, just Python name of notebook file and it'll just run as a Python script. So what you could theoretically do is you could like, I don't know if you like, I prefer Typer, but you can also define a ClickApp in here, and then your notebook can also have a self-contained command line application that is still in the Python file. So you can also use, um, Marimo inside of CI if you really wanted to.

Um, what happens to the outputs? Like, let's say visualization, something like that.

Let's say you So we don't store them. That's the short story. We don't store them. But what you can, of course, do is we do have— you can download this stuff as Markdown or as HTML. So if you have static outputs, that can definitely be nicely stored. Another thing you could also do is we have Pyodide nowadays. So what you could theoretically do is the outputs are still not stored, but you could also say, well, assuming you don't have any weird Rust dependencies in here that Pydite can't handle, you can also export this notebook to a static HTML file, which you can then host on GitHub Pages. So for lots of my new packages, I basically host this notebook on GitHub Pages, and that can also be like an export target if you want.

If you, uh, if you export a Marimo notebook to like, let's say, let's say you have a report that you made, all kinds of visualizations and whatever, you investigated some data, you sent, you want to send it to your colleagues, um, You know, just going to open up a, uh, an example. Do you have to export to HTML? Is there any way to send them the notebook with everything in it without them having to rerun it to get the same visualizations?

Um, we— I, if I'm not mistaken, yeah, that is from the— we need to run that from the command line. Like, we have an export command. I think you could also, uh, let's just grab Terminal. Yeah, uh, Maremo export. Let's just grab the help. Yeah, so you could run it like HTML would kind of be the safe bet. It just, um, if you want to export an Altair chart, for example, that'll be fine because you get everything that's just a static HTML file with some JavaScript and it's self-contained. But if you need to run some Python code in order to also update the chart because you're connecting to some sort of API or whatever, then you can hook it up to this WASM backend. But that's because you need some live Python to also run. You could also choose to export it to a Jupyter file if you really wanted to. We also support the IPython notebook, and then you can render that as you would normally too, if you really wanted to do that.

Okay. And can you go in the reverse direction? Like, you can import— nope, like, can you import HTML into this, like, HTML notebook into Maremo, or? IPython or how do you interchange?

I would have to double-check. The main thing with Jupyter that you have to be a little bit careful with, it could theoretically be that you have A equals in multiple cells and that's something we don't technically support. You will always have that. That's something to be aware of. But in general, the main idea behind this format is we don't store the actual outputs. You can do that by doing some sort of an export. But the main benefit of doing it this way is just the whole Git story is just so much nicer. That's the, I think, the main reason why Akshay, the creator, went with this style. I could be wrong on the history details there. But I think Akshay is actually in the YouTube maybe as well, so he can give details.

He's not in the Zoom, but yeah, maybe he's in the YouTube.

Um, oh yeah, Marimo, there you go. There's also a setting to automatically snapshot the notebook as IPython HTML as it runs. Actually, he's in the, he's in the YouTube.

Sorry, what did he say? There's options to—

there's also a setting to automatically snapshot the notebook as you run. So let's see if I can find that. So that might be like a setting here, like a user setting. When enabled, Marimo— here you go. When enabled, Marimo will periodically save this notebook in your selected format, HTML, IPython notebook, to a folder _marimo next to your notebook file. So if I select this and then hit save, and if I then go, uh, here, then yeah, we see the bulk HTML appear. So this, and this is the HTML that contains everything. So there is a setting that allows you to, as you are working, oh, all right, and then this is also an interesting thing. I need to install this dependency in order to also store the IPython notebook. Let's just have that install. There's a packages tab. That's also kind of like good to mention. So you don't have to do the annoying exclamation point pip install package name thing. Like we've got a a dedicated sidebar that allows you to install packages. And also, if I'm not mistaken, if I were to do something like import sklearn or, um, I don't know, some other, like, Torch, if I were to do that, right, is that also around? Okay, Flask. Yeah, whenever we see an import that isn't in the virtual environment, we can automatically detect that and we can immediately add it to the virtual environment. So that's actually something that just happened. Let's just save now, double-check the user settings. Okay, it should now save this as an IPython notebook. The dependencies are now installed. And if I were to now check here, we see that bulk.ipynb got saved. Yeah, and no renderer could be found for this MIME type. I think that's a VS Code thing, but the charts I do think would appear here if you were to add this to GitHub, if I'm not mistaken, could be wrong. But I would imagine that anything that You do get the Marimo slider stuff that won't render, of course, right? But I think if you were to add static matplotlib stuff here, it would be attached to the notebook. I think we can open this, open with, and then we can say as a text. Yeah, so the outputs are stored here, right? So that's something you can see. So if you really, if you really did want to store your like visual artifacts, you could, but Marimo chooses to just kind of forego that and have a clean Python file instead. That's the main vibe, at least, that I would describe it.

Okay, great. Yeah, that was really helpful to get that.

It is different. So the main thing is, it is different than Jupyter. But for me, at least, the main thing that kind of took me by surprise is the fact that like, oh, it's the reactive nature, and things can automatically run, and that has consequences. But the moment that you grok that, you can infer how everything else is supposed to run, but it is very much unlike Jupyter. If I were to, from my experience, the main thing that I miss from Jupyter that I did think was neat is, you know how you could be inside of a cell and outside of it? Mm-hmm. Those shortcuts just work differently because there's not really a notion of being inside of a cell or outside of it. A good consequence of that is that I do get these columns, but because we don't necessarily have a first cell and a last cell because it's all a DAG, That's also why a lot of these keyboard shortcuts that you're used to, that's also going to be just fairly different. Like, that's the main thing. That's the first half hour. Uh, oh, my fingers don't find the right shortcuts anymore. That's the main pain point I experienced when I tried it the first time.

What do you mean? I'm not sure I track the inside. I mean, I understand you're inside a cell and outside a cell, but would— I'm not sure. It seems like you're inside a cell right now. You're inside that first cell.

Oh, yeah, yeah, but, uh, usually, so if this were Jupyter, right, I could hit Escape. And then I'd be kind of outside of a cell.

Yeah.

And then you will be able to go through all the different cells. Let me put it this way. That was a shortcut that I used all over the place back when I used Jupyter for like quick navigation, basically. One thing, we also have a command palette. So like one thing you could do is you could go for like show the keyboard shortcuts. And then like all these shortcuts are things that you can also manually change. But these shortcuts are like pretty different from Jupyter. Basically because the way that cells interact is also very different from Jupyter. So this is, this is the main sort of productivity hump to get over, I might say.

Okay.

So, okay, having said all of this, like I've explained—

so is your main, like, the, the reason that you personally like Maremo, is there a certain feature that kind of, you know, takes up most of your— I mean, is, is there certain like flagship feature? Is it the reactivity? Or is it just like a whole bunch of them together?

It is, it is definitely. So you've got these reasons that pull you in. And then there's like the holistic picture that makes you stay. That's kind of the way that I would experience it. So for me, the AI thing on its own would not be a reason for me to start using Maremo. The way it does package management on its own would also not be the reason for me to use Maremo. The UI elements also for me, they're interesting reasons for me to try them, but it's not for me to stick around. However, when you add all of them together, and when it kind of becomes a holistic ball, then suddenly I become super duper productive. And I think the main moment for me, at my previous employer, I was doing a livestream, and I hit a bug during the livestream that was caused by, I deleted a variable that was kept in memory, I had a cell that still had that variable around, but it wasn't representative of what was in memory, like something along those lines. That will just never happen in Marimo. Like there will be a big red error in my face whenever a moment like that would occur. So, like, there was this one time when Marimo actually corrected me live. And that was a moment for me where I did start to think, okay, this is nice. Like, I like tools that correct me while I'm in the flow of it and not like 5 minutes after, when I'm trying to debug something.

That makes sense. I can see already for teaching, it's kind of interesting. I already really— I mean, the column thing is really good.

Yes.

For this, like, what you're doing right now, which is like teaching us.

Um, and let me give you another thing that's great for teaching then, since I have you. But, um, before getting there, I'm gonna have to explain a little bit of theory because I'm about to show you like a pretty complex visualization with like a couple of steps in it. It's a trick that I've used over and over again in the past. But let's say you've got a big dataset. This is a picture of a database, and we have lots of texts in there. And let's, for keeping things simple, let's say that this is customer support data, right? It could be that we are interested in a very specific phenomenon. Maybe we're interested in things like, oh, customers might be complaining about deliveries or something like that, right? And one thing I could do is I could type the word delivery, and I could do like a basic text search, and that wouldn't be a, bad idea. But there's a lot of these like delivery adjacent words and things, right? So instead what you might want to do is maybe do something with vectors instead. So one thing you could do is you could kind of build the very standard vector lookup thing. You take every single text, uh, turn that into an n-dimensional vector, and then you could sort of do the lookup. But the thing that I usually really like to do instead is I like to turn that high-dimensional array into like a low-dimensional array, preferably two-dimensional, and then I like to turn that into like one of these big scatter charts with UMAP, basically. It's a trick I think people have seen before, right? If you take lots of these texts that have clusters in them, if you pass them through UMAP after doing some sort of embedding, then you tend to get this 2D chart where there's just lots and lots of different clusters, basically. And if I'm then like— one thing that can happen—

just to clarify, UMAP is a clustering algorithm.

It's a dimensionality reduction technique. It is—

oh, dimensionality reduction. Okay, it's been a while.

Yeah, so it's, um, the main thing to think about, the main way that I'd like to think about it is PCA is really good at, um, keeping the global structure intact, but that does come at the cost of local structure. Like, it's kind of hard to find clusters when you do PCA. But then what UMAP does is it basically says, oh, but all the clusters that were there in the high-dimensional space, they persist in a low-dimensional space. It's just that distance between clusters no longer means anything. That's kind of the, the balance if I were to put it into words. But the thing that I'm about to show you is I'm about to show you a system where I can type a query and that is going to determine the colors that appear in this chart. And that I'm going to show you how that will just help me find data points that I'm interested in maybe annotating. And this is also like a perfect example of something that would be super hard to do if we didn't have something that was properly reactive. So, like, one thing I could do is, like, this is the, the Altair chart that's decorated with—

and now this looks so familiar now. I feel like you've been working on this derivative of this specific thing for a while. Even before Marima, I've seen you hack around in this area.

I actually— so I spent years building, like, standalone web apps and, um, like, Jupyter plugins even to get this workflow to work nicely for me because it's a very powerful workflow. Like, you solve a lot of problems if you do it this way. But if you have reactivity, it's just done. So what Maremo can do is it can— if I just show the code, it's kind of a cool trick. So you can give it a scatter chart from Altair, and Maremo can turn that into a selector widget. What does that mean? Well, what I'm able to do is I'm able to make like a selection here, like I'm able to select these data points, and then this widget has a value property, and I can just look at what I just selected basically. And maybe I should zoom out to really show that. I can select a different region here and then the DataFrame down below here updates. So what I'm actually able to do is like kind of look around this embedded space. Something over here, what's this? This is all about online grocery items. Stuff I see over here is about club cards. So there's like clusters over here, right? And the fact that I'm able to make a selection and get an impression of what the cluster is about, that's definitely kind of nice and useful. But what I can also do is say something like, oh, I'm interested in like, I don't know, Clubcard. And when this cell updates, this cell can update, which means that I can now make a selection. And lo and behold, this cell can also update. But there's something about that interactivity where I'm—

So is this stacking that filter kind of like you have that query?

So this is just to change the color of the dots. Oh, okay. That's the only thing that this does. I see. But the reason why I do that, there's a good reason for it. Is sometimes two clusters light up at the same time, and that's interesting. Again, I don't want to stare at my data, I want to interact with it, I want to actually look at it, I want to understand what clusters exist. And there's an interesting thing, like you have club cards, but if I were to go to like delivery, that's also a common theme. Oh, that's a, that's a lot bigger, for example. Just like being able to spot that and see that, oh, there's actually way more complaints about like deliveries than about like club cards. Just doing a basic vector lookup and then looking at some of those results wouldn't give you this extra information, right? And it's super easy for me to just whip this together inside of a Marimo notebook. So there's a ton of these, like, visual packages that I've made that I'm just not going to be maintaining going forward because they'll just become a widget inside of a Marimo notebook, and we don't need the complex web app around that anymore. And again, why does that work? Well, we have reactivity.

Nice.

So that's like that feeling, like, it's not necessarily upgrading the notebook, I'm kind of more upgrading my imagination. That's the thing that's making me stick. Like, I kind of know that I can get quite creative with the stuff that I want to do. And again, it's just nice and reactive. And it's just super nice. Any questions about this before I move on?

Can you scroll down a little bit? I just want to remind myself how you— okay, so you have the plot. So you have an Altair plot.

Going through the steps, step 1 is query goes in, then I've got an Altair chart over here, the colors update. Then I can make a selection over here. Whenever I make a selection, the widgets that I declare here that has a value property, that's a DataFrame and there's a text column, that's what you see over here. The step is change the color, make a selection. I can also pan around, by the way, I can make different selections. But every time I make a selection over here, Basically, this other cell will update as a reaction.

Can you show that part where you link the Altair chart to the, you know, so it's really this.

So this is the Altair chart. There is this little magic thing that we have around that. So there's a couple of libraries. Altair is one, we also do the same thing for Plotly. You're given an Altair chart, and Altair has this selector element, but it never really had a way to give that back to Python. So what Marimo is able to do is it is able to say, look, this is still an Altair chart, but you're also able to make selections. When you do, we have a widget over here. That widget has a property called value. So widget.value, and that is just this one DataFrame. That's basically just selecting the data. This is just a subset of the entire DataFrame. That's what's being selected, and that's something I have over here as a value. But this cell knows if I want to display this, this widget is a thing that I need. So then this cell goes, ah, I have declared widget. But then this cell says, but if I want to be able to declare widget, I need to have this variable plotter. And then I think I've got a cell somewhere, I don't know where, maybe down below over here. Yeah, plotter. And if I want to have plotter, what do I need? Well, you are going to want to have this transformed_x thing, and eventually It says, oh, I need this query over here.

So now that's interesting. Did you run that cell that was all the way down there, or does that matter? Like what?

It runs automatically, so it, uh, I don't have to worry about that. That's the whole, that's the whole point of Marimo. Marimo looks at the variables in the cells, and just by looking at that, it can kind of, um, it doesn't do it as at runtime, it does it before running, uh, all the code. But the whole point really is, oh, uh, I know what variables this cell needs, let's look for the parent let's keep on looking for the parents. Okay, we've mapped everything. Okay, now we can run everything once. Oh, a cell over here changes. Okay, what children do I gotta update? That is all handled for you.

Okay, 'cause like in a Jupyter Notebook, you would, yeah, you definitely would have to—

You would have to scroll up and down, run a few cells. I've been told there is actually a plugin nowadays that can do some of that stuff. And if you build your own widgets with the IPython widget library, there is actually like, there's a couple of party tricks you can definitely do. But this is just so much more intuitive. Like, this is a I don't have to think about it kind of a feeling.

And I see you've nerd sniped a lot of people already with this FastHTML. I'm not gonna spoil it, but I see that you are teasing it in the next column.

Um, oh, people have spotted that, have they? Um, we'll get to that. Okay. I'm talking to you. I'm assuming the FastHTML crowd is going to show up. I better come prepared. Prepared, right? Like, do the homework. But anyway, like, this on its own is kind of a cool demo. Like, already I like to think, like, okay, this is pretty useful. This is also good for qualitatively understanding what this dataset is about, basically, right? That's cool. But why stop here? Like, if I were to now think, like, what's actually useful to do, like, let's say, okay, we, we know that we are interested in getting some complaints about deliveries. If I make this selection, sure, some of them are going to be about deliveries, but some of them won't be. But these examples over here are like quite proximate to examples that are about deliveries, maybe, right? It kind of feels like we got to do some annotation here if you want to turn this into a classification problem or something like that, right? Then maybe a good next step will be just labeling. Hmm, that's where this little guy is going to come in immediately because there is a standard nowadays and it's a really cool project. It's called AnyWidget. AnyWidget solves the problem of building a widget that will work in Jupyter and in Colab and in Marimo and like all those different notebook environments. And AnyWidget is just kind of the standard that everyone else can wrap around. And this is a thing that allows you to make a widget basically. So Jupyter also has like a widget library. And the way that Jupyter handles it was always a little bit different than Colab, which is always a bit different than VS Code. The whole point of this library is that there's one standard that everyone can kind of just adhere to. It has a small surface area. But what it does allow you to do is build custom widgets for yourself inside of your notebook.

And who maintains— is this a Maremo project? Or is it a third party? Or so what's the background of it?

There's this guy called Trevor. If you— I used to host a podcast for my previous employer. Trevor was the first guest, Trevor Mance. He did a PhD kind of revolving around widgets like this, a bit more in the biomedical space, if I recall correctly. He is a direct colleague of mine, but it deserves to be stressed, AnyWidget is like all notebooks, and that's not going to change. So yes, Trevor's working with me and my colleagues, and we're definitely eager to invest in AnyWidget, but AnyWidget itself will never be Marimo-specific. This is across all the notebooks, basically. So if you build an AnyWidget for Marimo, it should also work in Jupyter. That's the way to think about it. However, the one thing I do want to mention is that the reactive nature of Maremo really lends itself well for any widgets, as you might be able to imagine, because what these widgets are able to do is you're actually able to change something on the JavaScript side of things and then see that reflected in your notebook. And maybe the best example of that, I'm just gonna maybe do something down below over here. This wasn't part of the original demo, but I might as well show it. I made this any widget library called draw_data. So from draw_data, I'm gonna import a Scatter widget. Let's just zoom in a smidge. Yeah, and then draw_data needs to be installed, so let's install that. It's installed, good. And now I should be able to do Scatter widget, wrap that with the Marimo AnyWidgetWrapper, and then this is a scatter thing. And then I should be able to just show you what the Scatter widget looks like. So this is a tool that allows you to literally draw a dataset. And this is amazing for teaching 'cause you can literally draw a dataset as you see fit. And the cool thing with that scatter widget is you can actually data as pandas, if I recall correctly. Yeah, so it can come out as a pandas DataFrame. And every time I draw, more rows are being added to DataFrame. So that's super neat. But what you're seeing here is I'm doing something in JavaScript land and somehow that finds its way back into Python.

So did you define this widget with Python or JavaScript?

Both.

Okay.

So if you look at the AnyWidget example on the main docs, I think it does a really good— it does a good job of explaining it. But let me just— can I zoom in? Yeah, this zoom in and change the color. Notice, by the way, that I'm doing the doodling thing again. Everyone buy the drawing tablet. It is amazing. So this is how you would define an AnyWidget. You have to import these two libraries. One of them is called AnyWidget, the other one is called Traitlets. Traitlets can be seen as a library that is able to bind variables such that JavaScript and Python can sort of communicate. And then there's this one value that I attach to this object. And then this is the basically ESM JavaScript string that has this one function called render. It needs to accept a model. The model represents all the variables that you declare from Python and that are shared between the JavaScript side of things and Python. And this element is basically the DOM element that it needs to attach onto. So in order to write these, you are going to benefit a lot from being able to do a bit of JavaScript. But what you can also do is have someone like Vincent, who actually doesn't mind writing JavaScript, and I will build these widgets for you such that you can just use Python inside of a notebook that you like, and you can still benefit from, like, having this interactivity as part of your workflow, basically. But the way it works, it's not— it's actually not that intimidating. So if I just go over, like, all the things that I— that are listed here, right? So, okay, what's the first thing that happens inside of the JavaScript thing? Well, we declare a function where we're able to get the value, which is this traitlet that we've defined inside of Python. After that, we're going to create a button element we're gonna say, hey, there's a count attached there. We're gonna add an event afterwards. Like whenever we do something in JavaScript, that's what this event here does. But we also have something clever here where we say like, oh, if ever the value changes from the Python side, this is how the JavaScript should respond. And that's about it. So it's basically what should happen if JavaScript makes a change to Python, what should happen if Python makes a change for JavaScript. And then you've got a button that has state And that's the way it works.

Cool.

AnyWidget is the bee's knees. Again, just like the drawing tablet. I don't understand why I'm the only guy making these widgets. There's a small crowd of people actually building these widgets. It's a cool Discord, by the way, like do definitely join.

Does it only work? Is it only appropriate for, you know, notebooks? Or can it— can you put it in your web app?

And just— yeah, It's not, so, I mean, Trevor would know the fine details there, but it's a different interop basically, because Jupyter has like a certain way that it expects variables to just kind of be there. And it's, I've not tried getting it to work inside of a proper web app. I'd like, I'm 99% certain it's going to be a painful journey and it's going to be way easier to just use something like FastHTML instead. If you wanna have this kind of interactivity, unlike a normal website. But the goal here really is to give you features for your notebook experience also. So it's kind of, it's not the target, I think, for the project. Again, if Trevor is watching, he can confirm, but my impression is that this is really just meant as a tool for you to build tools inside of your notebook. Fun fact though, there are actually a couple of very popular libraries that actually use this under the hood now. And if you go to the community tab, um, Altair actually uses this under the hood, which is like a thing that not everyone knows about. There are also these other libraries, um, more to do with like Mosaic. Um, there's also some like physics things in here, by the way. Um, but there are like— PyOpsPlot is also like a really cool plotting library, by the way. But there's like a small community, and there's definitely like very cool widgets, um, for sure. But it's kind of one of these libraries that are meant for package makers. So it's not that wildly known amongst the user group, if that makes sense.

Yeah.

But it is a super cool project. And if you're interested in learning JavaScript, this is probably the most fun you'll get, like building your own widgets. If people are interested in talking to me afterwards, I am actually doing a bit of prompt engineering right now to see if we can get these widgets automatically generated on the fly. Like the dream that I've got is that in a couple of months, you'll be able to go into Marimo And you'll be able to sort of start typing here, something like, hey, build me this widget because I want to do this kind of research. And the widget will just be generated for you inside of the notebook.

So, what's the DevEx of that widget building experience since you are writing JavaScript inside a string?

Oh, you can also have it in a file. And then you can just say, like, hey, there's a static folder here. Python pathlib library, open that file, put the string in here, that you can also do.

Is it a nice way to debug and let's say, kind of play with it, like really quickly when you're writing the JavaScript?

You know, I mean, the main— to me, it all became easier when I learned of this one shortcut, Shift Command C, because what you're able to do is you're able to click an element, click, and then DevTools will immediately bring you to that one element that you're interested in debugging. Like, it's the annoying thing was figuring out like, how do I select that one element inside of a Jupyter Notebook? And I don't want to click around here forever. That used to be the painful bit. But then once I got that figured out, it was pretty good. I will say you can also get quite far with vibe coding those kinds of things. It's just that if you want to debug it, it does help if you've done like D3 before, let's say.

Oh my goodness, D3. Okay.

Yeah, I like, I like that one.

It's a steep learning curve. Yes.

It is, but at the same time, it was also the reason that I started to learn programming. Without that library, I probably would never picked it up.

Yeah, it is super cool, no doubt.

Yeah. Yeah. So the, but like, one observation I do want to maybe make here is like, hey, this is kind of interesting. Like, when we're talking about interactivity, Marimo comes with widgets, but nothing is stopping you from making your own. And the way that I've set up this notebook is what I can do is I can actually maybe make a selection over here. This is something that I might want to annotate. And then here we are to the widget that I sort of want to really highlight here. This is an annotation widget. There are lots of ways to annotate data. Sometimes you want to make selections in an image. Sometimes you want to make selections for spans and text. Like, there's lots of different ways to do it. But 9 times out of 10, you can boil it down to yes, no, and skip. Like it's kind of a binary situation that you can often boil it down to because sometimes there's a classification. Can I apply this label? Yes, no. Sometimes it's, oh, there's these two variations, which ones do I like best? Sometimes it's, oh, I'm doing retrieval. Is this example a good thing that fits this query? So the way that this thing is set up is basically you have to generate a card here in the middle, and then you can say yes, no, or skip, and you can also go back if you want to. The way that you generate this is up to you. The only thing you gotta do is give me a string that really looks like it's HTML that I can render in here. So if you wanna use FastHTML for that, totally be my guest. But the thing you gotta do is you gotta give it a list of examples and a function that knows how to render each example in here. Then I can render that in the middle, and the widget around it will from there provide you keyboard shortcuts, which will also be displayed, and I've also built it in such a way that the gamepad API is also supported. And the thing that I'm then able to do is I'm able to sort of click around and say like yes, no, yes, no. I'm able to go back as well. But the really cool thing is sometimes you also want to add context, and if you're using Chrome, Chrome has a web API for speech detection and it's free. So what I'm also able to do is hit this button and then I can start talking. And then, oh, this is now a note that will be attached to this one annotation. And I'm done talking now, and I can go to maybe next. And if I were to go previous, I can see the note again, and I can sort of— there's lots of these examples I can imagine, like especially in LLM land where, oh, the bot made a mistake here, but for downstream tasks, I gotta explain why this is a specific kind of mistake, right? Um, so for that use case, I also make it easy for you to just add a note If you look at the annotations at the bottom here—

So can I ask you a question about this annotation interface? What part of this annotation widget, this card, I guess, with the buttons, like, is that all defined in FastHTML or what?

Up to you.

What in this demo, what was—

The render function. This render function. So this render function is FastHTML. I wrote kind of my own thing called MoeHTML. It does a few things that I like, but any, like you can also use Jinja, I don't really care. I will say FastHTML does lend itself very nicely for this though. But you have an example that goes in and what I wanna render is some sort of span on top that tells the user what the label is, what the task is basically. And then at the bottom, I have this example text that I just wanna show and there's like a couple of Tailwind classes that I attach. But the way that you render this, it has to be a function and it's up to you to choose what you like. But FastHTML does lend itself very nicely for this, especially because I think Isaac is in here, he made Monster UI and there's like a couple of these packages that come with nice UI elements. Those can just be dumped in here and they just work.

Can you scroll up again? I'm trying to just locate where this label appropriate thing is and the other text.

Sure.

In the UI.

Yeah, so let's first start here. So we've got those examples, right? Yeah, and let me, let me maybe start there. And again, let's just look at the first.

Oh, I see it now. It's part of that. Okay, I see it's underlined and it has the— but yeah, so okay, that, that text part is basically your— that's the renderer, and then all the buttons and stuff are kind of your widget that you already made that we plug in.

Yeah, so, so the thing I, I— well, the thing that I've built around it is, uh, whoops. The thing I built around it is this progress bar, these buttons, the notes thing at the bottom, and it's up to you to populate this with something meaningful.

I see.

And, and how do we do that? Well, you supply a function that can do it. And as far as I'm concerned, FastHTML lends itself quite well for that.

No, I really like this. This is like a really good trade-off, or this is like a kind of a good design pattern. This is, I think you would have to be really annotating a lot of data to know But I think this is—

I built annotation software before, so it's not my first foray. But the main thing that I like, all the things that you just said are true, but there's like this one extra thing I want to add here. You can still do Python around it. So if you want to do deduplication upfront, send it to a specific database, do user stuff, you're in a notebook. And you're free to do whatever. That's the thing that I really like about this approach mostly. Like, sure, the UI is kind of like nice and all that. And the gamepad is awesome. But the main thing that is really nice is it doesn't get in the way. You just add a widget to your notebook and, oh, it's pretty fun to just get started annotating and you can be done in 10 minutes and in an hour you have 500 examples. I don't know how fast people like to annotate, but it doesn't break your flow. That's the main thing I like about this thing. You don't have to start up a new app. You can just sort of add a widget and your notebook can now also annotate data.

Yeah.

And again, what I really want to get at here is I'm trying to get people to stop staring at their data and to start really looking at it. and the best way for me to do that is to make it fun and easy. And I like to think that this is indeed a very nice balance. The thing I made earlier, by the way, like this is I think a widget that combines a few elements. The speech-to-text thing, there's also a library called MoTalk that just has a widget that only does speech-to-text. There's this other widget called MoPad that is doing the gamepad stuff. Like there's also widgets just for that. But don't sleep on these widgets in general. Like other things that you could do, um, I don't know if people played around with the Gemini API, but the Gemini API actually allows for drawings and like images and stuff. So I made this paint widget, but this paint widget is actually like pretty dang neat. Like if you want a demo that I've done that you can do yourself as well, especially if you have a drawing tablet like this, you can actually draw a DAG like this and then tell Gemini like, hey, what's the JSON for this? And it will just go ahead and give it to you. I'm building another tool like this built on TLDraw, which has like proper vector images, and you can draw proper diagrams in that and then hand it off to Gemini and they can actually do mechanical engineering tasks for you. And again, like, why stop there? Uh, we can paint, but what we can also do, if you're into like PyTorch and stuff, I can also come up with widgets for like charts that live update. So I can have a cell over here that has a chart, and I can have another cell that updates that chart inline because I can change the state of the widget. And that's something you can now totally also have in your notebook. You can have these cells that sort of asynchronously update as while some training is happening. And on top of that, and that's kind of, I think, a nice place to maybe end as well, uh Hey, one question.

Those annotations, how do you save them? Do you like, do you do that? Yeah, like where do you—

yeah, so up to you. So like one thing, uh, as I annotate, right, this cell over here updates and there's a list of items. You can write some logic here that says every time, I don't know, I get like, um, I have like 10 annotations or maybe 20 and then 30, etc., then I'm going to send the request to some backend to store it in the server. You can also write a little bit of SQLAlchemy here if you really wanted to. That's kind of up to you. You can sort into a file as well. It depends on the— that sort of stuff usually depends on what you like to do as a team, but you can write Python around that to take care of it.

Okay. It's this LabelWidget.get_annotations. When— what is the event that triggers that to run?

So this thing triggers every, every time you make an annotation. So, you could store every single thing that goes in. You can also choose to batch this.

Okay. And you can— Yeah.

So, you could do something like if length of this thing mod 10, right? Then the cell will still run every single time, but it's not gonna cause an event to happen downstream. If you were to, like, add a mod here. Like, you could do stuff like this. But again, like, this is like, I remember working at this company and then the community would ask us things like, oh, could you add support for MongoDB? And like, oh, could you add support for Postgres? And oh, could you add support for UnicornDB? And like, you kind of get like all these crazy requests of things you would like to add. So the main thing that I would recommend people do here is just write Python. You can figure it out.

So is there something like get annotation, but singular, like git, and like, you know, just gives you the most recent one, you mean? Yeah, you just do it like this. Okay.

I mean, I made this thing like 2 days ago. So it's pretty fresh. You can definitely like, use it for a bit and then tell Vincent to maybe consider a feature, right? That's also something that can happen. Like the main thing I want to do here is like, this is also me presenting this for the first time to the outside world. I'm also here eager to get feedback, right? So like if there's a preference to maybe only have the latest thing or like something like that, we can always add that to the library. That's definitely reasonable.

Got it. Okay. I mean, this is like, it's not like, okay, this is not you saying this is our annotations or like our widgets library that we've, you know, thought about for an extremely long time. This is like, hey, you can build whatever you want in Marimo and kind of look what I built. You built this thing for annotations and you can build something too, if you want.

Yeah, I mean, it's kind of like Legos. That's the best explanation I would maybe put here. Like, um, Marimo defines how things can click together. And what I really like to do is come up with fancy new Lego bricks that you can maybe use, right? And a lot of it is because, oh, there's a new Gemini API. Oh, it does stuff with images. Oh, I need an image input. Oh, it doesn't exist yet.

Okay, let's build one.

Um, so a lot of the stuff that I do I'm technically the first hire over at Marimo. So I am technically like a developer, although a lot of stuff that I do is a bit more based around growth. And the way that I like to maybe do that is I also do the YouTube thing, of course. But a lot of it is also just come up with widgets to see if this can actually serve a community. So right now, I'm very much into the, you know, data quality. That's definitely like my background. I want to make sure we got good widgets for that. Depending on what we got to focus on this summer. I don't know, maybe we do some stuff for maps, right? Like, I can definitely see that kind of becoming a nice little thing to maybe work on. Um, so we'll see, like, what has priority. We're, we're a small team, but we ship, like, 3 releases a week, um, 4 if you include these widgets. Uh, but the thing that I'm trying to do is have, like, more widgets out there in the ecosystem. That's definitely kind of, like, the thing I really try to do over at Marimo, which is also, like, every month I tend to release 2 of these widgets.

Okay. There's been a lot of comments on the Zoom that you have a YouTube channel where you keep sharing different ways of using Maremo and it's highly recommended.

So it gets compliments online and we are not— we try to get— we try to— if we make a video, we try to not be too hypey about it and make it worth your time. We only try to show stuff that's actually kind of useful. That's definitely a goal that we have in mind. But yes, definitely check us out. We also have a really good Discord, by the way. Feel free to check that out as well. I do want to end on a final thing, because that's something that people sometimes forget. A thing that you can also do, just as a programmer, right? Let's make a class like, I don't know, FooBar. I don't know. And we're going to initialize it just like you would normally, right? I know you would do like self.thing equals thing. Let's maybe add a thing in here. Fine. Then if I were to do something like a foobar, this is what the representation looks like. This comes from the era where we are doing things inside of a terminal. That's fine. But for Pete's sakes, we're in a browser. We can do super awesome things in here. One thing you could do is maybe consider doing some interactive HTML there. A trick that you can do that's Marema-specific is you can add this display method, right? And then I made this one library called mo_html. Like, one thing that's kind of cool about it is you're able to do, like, you can add some text in here, hello. I could do, like, a class, uh, font-bold, if I'm not mistaken, right? If I were now to run this, oh, like, I can display a rich HTML thing in here that kind of belongs nicely to this class. And The best example is—

So there's also the dunder—

Yeah, that's the—

Like HTML thing and, you know.

Yeah, so there's this repper_html is also something that exists.

Yeah.

This will work across Jupyter and Marimo. The only thing, like the reason I bring this display up is because this is Marimo specific, but Jupyter does not look for this one, but we do. So if you wanna make something as Marimo specific, this is the way to do it. This is also definitely something that you could do though. So if I were to—

Why would you wanna make something Marimo specific? Like give us an example.

So, one thing that we've got, so, mo_app_meta and then mode. Right. So, this is something— so, right now, I have code that I can write that detects that right now this app is running in edit mode, right? But I can also run in app mode. Like, I can drag— I need to zoom out here, but I can drag the outputs of cells into a dashboard, if that makes sense. And this— in doing this, this is like the widget editor that Marimo gives you. So you can make these boxes smaller or bigger, but now you're running in app mode. Oh, maybe the way that you display something needs to be different if you're running it from edit mode or app mode, let's say. Okay, that distinction is like a very Marimo-specific thing, right? And there will always be things that are just really Marimo-specific. So for that, we have our own little verb, but it should be stressed. Like 9 times out of 10, I would already be super happy if more people would just do this.

Yeah, yeah, makes sense.

Learn a browser, like do it, do something cool with it. Because if you do that, then you can, I don't know, maybe make a dice library where, oh, I have a dice with 8 sides and it just looks like a chart with all the probability values that are in there. And what's kind of cute then is if you were to maybe add— you can add two dice together, hook up a slider to it, and this is just— your Python code just becomes so much more visual, right? And it kind of feels like more people should maybe get that as well. Like, this is also like something I would love to see more libraries do more of. Like, think more in terms of like, hey, we're building tools for a notebook. Like, we're not like—

Can you tell us a little bit more about app mode? Like, oh sure. Is that— can you— how far can you go with that? Like, in the sense, can you deploy basically web applications this way? Do you see people putting auth in front of it and stuff? You know, how does it—

what does it— well, so like, uh, there is a demo on our GitHub where there's a FastAPI app in front of this, and then FastAPI handles the auth for you. Um, The— because Marimo is built on top of Starlit, so like if you really want to do a proper integration, it helps if you're building something that's like Starlit compatible. Um, I will say there's like a few ways to deploy things inside of Marimo. So one way you could do it is you could say, oh, let's deploy this to Wasm, right? And then you can put auth in front of that. Another thing that you could also do is say, no, let's deploy on like a proper Python backend because we need like GPU support or what have you, right? If you want to take a notebook and run it in edit mode, you would call from the command line marimo edit notebook file. If you wanted to do something like, oh, actually run it as a web app, you would do marimo run instead. So that's the main, like, from the developer experience, that's the main distinction.

Okay. Yeah, it looks really cool. I need to give it another go. This is like a really targeted way to snipe me into Marimo that you have come up with right now.

Trust me when I say it wasn't just you that got nerd sniped by the gamepad, by the way. But like, I'm also just still kind of scratching the surface of what you can do with Marimo. It's like, we honestly, we are really good at making releases to the extent that like a lot of our team members don't always know about all the features just because we ship so much stuff. Like, that's actually— we had a small little meeting about that the other day. Definitely expect us to keep on adding features. You can also definitely expect us to build something cool over the summer and that we're probably hopefully going to announce in September. That's also something to look forward to. But the main thing that I really hope to get across here, these are all just tools, but please, we're not upgrading a notebook here, we're trying to upgrade your imagination as well. If you take anything away from this, it's really not so much like use Marimo. Use Marimo, it's really cool, like use it. But I want people to dream a little bit more, like that's, that's more kind of the point, especially because we're like, pretty soon we might be able to generate these widgets by talking to an LLM, right? And that might become a really big part of the future of how you work in a notebook. But in order to get there, I do need people to sort of let the idea of what the old way of working in a notebook was, I needed to have people to let that go a little bit. Because what Brett, it's precisely what Brett Victor said in that great talk, we don't really know what the future is gonna be yet, so we're gonna try a whole bunch of stuff. And we can be really creative, but only if we allow ourselves to be. Right, so that's the main sort of.

Yeah, no, that's great. End on Brett Victor, can't go wrong. Can you tell me a little bit before we go, how does Maremo make money? Like, can we, you know, is there a paid aspect, some kind of hosted version, something? How does it go?

So there's nothing I can, let me put it this way, I can't announce anything on that front just yet, but one thing I can say is historically there was a, It was 2 people before, and they got a grant, which allowed them to work for like, I think a good year and a half on Marimo. And then at some point, the choice was made to go with the VC route. So this is seed funding that we're at. It was a round led by some pretty big chiefs. Anthony from Kaggle was in there, I think was McKinney in there, like, big group of old school CTOs that wanted to do a seed round. So we're in that. The focus right now within the group is definitely like, let's just make open source really, really good. And let's see if we can get like a good community around it. That's the, like, I would argue primary goal. You can definitely imagine there's going to be at some point a version of Marimo that does some enterprisey stuff as well. And that's something that will definitely be explored. But what you're seeing right here and right now, this is, I believe, MIT licensed, it's on GitHub, this is not going to go away anytime soon. Like we're not going to and do a rug pull on something that's open source right now. That's not going to happen.

All right, sounds great. This is really great. I will, you know, Vincent, if you don't mind sharing the notebook or GitHub, wherever it's located.

If you go to the— OK, go ahead. Yeah, so if you go to coning/mo-label, yeah, so you can actually play with this thing on GitHub Pages because it is WASM compatible. But the notebook that I'm using right now, bulk.py, is on the GitHub repo itself. The datasets that you need are also self-contained in there.

Okay, MoeLabel.

All the plugins that I make on behalf, that are kind of like Marimo related, they all start with Moe. So there's MoeHTML, there's MoePad, MoeTalk, MoePaint, MoeLabel.

Nice. All right. Well, thank you so much for coming. I will post notes with all these links, including the drawing pad and all. Everyone wants to know what— everyone just wants the link. They just want to buy it. So I will figure it out and I will—

Mopaint. Mopaint. You can go to the— you can also go here and immediately play with Mopaint. It's going to load. It's going to work from GitHub Actions.

What I mean is the drawing, your hardware. Oh, that thing.

Yeah. Okay. There's a column code. Okay. So if If you go to calmcode.io, you scroll all the way at the bottom, and then there's this meta category, and there's content that is basically a small course that I made 5 years ago. All of it is pretty relevant still, just a different drawing tablet.

Oh, that's your website, Calm Code?

It's one of them. I have a few at this point, but like Calm Code is a learning thing I made over COVID. It is probably the cheapest DataCamp alternative you will ever see. It has like 106 courses about all sorts of Python and data topics. If people want to know more about like stuff that I do like day to day, I also have a blog, as well as a Substack if people are interested. And I guess like the final, like there's two things, yeah, a couple of personal plugs actually. I have a personal YouTube channel where I tend to review these ergonomic keyboards. The one thing I want to say to that is the ergonomic keyboard doesn't fix everything, but I've met a couple of people that really ruined their wrists in their career, and I'm super happy that I'm not one of them. I was early enough. Do take that stuff serious, and if you want to have honest reviews, definitely go here. But if you ever feel like wrist pain, take that serious. Do exercises, talk to a physiotherapist, don't ignore it. It could risk a career. Like, that is a thing I do want to maybe mention explicitly.

Yeah, I know people like that, for sure.

Yeah, and so you just want to prevent it. There's a couple of really good boards that help you do that. This is one of them, but full reviews can be found over here. And then the final thing, in this age of like LLMs and stuff, I'm also interested in writing software that's just a little bit more human. And this is more like personal project territory. And this is more like, I'm sure some people find this interesting. One thing I made a while ago, is this service called Dear Me Email. The way it works is you can send yourself an email and get it back in 30 days, and that's it. It's kind of like a way to reflect, if that makes sense. Um, I made this a while ago. I gave it a little boost now because I want to sort of get back into this human, uh, software a bit more. I know of at least one guy that was dealing with alcoholism that ended up sending himself emails, and it actually ended up helping a whole lot because it's really hard to be critical against yourself. Like if you yourself are reminding yourself of like goals that you've set and you're failing, there's something about that is really impactful. So if that sounds interesting, definitely come talk to me about that as well. And another thing I'm currently working on, I've noticed that it's very tempting to keep on asking ChatGPT for things, to the extent that like, I actually want to know things myself as well, like from the top of my mind. So I'm also exploring like a variation of Anki, That's just, you know, live and it's free and it's out there. Next week, I'm also gonna add gamepad support for this. But I'm definitely interested also like from the point of view of a developer. It's really great that we have these LLMs at our disposal, but I would hate it if you become so dependent on it that at some point it becomes kind of a learned helplessness kind of a thing. So if people are interested in maybe exploring that with me as well, definitely reach out to me on socials too. These are like side projects.

This is— Does this like project potentially pull from your existing conversation history to find repeated things or something? Or where are you?

I might go there. I might go there. For now though, I just, the thing I wanna focus on now is just that, oh, if you're on the bus, the mobile experience is just nice. And, uh, oh, if you wanna use a gamepad, that is just kind of nice. Step 2 would be, um, how do you generate these flashcards to start with? But part of me is also wondering like maybe the act of building the flashcards yourself and being very conscious about that is also good for like deliberate practice. If that makes sense. So I'm definitely eager to explore the LLM generates flashcard thing later. But for now, I just want to iron out lots of other things. It's more like, hey, do other people also have this feeling of I'm becoming too dependent on these LLMs? Would it be good to maybe invest in a few apps like this? If people are interested in that, then definitely also reach out to me because that's like, definitely territory that I'm personally exploring right now.

Okay, yeah, we should talk. You should also talk to Isaac. Isaac and I work with a company called—

Yeah, I read about that.

Oh, okay.

Yeah, I read about that. No, like, we will talk about that later. There's like one thing I'm wondering is if I want to make like a remember, like a flashcard thing that's specific for developers, or if I want to keep it general. Because I'm having the impression that actually, It makes sense to maybe do something bespoke for developers, because especially if you put like an LM in the mix, you can actually make something that's a bit more bespoke. But we'll talk about that later.

Okay, great. There's so much. Do you have a site that has all your things on it? Like, like in one, like, hey, I have doing this project, this other project, whatever.

Once a month, I try to send an email of like, hey, here's stuff that I've been working on that seems pretty neat. And also, people should blog more about prompts that they like, by the way. If like, today I learned—

Oh, I wish people did. Yeah.

Oh, like, I don't want to be on LinkedIn to get signals like this. I want to be like on the normal internet. So I try to share prompts as well. Like today I learned things from Simon Willison. I just try to do as much as possible. Okay, yeah. And I try to learn in public. That's kind of the person that I am. So if people are interested in that, I have a Substack. You can also— I'm also very active on like Discords these days. So the Marima one, the Anywidget one. I'm also like in the FastHTML one from time to time. But Discord is also where people can find me. And I'm on Twitter and places.

But yeah. All right. Okay. I promise this is goodbye now. We can actually say goodbye.

That was a good session. It was fun. Let's hang out again.

Yeah, yeah. We should totally do that. That was really cool. I'm gonna try to hook up my note my gamepad after this, and I'm going to check this out. But you have successfully sniped me into Maremo.

So sweet, sweet. Mission accomplished. I can now have a cake. Uh, cool. Then everyone, thanks for listening. There's a pretty full crowd as well, so that was also very nice to see.

Yeah, yeah, a lot of people will be getting the recording, so I'm gonna be sending it to all the students.