AI in a Day: GPT-3, Stable Diffusion, Embeddings, Fine-Tuning

Hello everybody. Welcome, welcome to this AI stream where I will go through kind of what I've learned, what I've been up to in AI. Building little experiments here and there, trying to wrap my head around what's going on. I find the best way to do that is just build, build small projects. So yeah, if people have questions, feel free to drop them in the chat. I will, I sort of plan, I don't know, spend third 15 minutes or something going through my journey the last month or two. And then I'm curious what questions people have and I'll go through all the code so you can see how some of this stuff works behind the scenes. Awesome. Let me share my screen and get started. I've been downloading this new tool. Actually, let's see if it worked. What would you like to do? Create a new program. Oh my gosh, it worked. Okay, so there's this new thing, so I'm writing a, I wanna write a script to download a bunch of tweets to, to kind of do Ask My Book for a Twitter user, for example. So anyway, there's this new app called GPT-3 Lab, GPT Labs, UPG. I don't know what that stands for. Anyway, I saw it, so I just, I started it literally 2 minutes ago before the stream started and I just typed in my Python. I want a, a, a language, or sorry, a program in Python and I want this program to download all of Bala GS's, I think, I just guess at his handle, as a CSV with one tweet per row, right? So basically there's all these tweets on the internet and now there's like a Twitter API. So I could, I would basically have to create a developer app on Twitter and then write a Python script to loop through some endpoint ad infinitum until it gets all the tweets and stores that, writes it to a CSV file. And so anyway, that would take, I don't know, like an hour probably or two for me to do. Not hard, but I would just, most of that time is just spent Googling like How do I do this with a, a requests API in Python or something? Right. And so anyway, let's see what it did. Let's see what it spit out. Import. So this is a presumably a library to, to hit Twitter's API. And this is the CSV library that I'm familiar with cuz I use it, used it to build Ask My Book. It creates that, that's pretty simple. It creates a list object. That's gonna hold all the tweets, which will be these kind of objects from Tweepy, these tweet objects. So it's gonna first do, it's gonna do an initial request for the maximum count, 200. This is pretty impressive to be honest. I love these comments. I'm really just reading the comments. I'm reading the code and saying it, but I realize like it's literally just telling you that, so I don't even need to talk necessarily. Good for if you're deaf. Oh, Julie, you have a question. Awesome. How do I raise your— allow you to speak? Let's do this. Allow to talk. You're allowed to talk. Am I doing something wrong? What's up, Julie? Can you hear me? Oh, she left. Oh no, she's here. She accidentally raised her hand perhaps. I'll keep going. If people have questions, actually, let me open up the— there is a Q&A widget. So yeah, use the Q&A widget or use the chat. Either is fine. Yeah. Anyway, I'll keep going. Okay. Get the tweets. That makes sense to me. And then you, okay, so this is a new function, but I assume this basically takes this list that is retrieved and puts it in this list. And base, you get the last ID, right? And no, this is actually a new thing called UPG, Patrick, which is, yeah, similar, probably based off the same exact APIs, right? Codex from OpenAI. And then basically while there are no— basically keep doing this until basically there's gonna be an API request that doesn't return any tweets cuz you've reached the end. Keep doing this. And then once you have— oh yeah. So then once you have all of— once you've run through this function, that basically means that this thing is now full of tweets, tweet objects. And then basically this goes through that and instead of these tweet objects, which are just made up by Tweepy. It turns them into these dictionaries in Python or tuples or whatever they are. And that contains this, the ID of the tweet, the, the, the time it was tweeted. And already a 2D array, I guess. And then the tweet itself, what's the text of the tweet? And then it would write this to a CSV. Yeah. That's exactly, that's exactly right. So I guess there's, that's pretty crazy. Yeah. That probably would've saved me. That's an hour of my time. I don't know, let's say I'm an engineer, get paid $100 an hour, $100 in value right there. Pretty cool. Anyway, that was just a tangent cuz I had it open. I wanted to talk about my sort of journey. Patrick says, wouldn't it be great to pipe the comments to a speech synthesis AI? Yeah, in theory. And one of the cool things is AI has become multimodal. This is what makes it general instead of just specific, which is pretty cool because it means, for example, I vastly prefer text, reading docs and trying stuff and copy pasting and, and all that stuff. But some people prefer audio, right? Yeah. Like without actually having a human having to do the translation, you can now pick your format, which I think is pretty cool. AskMyBook is an example of that. Anyway, I wanted to go through my sort of AI journey. I know some people have seen some of this kind of stuff, and so I think it would provide some good context. So I think this is the first moment that I think a lot of people's minds got blown, which was GPT-3 came out and they released this thing where you could effectively type into a text box and get a result. And the way that this works simply is that it basically makes a guess at, based on the prior text, like what would be the next word or token based on a snapshot of the internet, right? So for example, if I say 1, 2, the AI is gonna be pretty good at guessing that the next thing is 3, right? Or A, B. So you can, that's really simple. Obviously that complexity ramps up pretty quickly in language. So yeah, that, but that's simply what it does. And I think what was so powerful about this to me was that it allows— made AI accessible. It's kind of no-code AI, right? Whereas before you would have to write a bunch of Python or download some model or I don't even know. But this is like really makes sense to me because it's like just, oh, I just have to send sort of an API request with this string, right? All of this text. And then it'll just come, it'll return more text, right? Which is the answer. It's this very simple call response question answer thing, which is how it kind of APIs. Work, right? So for example, if I wanted to write, let's say I wanted an app that made me more excited, right? So there's a text box that's, you type into the text box and then you, there's another text box which comes out with the result. You just have to say, the following is a conversation with, or the following is a set of, you might not even need this actually. Let's, it would be something like this where you'd say input output, or the following is a list of inputs and outputs where the first input— where the input is happy and the output is super happy, right? And then you might even want to be more specific. You say the list of 5 pairs of inputs and outputs, right? And then you— or let's say 2 pairs just to simplify this. And then the input would be the user's input, right? Whatever the user put would be like, let's say user input or something, and then the output and you would just leave this blank, right? So this is like what, almost what you would send to the API and then the API would just return whatever the answer is, which would like, let's say the user, or let's say this is what I would send to the string that I would actually send from my local script would be something like, my name is Sahil Lavingia and I am the founder of Gumroad. And then the thing would basically spit back the rest of whatever it would guess, right? Which might, and it might make some mistakes because it might think, oh, like super is supposed to be added. I might say I'm the super founder of Gumroad, like something like that. But this is like really cool to me because it's freeform. It's like very dynamic. I think a lot of people think about programming like this very like hard and fast mathy sort of thing. There's no math here, at least not that you see. It's this much fuzzier thing. And there's a great blog post about this, I believe, called Software 3.0, which I highly recommend, I think it's this blog post. No, it's Software 2.0, which everyone should read. I'll put this in the chat. But he talks about this sort of, we're used to writing code in this like very humanistic way, right? Where you say print 1 1 and it prints. All of a sudden you're saying like input, this is a conversation between a math teacher and a math student. The math teacher says, what is 1 1? The math of the student replies to, obviously you wouldn't use API, uh, AI for that is a very, math is not fuzzy, right? So generally it's not a good fit, but for plenty of other domains like music or art that are fuzzier, all of a sudden you can have a similar interaction and get some responses, which is like pretty, pretty cool. And so that's, he talks about the math behind like, what does that actually look like and mean and what's actually happening behind the scenes. But pretty simply, that's what's the transition that I think is pretty cool. Ooh, someone else raised their hand. So let's see if I can. See that list and allow you to talk. Where are you? Oh, you unraised your hand. Oh, anyway, so I saw this and I was like, this is cool. I experimented with it like a bunch of people did and nothing really, I don't know, like I think it's cool. I think I started using Copilot, but I didn't have anything that I really wanted to build. I didn't feel like I was like, oh, I can go build something. And then fast forward sort of 2 years, I think DALL-E came out last year and, and then I saw this from Peter Levels, which I thought was like really cool. And like a really functional use case of using DALL-E, which is like similar in a sense to GPT-3 images instead of text, but this sort of similar fuzziness. And then really what changed the game was Stable Diffusion, right? So DALL-E was released by— for folks that don't know, DALL-E was, is kind of GPT-3 for images released by OpenAI, which is really awesome. Everyone's seen this, I'm sure. And the kind of the joke is that it's called OpenAI, but It's not actually open source, right? It's open in cer— certain ways, but not in others. And then really what changed the game was, I believe late September there was the Stable Diffusion release and the Stable Diffusion release. Let's see when that came out. I guess August 22nd. So sorry, I'm, I was off by a month. And so it's probably not, I would say like from, in my eyes, not as good quote unquote as DALL-E in terms of like the images that it produces being photorealistic, et cetera. But the. The real core difference is that it's open source, right? Or parts of it, I think, are at least are open source. We do, it's a binary kind of model. But what open source means is you can look inside of it. And what that means is you can spin up your own version of DALL·E by just almost like unlearning. Imagine a brain. This is imagine if the brain was like a crepe cake or something like that, right? Where you have like layers of understanding that build up over time. Like there, let, let's say for example, like on a very base level, you have the laws of physics and then you have like the periodic table of elements and certain Maybe at some level of abstraction you have ping pong, right? Which is based on all these things, but you don't have to actually think about those sorts of things, right? If you wanna make up a new game, like you can take ping pong and say it's like ping pong except really big and with tennis balls and we're gonna call it, we don't use a table, so we're just take the table out of and it's just, we call it tennis now, right? And it's the origin of tennis for folks who don't know. And you don't have to learn the laws of physics to do that. You can just change the rules at a like very high level of abstraction and come up with a very new game. And that's what open source allows in this way, which is you can peel back the onion the 1, 2, 3, 4 layers of the crepe cake and then change that. And, and then all of a sudden you have a new model that can do a new thing. And that's really what enables all of this kind of stuff, right? Like you've seen here, this is a model, for example, like imagine DALL·E, but trained on, I assume in this case, like a lot of like catalog imagery, right? That you would see from like IKEA or other brands, right? And it's, it anyway, so that really got my attention because it was like, all of a sudden I can, if I can think of some vertical use case, I could start experimenting with this. And then I saw this, which I'm sure a lot of people have seen, and I think this grabbed a lot of people's attention, which is like this Pokémon generator, which I believe we can actually try. And this is a Yoda Pokémon. What happens if you put my name in or something weird stuff might happen. Oh, did I disable chat? Oh, you're gonna have to ask questions then, huh? I'm sorry. Oh wait, I just enabled it so I, people can chat now. Sorry about that. Sorry everybody. So this is me as a Pokémon, like water Sahil Lavingia. At least show you that it's like having some impact, right? The prompt. But imagine looking similar to that kind of fuzziness, right? I can just type a prompt and get it, get an answer, which I think is pretty cool. So let's see what this comes up with. A water sawhill. Awesome. People are using the chat. Great. Yeah, this is pretty cool. It looks like a face almost, right? The eyes and the nose and the mouth. Weird, right? It's learned this somehow, right? Like it has a— it picked up this pattern where you have eyes on the top, you have this kind of nose object, and then like some sort of mouth. And there's some symmetry. Obviously it loves— it has definitely picked up on that kind of symmetry. But anyway, That was really cool. I'm like, that I think to a lot of people is mind-boggling, right? Like now it's almost like we don't even think about it because we've seen it with a bunch of these kinds of examples. But like that, like this image, like you can't find something too similar to this on the internet. It's crazy, right? It would be like Googling like water, water fish and like finding, finding nothing. It's just like a, a thing that doesn't happen, but we get used to it so quickly. Anyway, that really got my attention. And so that's then the cool thing that he did is he actually open sourced all this, right? So he. Released this, you can go find this from that tweet. And so that's my, what I do always is I just copy, like my goal is just to exactly do this person did, and then I can modify my approach and come up with something else. And in theory, my view of it is like, all I need to do is find a different set of images that are labeled, right? In his case, he had to find these Pokémon images. Let me see if I can find the actual. He actually, he has a good, where's he? He has some thread somewhere, but basically you have to find these images. Why does he not, where does he talk about this? Oh yeah, here it is. I think this one. Yes. So he talks about what he had to do, right? Which is he had to get these images and then label them all. And he has a CSV file, let's say, of images. And then the labels for these images, right? And there's, I think there's 200 Pokémon or something like that, or 1,000 Pokémon that he did it. So I was like, okay, cool. All I have to do is find like 1,000 tagged images of some other thing, and then I can, in theory, have a similar UI here that would allow me to generate whatever, right? And so that was cool. And then I basically copied this tutorial and I spent quite a while trying to figure out how to do it, and I ended up spinning up this Lambda Labs instance. And so there's a, Or I, I did this locally initially. I did it with Pokémon. So if you just copy, literally just copy paste this code, just go through this, you'll end up with this. And then you can very easily locally run these commands in your console and get these images spit out. And then once I did that, I was like, okay, that's cool. I need a set of images that I can use that are tagged something by something else. And so I had this idea. Of going to the Reddit Progress Pics subreddit and I downloaded 1,000 of these similar, like that script of scraping Apologies tweets or whatever. Similarly wrote a script. I, and I did all of the thousands of, 1,000 of these images and labeled them with the titles of these posts. And you can tell they're like labeled, right? Where, sorry if these are NSFW or whatever, but they're labeled, right? Where they're like, there's. Female, 30 years old, 5'2", et cetera. So I was like, okay, I can— there's some— obviously I have no idea what this model is gonna turn out. It's gonna be weird, but at least there'll be, there should be some significance. And so that's what I did. And so I built this thing and I'll show you the code that I wrote to do it because it's on GitHub. Yep. Right here. And so I took that repo, right? And I just cloned it. And I started work— working on my own, the changes that I would need, right? For example, and I write my own READMEs even though no one's gonna read them, just so if my computer gets destroyed or something, I can spin back up. But basically I took his README and I just started adding my own stuff, right? So for example, he was— he didn't talk too much about, okay, I— you need to like actually label these things. And so I wrote a script called prep_images that takes all those images and converts it into the right format. I wrote a conversion script, things like, for example, if you wanna split the images, like all of this stuff you have to do., right? So this is all stuff that like has nothing to do with AI. You're just, you just need 1,000 images that are labeled nicely. And so you test it like this. And then I was using Lambda Labs. Lambda Labs is imagine AWS, but for GPUs basically is a kind of a simple way to say it. GPU instances. You would just similarly to spin up an instance, you would, you spin up a GPU and then you can SSH into it and basically, use it like a local computer except it's in the cloud and has a gazillion times more power than your local thing. So I had to figure out how to do that and then train the model. And so you basically run this thing and then it spends like 24 hours training and you, it spits back this checkpoint, which is basically like a binary file that has all of this. It's like a brain, like as, as how I think about it is it's the thing that it's the API that you can ask questions. And then you have a script. So I basically took this script. And ran it. And so I took that prompt and I made my own prompt, which is based off my own weight loss and said male, 30 years old, 5'7", et cetera. And it, I wanna, what, what is this? And let me, let me pause my share and see if I can actually find an image that I sent someone. Cause it, they're pretty, and pretty grotesque to be honest. Let's see if I can find one. But again, you don't know. If this is gonna be any good until I'll show you a less than grotesque one. Yeah. So here's one that, yeah, so this is one and this, this literally that, that text generated this image. And obviously there's nothing useful here. I don't know what you would do here, but it's like crazy to me that like it picked up quite a few things. Like I would say, for example, if you put in someone who's female, 85, who's 500 pounds going down to 200, it'll create a very different image. And it won't be, it'll be like this in the sense that it's like weird and has artifacts and is actually not that specific to like the specific weight or, or whatever. But it's clearly there, right? Where it's able to pick up certain things like age overall, maybe, maybe it's to, to the hundreds of pounds or something like that, the sort of the tolerance level. But anyway, I was pretty proud of this. Like I spun up a model and I now had a thing that could, if I wanted to, I could build a little website that could allow people to upload an image and get this weird thing out. Right. And so that was like the first experiment that I ran. Let me see what else I have here that, yeah, using, using this example code base. And I was pretty, pretty happy with that, but wasn't, I didn't think there was any real market opportunity. There was no real value in that to the market. And so I was like, okay, what else can I build? And then I had this idea of using, basically taking my book, which is about 52,000 words and basically trying to figure out if I could— initially it was looking at it through this lens and this lens, I guess I should explain, is called fine-tuning is the approach, right? Which is you take an existing model and you retrain it based on a very limited set of data. And that way it's good at producing answers to that kind of thing, right? And so my initial approach was like, oh, I can just like fine-tune a model on text, on my book, like I can have a bunch of questions and then I can have a bunch of answers and then I can have a model that sort of is able to do that. And I have trained that and it just was terrible. It's not good enough yet. I think it needs to be trained on way more questions. And then I found another approach, which is embeddings. And so there's fine-tuning, which is this one, which is, has some pros, has some cons. And then there's embeddings. And embeddings is the thing that I got really excited about because it's a really good fit for specifically questions and answers. And the way to think about embeddings is, it's complicated and actually came out in like January or something. But the way to think about embeddings is if we go back to this prompt, at least the way that I think about it is, right? This is the thing that we're familiar with and it's really good generally. But if I asked it a question like, question, what is the minimalist entrepreneur about answer, it would probably, it would basically just do the equivalent of a Google search. Like it wouldn't be specific to who Sahil Lavingia is supposed to be answering this. It, it's supposed to have some content from the book itself. And so what embeddings does is it basically allows you to add stuff like to the prompt and do it in a very smart way. Instead of, I can't just take the entire book and put it in. This is the book, The Minimalist Entrepreneur, 52,000 words. Question, answer, right? That's, that would be ideal actually, because then in theory, like I don't even have to use embeddings. It's just insanely simple. And I'm, I assume that's eventually where we'll get at some point. But for now there's like a limit on how much sort of you can stuff into the prompt. It also costs money, right? If you have, even if it is possible, you might want to be more efficient with it. And so what basically what embeddings allows you to do is take like the best or like the most relevant words of those 52,000 and stuff those into the prompt, right? So for example, you could say, saw, and this is, and I actually, I'll just show you now, like kind of the code example. Let me this one, right? So this is like a local Python script, which basically does what I was saying before. And you can see here, oh yeah, I've been testing some stuff. But basically it takes, it, you can, this is the, this is the important line, which is it takes the most relevant document sections, right? So it pulls these from. This embeddings.csv and from the context embeddings, which come from the book, which is just like pages.csv somewhere. Where is that? Context embeddings. Oh yeah, here it is. So it downloads these files and anyways, yeah, basically fetches them smartly. So it, it would be like doing a really simple Google search except it's just better, right? So if I wanted to, if my level of engineering, like I'm not the best engineering engineer on the planet, what I would do is I would write a really simple like search, right? If I was coming up with my own AI, I would basically say I would write a really simple script that would maybe try to pull out nouns from the question, right? So for example, let's say the question is like, how do you define community? I would have to come up with some sort of script that would try to pull out the nouns. And then I would try, I would probably do a search on that 52,000 words and I would like probably find all the instances of those words and then I would pull maybe 100 words before, 100 words after, and then like stuff that into the prompt. And that would be my hacky version. And that's the software 1.0, right? Where it's very ma— it's completely deterministic, right? It's just looking for equals, right? You can't do any fuzziness. Like for example, what if I didn't use the word community, but I used the word group, right? Or something like that. And so that's where the magic of AI comes in and embeddings comes in is it's like doing that, which is, it's like asking the question, Based on this question, right? Let's say this question was question equals, let's say, what is community according to you, right? It would basically send that to the book or a file that resembles the book. It's basically the book, but turned into this mathematical representation of all the concepts, talking about all the other concepts in the book and says, hey, based on this question, what are the 5? I believe that's how this code works, which, or this might just get all the, the top let's say 50 most relevant sections, right? So based on how I fed, how I created the data and the way I did it specifically is that every page is basically a section. So basically what are the 5 or what are the most, the 5 most relevant pages from the book and then go through them until there's no space left in the prompt. And then the prompt itself is also engineered, right? So for example, Let me just pull in what I actually wrote. This is basically the prompt, right? Which is Sahil Lavingia is the founder and CEO of Gumroad, blah, blah, blah, blah, blah. Context that may be useful. And I, I try to come up with, and I'm sure this is like super MVP, right? So this is like kind of part of what's really fun is you can, there's like a kind of a new kind of engineering that didn't exist before. And anyway, so you, I came up with this simple line. These are questions and answers by him. Please keep your answers to 3 sentences minimum. Speak in complete sentences. Stop speaking. This doesn't really, like, you're just like trying stuff out, right? Like one of the The cool things is like all this stuff will eventually get blogs blogged about. Like people will figure out what's, what works well, right? But right now we're all like digging for gold in a way. And then context that may be useful, pulled from the Minimalist Entrepreneur. This is where these 5 pages would come from. So it would say like page 1 and then it would just say community, right? Something like that. Page 57. Again, you can just imagine 5 of these and then question. And then that would come, what is community according to you? Or actually here there would be examples. So you go back to the code, there's like a bunch of these examples that I stuff too, right? So there's all these examples. So it would be like this. And so I come, come, I try to stuff it with examples that kind of resemble my voice or, or whatnot. And then it would be something like this, right? And then boom, this is, would be the kind of the answer we can get spit back out. And then that would return that to the user, which is exactly what happens. And so if you go back to the code base, I can explain now, you, you know, you have all the concepts now. It's just going through it. You go to this website, askmybook.com, which is just this input. You ask this question. It's The Minimalist Entrepreneur is a book. So you don't have to listen to my voice twice over. And what does this do? This is just a page. Really simple. Actually, let me view source. It'll be better. You can ignore all the style, but basically it's just a form, right? It's just this form. It's just a very— you don't even need JavaScript in theory. And all it does is it takes that question and it sends it to the backend. There's also this I'm feeling lucky button that basically just pre-populates it with a bunch of stuff. And there's some— this is all— you can look at this code. Literally, if you copy paste it, it should work. Like it's not. It should, it, it's not minified or anything, so you can see my ugly code and then it hits that backend. And so let's go to the backend views and see what happens. So it basically goes and hits this def ask backend and it does exactly like that script. Imagine like I took that local script, cuz once I had that local script running and working, all I gotta do is host that script on the internet, right? And then basically hit that script and it'll return and then show it to the user. And so it hits, I get this question asked. I, if it doesn't have a question at the end, a question mark at the end, I add one. And then this is a cache, right? So if the question has been asked exactly like it's been asked before, it just auto returns it. Otherwise it does all that stuff that I talked about, which is it, it get, it fetches the pages and then it fetches those embeddings, which is that abstract mathy representation. And then it answers that question, right? And the way it answers that question is exactly that same thing, which is if you just go to answer question. Answer query with question. It does the prompt and this is the OpenAI completion. All I do is I just send that prompt and some parameters and return what it returns, right? And if you go to the prompt construction, which is this, the def construct prompt, it'll just take you right here and you'll see that part, right? Which is basically based on those pages, which is that CSV file and those embeddings, which is another CSV file and the question, which is a string of the question the user asked. Do that thing, right? Go through those pages or go through the, use the embeddings, like basically use the embeddings to figure out what pages are most relevant. And then based on how much space, which is 500 tokens, I think, stuff it and then add these, add that sort of prompt engineering bit and return that string. And then it comes back with that string. And I'm not done yet because there's one more API, which was that voice. Right. And which is pretty simple, which is I just take that answer, which is a string, right? The answer is like the middle song is about whatever. And then I hit this other API with a bunch of parameters. Doesn't really matter. They're just, just copy pasted from their docs. The only things that matter are basically that I'm hitting the right voice, basically using my voice. And then I just have the string of the answer and it just spits back this response. And then, which is a JSON object. And then I just basically just spits it back the URL of the audio that I would need to play. And then I return that as JSON. So I have the question, I have the answer, and then I have the URL of the audio. And then if you go back here in this code, you'll see basically get the audio and then, or get the audio element and then basically play it, use it to play that. Basically take that URL, stuff it into that audio element and hit play, right? Start playing that audio at 50% volume cuz no one needs to hear my voice that loudly. And that's it. That's really the, that's all that this is. And if you look at how many lines of code is this, right? This is obviously there's a little bit more and I'll go through the scripts as well, but this is like the index file is like what, like 100 lines of code or so. There's the views, which is about, and I'll show you, there's a little bit more complexity here, but if you really look at this, it's just, it's probably another 100 lines of code or something like that. And that's it. Pretty simple. And then I wrote some other scripts, like for example, I have to take the, I have a PDF, right? The manuscript PDF. I have to convert that into that CSV. So I wrote some code to do that. Just extracts the pages, about 50 lines of code. And I'm sure this could be done in 10. It's not the most efficient. Then I similarly, I take that CSV and I have to generate those embeddings. This is even simpler, which is literally like completely copy pasted from OpenAI. The only thing I change is like making sure I pull the right file. And then I have this script, which is the thing that I wrote that, that basically using the embeddings. And the pages CSV. And then once I had that and I was like, okay, cool, I have a little local script that I can ask questions. Now I just have to get it onto the internet. And really simple to do that. I just literally just this, I just followed this documentation and then I replaced the index with my own code and the, wrote that, basically took that prompt and that I, that Python script and kind of put it in the, in a, in the backend. And yeah, that, that's about it. It took me about 10 hours, I think, all in to build. Let me see what el— what other things I have to show. I think I showed everything. So yeah, this is the example that OpenAI provides. That is how to— basically they, the example they use is questions about the Olympics. And so they download a bunch of, they create embeddings off of Wikipedia. So basically instead of creating embeddings off Wikipedia, I have to create them off of a PDF of my book, right? A little different, but the end output is the same, which is you have a CSV of all the data, right? Basically that the embeddings can be trained on. And then similarly, you can, you, you can see how they do this kind of prompt engineering. And you can do stuff like this, right? You could say, answer the question as truthfully as possible. And if the question is, if the answer is not contained within the text below, respond saying, I don't know. So for example, a bunch of people have asked me like, this is, Ask My Book is currently very broad and I wanted it to be broad cuz I'm curious the kinds of questions that people will ask this. But I actually think it should be pretty specific. I think ideally, and you could totally do this, I think someone asked like, can you, can this link to the page that it gets the information from? 100%, it could do that, right? Because ultimately, at the end of the day, all you have to think about is what are you putting in this, right? So if you are able to take something and put it in here to fit above the fold, then in theory GPT-3 should be able to help you out. So for example, let's say You, I wanted to return the pages, right? I wanted to, I wanted it to say, if you're interested in learning more, reading more, this is in page, let's say this is in chapter 3, ideally. Let's go for the ideal, right? Which would be, I would say, yeah, community is a group of people with a shared interest, purpose, or goal. If you're interested in learning more. Check out pages 52 to 57, chapter 3, we need first or whatever, right? Something like that. You would basically just have to say, okay, how do, like, how do I get GPT-3 to do that, right? Like, how would I get it to respond? And one would be All of these examples, you'd probably wanna provide that, right? So you'd probably want it to train it with a pattern, right? And so you would go through and you would, you would do this realistically. You would get the right answer and then you would, you would say, okay, this is page 28 to 29 in chapter 2, profits even firster. And boom, all of a sudden it will start doing it. It'll be wrong because it has it doesn't actually know like the chapters necessarily, or maybe it'll be surprisingly right. I've been impressed sometimes with how good it is, but I think at least it'll start to respond with stuff. It'll start to say, if you're interested in learning more, check out pages, and it'll just try. The cool thing about AI is it doesn't have imposter syndrome, so it'll try and it will spit something back and it might be stupid, really bad, right? And then you'd be like, okay, it doesn't have it doesn't know. It has to, you have to, it's not fair. Like you can't make someone take a test without giving them, allowing them to study, right? Or think about it like an open book test. So you'd have to get smarter about the embeddings. And I think that's where kind of the engineering would come from, which is you have that manuscript PDF and you have to get it, get the right context into the prompt. And so what I would do is I would basically go back to that script and so go back to PDF to Pages content, right? You, this is a, you know, very simple script. It basically opens up the PDF and then it goes through each page and it extracts it. And all it extracts basically is, it basically puts it into a CSV where the first column is page 7 and the second column is the text. And then the number of tokens, which I don't think I actually use, but I copied it from the internet. So that's there. And so it's kind, it's like this, right? But it won't be able to do things like that or the chapters. And so you would basically, I would have to be clever and say page 1, and then I would have to basically make this script smarter and smarter so that it says chapter 1. Or you could have, you could say page 1, chapter 1, page. And so every page is tagged with a thing, and then if you provide examples that are referencing this, it should be able to guess and say, oh yeah, okay, cool. It's, I'm answering this question based on this, and so I should be able to respond. Oh, Ted, what's up? Let me ask, answer some other questions and then I'll let you chat. Mim says, how does the embeddings know what pages are most relevant to the answer? So how does it know is a good question. That's like the AI magic. And let's see what they say. It's a dense information, dense representation of the semantic meaning of a piece of text. Each embedding is a vector of floating point numbers such that the distance between two embeddings in the vector space is correlated with semantic similarity between two inputs in the original format. So basically it's compression, right? And so instead of— let's say you have 50 images, instead of having 50 massive images, you have 50 labels of the images, right? So it's kind of like that, where it's just you don't have 52,000 words, you have like this concept connects to this concept and this concept connects to this concept. And it's just, yeah, it's just much more broad. I'm sure Ted can actually answer that way better. So let me see if he, if he wants to give it a shot. Ted, what's up?

Hey, how's it going?

Can you hear me?

Yeah. Sorry that you, what was embeddings? I was typing the question to you in the chat box, so I didn't hear what the prior question is.

I was wondering if you wanted to take a stab at answering what embed, how, like how embeddings work or what's magical about them.

Oh yeah, sure. I, there, I think a great, you, the way you described it as think of it as building your own search engine is great. I think another, in terms of how to think about what they actually are, you can also think of it as like the MRI of a neural network when it's thinking about a thing. So one way to talk to GPT-3 is to give it some input and then look at what it says back. But another way to talk to it is to give it some input and then take an MRI of its brain right in the middle of it thinking about your input. And then the reason why that's interesting is that gives you back a bunch of numbers. But those numbers have this really useful property, which is it— the numbers, if you think of the numbers as coordinates in some big space, like you're plotting them on a graph, similar things tend to be nearby similar things. And so when it's thinking about your question, the set of numbers it's giving you back, if you were to plot that out in a giant graph, those dots on the graph, it's going to locate your question into a similar spot. On that graph as where it located the answer to your question. Because that's how, you know, when you think about it metaphorically, that's like what your brain should be doing. And so it's just a way to like peek into the MRI scan. And there's some good things about it. There's some tricky things about it. But it gives you the answer to what other things do I already know about that are nearby?

This is how I think about it. Awesome. Cool. Yeah. And I think this, this is like what it might look like if you were to visualize it, which is almost yeah, just kind of. MRI of like how these concepts in your brain may be connected to each other. It's kinda like when you smell something and then like your brain is like gonna spin up like 20 different memories or something. But you have gazillions of memories, right? Like how is it picking those? It has to because you'd go insane if you just got every memory. So that's what the brain is good at. It's good at surfacing like the right, when I do, when I say like, when I talk, I'm able to, my brain is constantly like getting all of these weights and able to fill in the next thing I'm going to say based on all the, all the things I've heard and all the things I've said. And, and that would take just too long. So it has to get efficient. And that's what I think is cool about GPT-3 is it made, it's really like you're taking 52,000 words, you're finding the right 500 words and you're using that in the prompt. And so for me it's really a simple thing, which is it's yeah, it's like Google search except you're not searching the whole internet, you're searching a refined space, but with the power of, and the flexibility and the fuzziness that Google has. It's just like, I can't build my own Google, but all of a sudden I can build my own search engine on my little, on my document. And it scales, right? You can take that script if someone wanted to, just like Wikipedia was mentioned before, you could do this. I'm gonna talk to Tiago Forte from Building a Second Brain. Ted, you might have actually, have you talked to him? Did, did you reach out to him? Someone I saw reached out to him, so maybe he's already figured it out, but he has like a million words sitting around. He's like the king of that. And so I think this, it would be great if he could be able to ask a question. Right now he probably just has to do Ctrl+F and say, okay, I'm, I don't know, I'm woodworking or something, but you're gonna miss all the things in which maybe he misspelled woodworking. And so anyway, it's pretty cool. And then Mim asks, is the voiceover dynamic, dynamically generated? So it is. Yeah. So basically it's just another API that I hit that you, you, and I'll show you actually what it is. It's called Resemble. It's pretty cool. See if it works. So you can see there's been 3,500 clips generated. They have a UI, so if you wanna do this manually, like if you're you have some more manual, smaller scale use case, you can just go in and type a thing. You have your voice in here. So I just sent them some recordings of mine and they were able to spin up this voice and it's pretty crazy. And then you just type into this and it just, it's pretty fast. Well, yeah, I don't know. Maybe it's not that fast. It's slow right now. Hi. Hi. Yeah, sounds like how I would say it. And yeah, what other things are here? No, I think I covered everything. Any other questions? I noticed you are using the Curie model. Why not DaVinci? Speed and cost would be the, basically how you would actually speed, cost, and you know how good it is, right? So you have to just measure. I think Curie is just 10 times cheaper than DaVinci. And so it's just a really easy, I would think about DaVinci is what you would use if you're like really trying to impress someone. And then in anything else, you probably want to use Curie or something like that. And you can see this if you go DaVinci. There's Curry. You can, it's actually kind of crazy, like how, how big the gaps are. It's almost you wanna hire a first grader or a third grader. So you can see Ada is both the fastest and the cheapest, but presumably it, it's not gonna get super smart. So if you're at, if you're building, let's say a math thing, I would like a, let's say a little like chatbot where you can ask it fuzzy math questions and it would answer. I would say it's probably, you could lean more to this side, but if you're asking things that are more conceptually interesting, maybe this side, but you can see like the difference, right? So this is 4 thousandths of a cent or 4 ten thousandths of a cent. And this is 2 cents. This is, it's something like a 50x difference or something like that. It's a big difference, or sorry, of a dollar, not of a cent. So yeah, you'd wanna experiment with those things. And speed is another one. Ted says, what did you include in your embedding lookup? Every sentence, first sentence of paragraphs, just section titles. So I put in just pa— literally it's just page 1 is the first column. So page n and then the entire, whatever the text is on the page, literally. So it's like very, it's not that smart at all. Even, and I use like the, some one version that I had laying around. So that has all these weird comments and stuff in it and it's still pretty decent. But yeah, I think that's a good example of something that I think you could severely improve and I think there's a huge opportunity for kind of startups to get into this, right? Where you're able to set up the right sort of format. And so let's say you have a book, you might want to be able to split it up into sections or chapters or paragraphs. I think there's a bunch of different things that will work better for different things. I tried to, I, in an ideal world, I would've done probably, I probably would've included page, chap, and section. And done it per paragraph, right? So every paragraph was its own, or every sec— actually every section was its own line in the CSV instead of every page. And then, but that I just, yeah, haven't done that yet. Julie asks, is improving the quality of the embedding solely an engineering challenge or is it a writing content dataset challenge too? Could you have written the book differently so that Ask My Book works better? Yeah, I think It's mostly a, I guess it's both. Like you can certainly, if you were writing content specifically to train the AI, I would think about it like a database of a data, right? Like I would say, I like, I'm writing that, I'm taking that PDF and I'm using that Python script to turn it into this CSV. And I can show you what that CSV might look like. If I go to AskBook and find The pages. You can see an example of what it's really simple. Literally title, page 1 content, and it just stuffs it with introduction, fa— this is getting, it's the page number in the corner that it's probably pulling, right? Chapter 1. It's just literally, it's just that. And so you could make it way smarter, but I would think about it like this is how you would produce the content in the first place if you were using it to be trained, right? You would never even need the script. You would create like a Notion database or table or Google Sheets or Airtable, and then you would just create the data in that format right away. So it's, I would say it's 80 to 90% a content, like getting the right data content and getting into the right format. And then a small really engineering challenge, but the engineering stuff is like, it's pretty simple in my view. Mo says, what text use cases do you think it wouldn't be suited for? I'm curious about fiction. Is it, would it be useful? If you were to, to be able to ask questions of the Harry Potter franchise or something like that. My guess is asking questions, it wouldn't be that interesting. That's probably better for Google or Wikipedia or something, but genera— generating might be interesting, right? So for example, if you were like, hey, I wanna write a story, and what if that embeddings was able to pull from the canon that JK approved, you could build a Harry Potter writer or something like that. But I think, sorry, I meant to ask what it's not suited for. Which I'm probably too optimistic about this. I can't see like what it wouldn't be good for. Maybe if it, there's certain things that you can't compress that much, right? So for example, Gödel, Escher, Bach, or like certain, certain books like that. There's just no compression algorithm. Ironically, Gödel, Escher, Bach is a lot, largely about information and compression and things. And highly recommend, by the way, reading that book if people are interested in this kind of stuff. But yeah, if there's something that like you can't compress, like it would just, for example, like an M. Night Shyamalan movie or something like that, right? Where it's like, about the whole thing only works, or Fight Club or something like that. It wouldn't work for that kind of stuff. And it's just AI, right? So you can almost think about it. There's certain things that are just like higher grade reading level. Why is that? Because it's more complex, different words, like all the, these reasons. And I, I would assume a lot of those map probably to the order in which AI will be able to think quote unquote about it and, and answer questions. Flynn asks, what are your next steps with Ask My Book? Oh, I can show you. So I have a little Notion that I use. Everyone probably knows that I like Notion. And so I just have a little thing that I talk about, or I, my to-do, my internal to-do list. So it's pretty short, which is basically I wanna obfuscate the IDs right now. It's like question 1, 2, 3, 4, right? So I wanna change that so that people can't see what other questions people are asking. I think one really interesting thing about this is that you can ask someone questions without them knowing that you're asking them questions, which I think could be imposter syndrome. Like it might be beneficial to be able to ask Peter Thiel a question without him judging you for being stupid or you thinking that he would, right? So I wanna obfuscate the IDs so people don't get a sense of the other kinds of questions people are asking. And then I wanna do it for another author. I wanna, I mentioned I wanna try training it on like tweets and other things and see if I can see if it scales and how fast I can do it the second time. Uh, and then also just get a sense of what's different, what's the same in two. Once you go from one to n, it's like going from making one car to making two cars, right? You have to start to think about the factory a little bit more. And then I'm gonna meet with Penguin Random House, which is my publisher. And just get their thoughts on what they think about this, if it's interesting to them at all, whether this is— would this be something you would embed in like an author's website? Is it too much? Are you giving away like the farm when you allow people to ask questions of a book? Is this— a lot of publishers are concerned about the sort of Amazon monopoly on books and digital books and audiobooks. So maybe there's like a new format that might be compelling to them. Who knows? Oh, and then I'll— and then the last one is to try fine-tuning a model With a bunch of real answers. So one layer, oh, one thing that I didn't talk about that I should is if you go, if you can't see this because you have to be logged in, but basically I see all the questions that people have asked that I haven't yet answered myself. And so basically I want to go through and answer these kinds of questions. I built this like little dashboard and you can imagine similar to that CSV, if you have a similar CSV of like question, answer, question, answer, question, answer, question, answer. Then I want to try fine-tuning a model based on that. So instead of effectively, instead of having to stuff the prompt and you're gonna be limited with how many of these you can do, you can almost think of you train a model, a custom model on hundreds of these. They recommend 500 or more, which is why I haven't done it. It's just gonna take a while. But then you could do both. You could have, you could still have this and then you could, you could simplify and you can try different things in terms of where you want the intelligence to come from. And you can see here I've actually trained a couple different models. So there's two that I've fine-tuned on 10 answers just to see, and it's not good. It's not, it just says weird stuff. And I'm like, oh, that's creepy. Cuz I train on 10 answers just to do it and go through the code.

Yeah.

And in theory, if you train it enough, you don't need anything, right? You can just do Q or you, you, you don't need as much because you, in theory, you've taken all of that context and put it into the model itself, right? In the brain. Like a very kind of like shallow layer of the brain in a sense, right? So yeah, that, those are some of the next things that I wanna do with Ask My Book. So it's a combination of better data, better prompt engineering, and then advancements just generally in GPT-4, et cetera. And then tweaking the parameters. There's all these kind of temp things you can tweak. There's also a lot of, I think, magic that's really simple heuristics. For example, I want the, I want it to be more conversational, which means that there should be longer sentences and shorter sentences so that But that's simple math, right? I can just pick a random number between 10 and 500 and it's— so you can get smarter and smarter over time. So there'll still be like dumb parts, right? Quote unquote dumb parts of me just like pretending to be smart, pretending to be AI, but then isn't that what AI is trying to do? Right? So it's all just layers of— it's all a spectrum, right? Intelligence. So anyway, I think I answered everybody's questions. Oh, there's one more, which is how good is fine-tuning? For images, it's very good. For text, it doesn't seem to be that valuable, but I think it'll get there. I think my plan at some point is to go through these, answer as many of these questions. Like for example, I say the book is something like that, right? And I would hit answer and then it would train. It would, right now it doesn't actually do anything. It just stores that data. And then eventually I export that data, train a new model on it, and hope at some point it gets good enough that I don't need as many Qs and As and things. And the reason this takes so long is 'cause it's badly written code. I'm sure I could make this like much better. If an engineer wants to help make it better, I don't know if this will be a product. We, what I don't know, I'm just, I just wanna learn. I want to focus on basically like I asked my own book, the what should I do? And it says to focus on solving a problem that you have. So that's what I'm doing. And we'll see, we'll see where things go. So hopefully this was useful. Everyone knows hopefully where to find me @sahil_lavingia on Twitter. If you have questions, happy to send you stuff, add you to code, send copy paste code to you. But I highly recommend just like, just copy, just do exactly what Justin has done with the Pokémon thing, with the Olympics example, like just try to do exactly and then start messing around. But I'm excited. I think it's gonna be an interesting few years. So hopefully that was useful and I'll catch you guys later.