The AI Founder Gene: Being Early, Building Fast, and Believing in Greatness — with Sharif Shameem of Lexica

Hey everyone, welcome to the Late in Space podcast. This is Alessio, partner and CTO in residence at Decibel Partners. I'm joined by my co-host Swyx, writer and editor of Late in Space.

And today we have Sharif Shameem. Welcome to the studio.

Awesome, thanks for the invite.

Really glad to have you. You've been a dream guest, actually since we started. drafting guest lists for this pod. So glad we could finally make this happen. So what I'd like to do is usually introduce people off of their LinkedIn and then prompt you for what's not on your LinkedIn and to get a little bit of the person behind the awesome projects. So you graduated University of Maryland in CS?

So I actually, I didn't graduate, but I decided—

You did not graduate? You dropped out?

I did drop out.

What was the decision behind dropping out?

So first of all, I wasn't doing too well in any of my classes because I was working on a side project that took up most of my time. And then I spoke to this guy who ended up being one of our investors. And he was like, actually, I ended up dropping out. I did YC and my company didn't end up working out. And I returned to school and graduated along with my friends. I was like, oh, it's actually a reversible decision. And that was like that. And then I read this book called The Case Against Education by Bryan Caplan. So those 2 things kind of like sealed the deal for me on dropping out.

Are you still on hiatus? Could you still theoretically go back?

Theoretically, probably. Yeah.

Yeah.

Still on indefinite leave. Yeah.

Yeah. Then you did some work at Mitra.

MITRE.

Yeah.

So they're like a lesser known, so they're technically like an FFRDC, a federally funded research and development center. So they're kind of like a large government contractor, but nonprofit. Yeah, I did some computer vision work there as well.

But it seems like you always had like an independent founder bone in you because then you started working on VectorDash, which was distributed GPUs.

Yes. Yeah. So VectorDash was a really fun project that we ended up working on for a while. So while I was at MITRE, I had a friend who was mining Ethereum. This was, I think, 2016 or 2017.

Oh my God.

Yeah.

And he was mining on his NVIDIA 1080 Ti, making around like $5 or $6 a day. And I was trying to train a character recurrent neural network, like a character RNN on my iMessage text messages to make like a chatbot of myself. Because I was just curious if I could do it because, um, iMessage stores all your past messages from like years ago in like a SQL database, which is pretty nifty. But I wanted to train it and I needed a GPU and it was, I think, $60 to $80 for like a T4 on AWS, which is like really slow compared to a 1080 Ti. If you like normalize the cost and performance versus the 1080 Ti and someone mining Ethereum, it's like a 20x difference. So I was like, hey, his name was Alex. Alex, I'll give you like $10 if you let me borrow your 1080 Ti for a week. Like, I'll give you $10 per day. It was like $70. And I used it to train my model and it worked great. The model was like really bad, but like the whole trade worked really great because I got a really high performance GPU to train my model on. He got much more than he was making mining Ethereum. So we had this idea. I was like, hey, what if we like built this marketplace where people could rent their GPUs where they were mining cryptocurrency and machine learning researchers could just like rent them out and like pay a lot cheaper than they would pay AWS. And it worked pretty well. We launched in like a few months. We had, we had over 120,000 NVIDIA GPUs on the platform. And then we were the cheapest GPU cloud provider for like a solid year or so.

Yeah.

Like we, you could rent like a pretty solid GPU for like 20 cents an hour and cryptocurrency miners were making like more than they would make mining crypto. 'Cause this was like after the Ethereum crash. And yeah, it was pretty cool. It just turns out that like a lot of our customers were like college students and like researchers who didn't have much money and they weren't necessarily the best customers to have as a business. Startups had like a ton of credits and like larger companies were like, actually, we don't really trust you with our data, which makes sense. Yeah, we ended up pivoting that to becoming a cloud GPU provider for video games. So we would stream games from our GPUs. Oftentimes, like many were located in a, just a few blocks away from you because we had the lowest latency of any cloud GPU provider, even lower than like AWS and sometimes Cloudflare. And we decided to build a cloud gaming platform where you could, uh, pretty much play your own games on the GPU and then stream it back to your like Mac or PC.

So Stadia before Stadia.

Yeah. Stadia before Stadia. It was like a year or so before Stadia.

Wow. Weren't you like jealous of, I mean, I don't know, it sounds like Stadia could have bought you or Google could have bought you for Stadia. Like that never happened?

It did never happen. Um, yeah, it didn't end up working out for a few reasons. Uh, the biggest thing was internet bandwidth. So a lot of the hosts, uh, the GPU hosts had lots of GPUs, but, uh, average upload bandwidth in the United States is only 35 megabits per second, I think. And like a 4K stream needs like minimum 15 to 20 megabits per second. And then, uh, so you could really only utilize one of those GPUs, even if they had like 60 or And then you went to Debuild, July 2020 is the date that I have.

I'm actually kind of just curious, what was your GPT-3 aha moment? When were you GPT-3 pilled?

Okay, so I first heard about it because I was also working on another chatbot. So this was after— everything ties back to this chatbot I'm trying to make. This was after working on VectorDash. I was just hacking on random projects. I wanted to make the chatbot using not really GPT-2, but rather just It would be pre-programmed. It was pretty much you would give it a goal and then it would ask you throughout the week how much progress you're making to that goal. So it would take your unstructured response, usually a reply to a text message, and then it would like plot it for you in like a table and you could see your progress over time. It could be for like running or tracking calories. But I wanted to use GPT-3 to make it seem more natural because I remember someone on Bookface, which is still YC's like internal forum.

Oh, cool.

They posted and they were like, OpenAI just released AGI and it's GPT-3. I asked it like a bunch of logic puzzles and it solved them all like perfectly. And I was like, what? How's no one else talking about this? Like, this is either like the greatest thing ever that everyone is missing or like it's not that good. So like I tweeted out, uh, if anyone could get me access to it. A few hours later, like Greg Brockman responded.

He is everywhere.

He's great.

Yeah.

He's on top of things. And yeah, by that afternoon I was like messing around with the API and I was like, wow, this is incredible. You could like chat with like fake people or people that have passed away. Uh, you could like, I remember the first conversation I did was, uh, this is a chat with Steve Jobs. And it was like, interviewer, hi, uh, what are you up to today? And then Steve, and then like you could talk to Steve Jobs and it was like somewhat plausible. Oh, the thing that really blew my mind was I tried to generate code with it. So I'd write the function for a JavaScript header or the header for a JavaScript function and it would complete the rest of the function. I was like, whoa, does this code actually work? Like I copied it and ran it and it, it worked. And I tried it again. I gave it like more complex things and like, I kind of understood where it would break, which was like, if it was like somewhat, like if it was something you couldn't easily describe in a sentence and like contain all the logic for in a single sentence. So I wanted to build a way where I could like visually test whether these functions were actually working. And what I was doing was like, I was generating the code in the playground, copying it into my VS Code editor, running it, and then reloading the React development page. And I was like, okay, cool, that works. So I was like, wait, let me just put this all in like the same page so I can just like compile in the browser, run it in the browser, and then like submit it to the API in the browser as well. Um, so I did that and it was really just like a simple like loop where you just type in the prompt, uh, it would generate the code and then compile it directly in the browser and it showed you the response. And I did this for like very basic JSX React components. I mean, it worked. It was like pretty mind-blowing. I remember staying up all night, like working on it and it was like, the coolest thing I'd ever worked on at the time so far. Yeah. And then I was like, so mind blown that no one was talking about this whole GPT-3 thing. I was like, why is this not on everyone's minds? So I like recorded a quick 30-second demo and I posted on Twitter and like, I go to bed after staying awake for like 20 hours straight. And I wake up the next morning and it had like 20,000 likes and like 100,000 people had viewed it. I was like, oh, this is so cool. Um, and then I just kept putting demos out for like the next week.

Yeah.

Yeah, that was like my GPT-3 spark moment.

And you got featured in Fast Company, MIT Tech Review, a bunch of stuff, right?

Yeah, yeah. I think a lot of it was just like the API had been there for like a month prior already.

Not everyone had access.

That's true. Not everyone had access.

So you just had the gumption to tweet it out. And obviously Greg, on top of things as always.

Yeah, yeah. I think it also makes a lot of sense when you kind of share things in a way that's easily consumable for people to understand. Whereas if you had shown a terminal screenshot of it generating code, that'd be pretty compelling. But whereas seeing it get rendered and compiled directly in front of you is a lot more interesting. There's also that human aspect to it where you want to relate things to the end user, not really just like— no one really cares about evals when you can create a much more compelling demo explaining how it does on certain tasks.

Okay, we'll round it out soon. But in 2022, you moved from Debuild to Lexica, which was the search engine. I assume this was inspired by Stable Diffusion, but we can get the history there a little bit.

Yeah, so I was still working on Debuild. We were growing at a modest pace, and I was in the Stable Diffusion—

I was on the signup list. I never got off.

Oh yeah, well, we'll get you off. It's not getting many updates anymore, but yeah, I was in the Stable Diffusion Discord, and I was in it for many hours a day. It was just the most exciting thing I'd ever done in like a Discord. It was so cool. Like people were generating so many images, but I didn't really know how to write prompts and people were like writing really complicated things. They would be like, like a modern home training on ArtStation by Greg Rutkowski, like 4K Unreal Engine. It's like that, there's no way that actually makes the images look better, but everyone was just kind of copying everyone else's prompts and like changing like the first few words.

Yeah.

Yeah.

So I was like using the Discord search bar and it was like really bad because it showed like 5 images at a time. And I was like, you know what? I could build a much better interface for this. So I ended up scraping the entire Discord. It was like 10 million images. I put them in a database and I just pretty much built a very basic search engine where you could just type for, type a word and then it would return all the prompts that had that word. And I built the entire website for it in like 20, in like about 2 days. And we shipped it the day, I shipped it the day after the Stable Diffusion weights were open sourced. So about 24 hours later. And it kind of took off in a way that I never would have expected. Like, I thought it'd be this, like, cool utility that, like, hardcore Stable Diffusion users would find useful. But it turns out that almost anyone who mentioned Stable Diffusion would also kind of mention Lexica in conjunction with it. And I think it's because it was like, it captured the zeitgeist in, like, an easy-to-share way where it's like this URL and there's this gallery and you can search. Whereas running the model locally was a lot harder. You'd have to deploy it on your own GPU and set up your own environment and do all that stuff.

Oh, my takeaway. I have 2 more to add to the reasons why Lexica works at the time. One is lower latency is all you need. So in other words, instead of waiting a minute for your image, you could just search and find stuff that other people have done. That's good. And then 2 is everyone knew how to search already, but people didn't know how to prompt. So you were the bridge.

That's true. Yeah. You would get a lot better looking images by typing a one-word prompt versus prompting for that one word.

Yeah.

Yeah, that is interesting.

The numbers kind of speak for themselves, right? Like 24 hours post-launch, 51,000 queries, like 2.2 terabytes in bandwidth. Going back to the bandwidth problem that you had before, like you would've definitely run into that. Day 2, you doubled that. It's like 111,000 queries, 4.5 terabytes in bandwidth, 22 million images served. It's pretty crazy.

Yeah, I think we're, we're like doing like over 5 billion images served per month now. It's like, yeah, that's, it's pretty crazy how much things have changed since then.

Yeah, I'm still showing people like today, even today, now it's been a few months now. This is where you start to learn image prompting because they don't know.

Yeah, it is interesting. And I, it's weird because I didn't really think it would be a company. I thought it would just be like a cool utility or like a cool tool that I would use for myself. And I really was just building it for myself just because I didn't want to use the Discord search bar. But yeah, it was interesting that a lot of other people found it pretty useful as well.

So there's a lot of things that you release in a short amount of time. The God Mode search was kind of like obviously the first thing, I guess, like maybe to talk about some of the underlying technology you're using, CLIP to kind of find, you know, go from image to like description and then let people search it. Maybe talk a little bit about What it takes to actually make the search magic happen? Yeah.

So the original search was just using Postgres's full text search, and it would only search the text contents of the prompt. But I was inspired by another website called Same Energy, where it's like a visual search engine. Uh, it's really cool.

Do you know what happened to that guy?

Uh, I don't.

Um, he released it and then he disappeared from the internet.

I don't know what happened to him, but I'm sure he's working on something really cool. He also worked on like Tabnine, which was like the very first version of Copilot, or like even before Copilot was Copilot. But yeah, inspired by that, I thought like being able to search images their semantics, the contents of the image was really interesting. So I pretty much decided to create a search index on the CLIP embeddings, the CLIP image embeddings of all the images. And when you would search it, we would just do like KNN search on pretty much the image embedding index. And we had way too many embeddings to store in like a regular database. So we had to end up using Faiss, which is a Facebook library for like really fast KNN search and embedding search. That was pretty fun to set up. It actually runs only on CPUs, which is really cool. It's super efficient. You compute the embeddings on GPUs, but you can serve it all on an 8-core server and it's really, really fast. Once we released the semantic search on the CLIP embeddings, people were using the search way more. And you could do other cool things. You could do similar image search where if you found a specific image you liked, you could upload it and it would show you relevant images as well.

And then right after that, you raised your seed round from AI Grant, Nat Friedman, Dan Gross.

Yeah, we raised about $5 million from Daniel Gross, and then we also participated in AI Grant. That was pretty cool. That was kind of the inflection point. Not much before that point, Lexic was kind of still a side project. And I told myself that I'd focus on it full-time, or I'd consider focusing on it full-time if we had broke like a million users. I was like, oh, that's gonna be like years away for sure. And then we ended up doing that in like the first week and a half. Um, I was like, okay, there's something here. And it was kind of that like DBuild was growing pretty slowly and pretty linearly. And then Lexica was just like this thing that just kept going up and up and up. And I was so confused. I was like, man, people really like looking at pictures. This is crazy. Yeah. And then we decided to pivot the entire company and just focus on Lexica full-time at that point. And then we raised our seed round.

Yeah. So one thing that you casually dropped, I don't want to let slip. You said you were working on Lexica before the launch of Stable Diffusion, such that you were able to launch Lexica one day after Stable Diffusion.

Yeah.

How did you get so early into Stable Diffusion? Because I didn't hear about it before.

Oh, that's a good question. Where did I first hear about Stable Diffusion? I'm not entirely sure. It must have been somewhere on Twitter or something.

That changed your life. Yeah.

Yeah, it was great. And I got into the Discord because I'd used DALL-E 2 before, but there were a lot of restrictions in place where you couldn't generate human faces at the time. You can do that now, but when I first got access to it, you couldn't do any faces. It was like, There were like, the list of adjectives you couldn't use was quite long. Like I had a friend from Pakistan and it couldn't generate anything with the word Pakistan in it for some reason. But Stable Diffusion was like kind of the exact opposite where there were like very, very few rules. So that was really, really fun and interesting, especially seeing the chaos of like a bunch of other people also using it right in front of you. That was just so much fun. And I just like wanted to do something with it. And I thought it was Honestly, really fun.

Well, I was just trying to get tips on how to be early on things because you're pretty consistently early to things, right? You were Stadia before Stadia, and then obviously you were on—

Well, Stadia is kind of shut down now, so I don't know if being early to that was a good one.

I think just being consistently early to things that have a lot of potential, one of them is going to work out, and that's how you got Lexica.

How did you decide to go from search to running your own models? That's a good question.

So we kind of realized that the way people were using Lexica was they would have Lexica open in one tab, and then in another tab they'd have like a Stable Diffusion interface. It would be like either a Discord or like a local run interface, like the Automatic Gradio UI, um, or something else. I just, I would watch people use it and they would like alt-tab back and forth between Lexica and their other UI. And they would like scroll through Lexica, click on the prompt, or click on an image, copy the prompt, and then paste it. And like maybe change a word or two. And I was like, this should really kind of just be all within Lexica. Like, it'd be so cool if you could just like click a button in Lexica and get an editor and generate your images. And I found myself like also doing the Alt+Tab thing where it was like really frustrating. I was like, man, this is like kind of tedious. Like, I really wish it was much simpler. So we just built generations directly within Lexica. Um, so we do, we deployed it on I don't remember when we first launched. I think it was November or December. And yeah, people love generating directly within it now.

I was also thinking that this was coming out of AI Grants where I think AI Grants was a very special program. I was just wondering if you learned anything from that special week where everyone was in town.

Yeah, that was a great week. I loved AI Grants.

Yeah, bring us in a little bit because you weren't there. Oh, sure.

Yeah, it's really, really cool. All the founders in AI Grants are fantastic people. So I think the main takeaway from AI Grants was You have this massive overhang in compute or in capabilities in terms of like these latest AI models. But to the average person, there's really not that many products that are that cool or useful to them. Like the latest one that has hit the zeitgeist was ChatGPT, which used like arguably the same GPT-3 model, but like RLHF'd, but you could have arguably built like a decent ChatGPT product just using the original GPT-3 model. But no one really did it. Now, there were some restrictions in place, and OpenAI slowly released them over the few months or years after they released the original API. But the core premise behind AI Grant is there are way more capabilities than there are products. So focus on building really compelling products and get people to use them, and focus less on things like hitting state-of-the-art on evals and more on getting users to use something.

Make something people want.

Exactly.

Yeah, we did an episode on LLM benchmarks, and we kind of talked about how the benchmarks kind of constrain what people work on, because if your model is not going to do well on the well-known benchmarks, it's not going to get as much interest and funding. So going at it from a product lens is cool.

My hypothesis when I was seeing the sequence of events for AI Grants and then for Lexica Aperture was that you had some kind of magical dinner with Emad and David Holtz, and then they taught you the secrets of training your own model. I don't know. Is that how it happens?

No, there's no secret dinner. The Illuminati of text-to-image. We did not have a meeting. I mean, even if we did, I wouldn't tell you. But it really boils down to just having good data. If you think about diffusion models, really the only thing they do is learn a distribution of data. So if you have high-quality data, it'll learn that high-quality distribution. Or if you have low-quality data, it will learn to generate images that look like they're from that distribution. So really, it boils down to the data and the amount of data you have and the quality of that data, which means a lot of the work in training high-quality models, at least diffusion models, is not really in the model architecture, but rather just filtering the data in a way that makes sense. So for Lexica, we do a lot of aesthetic scoring on images, and we use the rankings we get from our website because we get tens of millions of people visiting it every month. So we can capture a lot of ranking. Oh, this person liked this image when they saw this one right next to it, therefore they probably preferred this one over that. You can do pairwise ranking to rank images and then compute like Elo scores. You can also just train aesthetic models to learn to classify a model whether or not someone will like it or whether or not it's like, or rank it on a scale of like 1 to 10, for example. So we mostly use a lot of the traffic we get from Lexica and use that to filter our datasets and use that to train better aesthetic models.

You had been a machine learning engineer before. You've been more of an infrastructure guy. With Dabuild, you were more of a prompt engineer with a bit of web design. This was the first time that you were basically training your own model. What was the ramp-up like? Not to give away any secret sauce, but I think a lot of people who are traditional software engineers are feeling a lot of, I don't know, fear when encountering these kinds of domains.

Yeah, I think it makes a lot of sense. And to be fair, I didn't have much experience training massive models at this scale before I did it. A lot of the times it's really just like, in the same way when you're first learning to program, you would just take the problem you are having, Google it, and go through the Stack Overflow posts, and then you figure it out. But ultimately you will get to the answer. It might take you a lot longer than someone who's experienced, but I think there are enough resources out there where it's possible to learn how to do these things. Either just reading through GitHub issues for relevant models.

Oh, God.

Yeah, it's really just like, you might be slower, but it's definitely still possible. And there are really great courses out there, like the fast.ai course is fantastic. There's the Deep Learning book, which is great for fundamentals. And then Andrej Karpathy's online courses are also excellent, especially for language modeling. You might be a bit slower for the first few months, but ultimately, I think if you have the programming skills, you'll catch up pretty quickly. It's not like this magical dark science that only 3 people in the world know how to do well. Probably was like 10 years ago, but now it's becoming much more open. You have open source collectives like EleutherAI and LAION where they share the details of their large-scale training runs. So you can learn from a lot of those people.

Yeah. I think what is different for programmers is having to estimate significant costs upfront before they hit run, because it's not a thing that you normally consider when you're coding. Burning through your credits is a fear that people have.

Yeah, that does make sense. In that case, fine-tuning larger models gets you really, really far. Even using things like lower-rank adaptation to fine-tune, where you can fine-tune much more efficiently on a single GPU. Yeah, I think people are underestimating how far you can really get just using open-source models. I mean, before Lexica, I was working on DeBuild and we were using the GPT-3 API, but I was also really impressed at how well you could get open source models to run by just using the API, collecting enough samples from real-world user feedback or real-world user data using your product, and then just fine-tuning the smaller open source models on those examples. And now you have a model that's pretty much state of the art for your specific domain, whereas the runtime cost is 10 times or even 100 times cheaper than using an API.

And was that like GPT-J, or are you talking BERT?

I remember we tried GPT-J, but I think Flan T5 was the best model we were able to for that use case. PlantyFive is awesome if your prompt is small enough, it's pretty great. And I'm sure there are much better open source models now, like Vicuña, which is the GPT-4 variant of LLaMA fine-tuned on GPT-4 outputs. Yeah, they're just going to get better and they're going to get better much, much faster.

Yeah. We were just talking in a previous episode to the creator of DALL-E, Mike Conover, which is actually commercially usable instead of Vicuña, which is a research project.

Oh, wow. Yeah, that's pretty cool.

I know you mentioned being early. Obviously agents are one of the hot things. In 2021, you had this please buy me AirPods demo that you tweeted with the GPT-3 API. Obviously one of the things about being early in this space, you can only do one thing at a time, right? And you had one tweet recently where you said you hope that that demo would have opened Pandora's box for a bunch of weird GPT agents, but all we got were docs powered by GPT. Can you maybe talk a little bit about things that you wish you would see? Or in the last few weeks we've had HuggingGPT, BabyAGI, AutoGPT, all these different kind of like agent projects that maybe now are getting closer to the, what did you say, 50% of internet traffic being SchizoGPT agents? What are you most excited about, about these projects and what's coming?

Yeah, so we wanted a way for users to be able to paste in a link for the documentation page for a specific API and then describe how to call that API. And then the way we would need to pretty much do that for Debuild was we wondered if we could get an agent to like browse the docs page, read through it, summarize it, and then maybe even like do things like create an API key and register it for that user. To do that, we needed a way for the agent to read the webpage and interact with it. So I spent about a day working on that demo where it would just take in the webpage, serialize it into like a more compact form that fit within the 2048 token limit of like GPT-3 at the time. And then just decide what action to do. And then it would, if the page was too long, it would break it down into chunks. And then you would have like a subprompt decide on which chunk had the best action. And then at the top node, you would just pretty much take that action and then run it in a loop. It was really, really expensive. I think like that one 60-second demo cost like $100 or something, but it was like wildly impractical, but you could clearly see that agents were going to be a thing, especially ones that could read and write and take actions on the internet. It was just prohibitively expensive at the time, and the context limit was way too small. But yeah, I think it seems like a lot of people are taking it more seriously now, mostly because GPT-4 is way more capable. The context limit's 4 times larger at 8,000 tokens, soon 32,000. And I think the only problem that's left to solve is finding a really good representation for a webpage that allows it to be consumed by a text-only model. So some examples are you could just take all the text and pass it in, but that's probably too long. You could take all the interactive-only elements like buttons and inputs, but then you miss a lot of the relevant context. There are some interesting examples which I really like, is you could run the webpage or you could run the browser in a terminal-based browser. So there are some browsers that run in your terminal which serialize everything into text. And what you can do is just take that frame from that terminal-based browser and pass that directly to the model. And it's a really, really good representation of the webpage. Because they do things where for graphical elements, they kind of render it using ASCII blocks, but for text, they render it as actual text. So you could just remove all the weird graphical elements, just keep all the text, and that works surprisingly well. And then there are other problems to solve, which is how do you get the model to take an action? So for example, if you have a booking page and there's a calendar and there are 30 days on the calendar, how do you get it to specify which button to press? It could say 30, And you can like match string-based and like find the 30. But for example, what if it's like a list of friends in Facebook and you're trying to delete a friend? There might be like 30 delete buttons. How do you specify which one to click on? The model might say like, oh, click on the one for like Mark, but then you'd have to figure out the delete button in relation to Mark. And there, there's some ways to solve this. One is there's a cool Chrome extension called Vimium, which lets you uh, use Vim in your Chrome browser. And what you do is you can press F and over every interactive element, it gives you like a character or 2 characters, where if you type those 2 characters, it presses that button or it opens or focuses on that input. So you could combine a lot of these ideas and then get like a really good representation of the web browser in text, and then also give the model a really, really good way to control the browser as well. Um, and I think those 2 are the core part of the problem. The reasoning ability is definitely there. If a model can score in the top 10% on the bar, it can definitely browse a web page. It's really just how do you represent text to the model and how do you get the model to perform actions back on the web page? It really is just an engineering problem.

I have one doubt which I'd love your thoughts on. How do you get the model to pause when it doesn't have enough information and ask you for additional information because you underspecified your original request?

This is interesting. I think the only way to do this is to have a corpus where your training data is like these sessions of agents browsing the web, and you have to pretty much figure out where are the ones that went wrong, or the agents that went wrong, where did they go wrong, and just replace it with, hey, I need some help. And then if you were to fine-tune a larger model on that dataset, you would pretty much get them to say, hey, I need help, on the instances where they didn't know what to do next. Or if you're using a closed-source model like GPT-4, you could probably tell it, if you're uncertain about what to do next, ask the user for help, and it probably would be pretty good at that.

I've had to write a lot of integration tests in my engineering days and like the DOM—

They might be over.

Yeah, I hope so.

I hope so. I don't want to—

I don't want to do that anymore. Yeah, I don't want to write them the old way.

Yeah, but I'm just thinking like, you know, we had the robots.txt for like crawler. Like I can definitely see the DOM being reshaped a little bit in terms of like accessibility. Like sometimes you have to write XPaths that are like so long just to get to a button. But there should be a better way to do it. And maybe this will drive the change, making it easier for these models to interact with your website.

There is the Chrome Accessibility Tree, which is used by screen readers, but a lot of times it's missing a lot of useful information. But in a perfect world, everything would be perfectly annotated for screen readers and we could just use that. But that's not the case.

GPT-4 multimodal. Has your buddy Greg hooked you up with that? And do you think that that would solve essentially browser agents or desktop agents?

Greg has not come through yet, unfortunately. It would make things a lot easier, especially for graphical-heavy webpages. So for example, if you were using Yelp and using the map view, it would make a lot of sense to use something like that versus a text-based input where, how do you serialize a map into text? It's kind of hard to do that. So for more complex webpages, that would make it a lot easier. You get a lot more context to the model. And it seems like that multimodal input is very dense in the sense that it can read text and it can read it really, really well. So you could probably give it a PDF and it would be able to extract all the text and summarize it. So if it can do that, it could probably do anything on any webpage.

Yeah. And given that you've some experience integrating CLIP with language models, how would you describe how different GPT-4 is compared to that stuff?

Yeah, CLIP is entirely different in the sense that it's really just good at putting images and text into the same latent space. And really the only thing that's useful for is similarity and clustering.

Literally same energy, right? Yeah. And then there's Blip and Blip 2. I don't know if you've messed with those.

Yeah, Blip 2 is a lot better. There's actually a new project called, I think, MiniGPT-4.

Yes, it was just out today.

Oh, nice. Yeah, it's really cool. It's actually really good. I think that one is based on the Llama model. But yeah, that's like another—

It's Blip plus Llama, right? So they're like, they run it through Blip and then have Llama ask your— interpret your questions so that you do visual QA.

Oh, that's cool. That's really clever. Yeah, ensemble of models are really useful.

Well, so I was trying to articulate because there's 2 things people are talking about today. The moment you wake up, you open Hacker News and go like, all right, what's the new thing today? One is Red Pajama and then the other one is MiniGPT-4. So I was trying to articulate why is this not GPT-4? What is missing? And my only conclusion was it just doesn't do OCR yet. But I wonder if there's anything core to this concept of multimodality that you have to train these things together? What does one model doing all these things do that is separate from an ensemble of models that you just duct tape together?

It's a good question. This is pretty related to interpretability. How do we understand that? Why do models trained on different modalities within the same model perform better than 2 models trained separately? I can kind of see why that is the case. It's kind of hard to articulate, but when you have 2 different models, you get the reasoning abilities of a language model, but also the text or the vision understanding of something like CLIP, whereas CLIP clearly lacks the reasoning abilities. But if you could somehow just put them both in the same model, you get the best of both worlds. There were even cases where I think the vision version of GPT-4 scored higher on some tests than the text-only version. So there might even be some additional learning from images as well.

Oh yeah. Well, the easy answer for that was there was some chart in the test that was interesting.

Interesting.

I was like, oh, when I read that, I was like, oh yeah, okay, that makes sense. Oh, okay.

That makes sense. I thought it'd just be like it sees more of the world, therefore it has more tokens in it.

So my equivalent of this is I think it's a well-known fact that adding code to a language model training corpus increases its ability to do language. Not just with code. So the diversity of datasets that represents some kind of internal logic, and code is obviously very internally logically consistent, helps the language model learn some internal structure. My ultimate test for GPT-4 is show the image of, is this a pipe? And then ask it if it's a pipe or not and see what it does.

Interesting. That is pretty cool. Or just give it a screenshot of your VS Code editor and like ask it to fix the bug. Yeah, that'd be pretty wild if it could do that.

That would be adult AGI. That would be, that would be the grown-up form of AGI.

On your website, you have this startup manual where you give a bunch of advice.

Oh, this is fun.

One of them was you should be shipping to production like every 2 days, every other day. This seems like a great time to do it because things change every other day. But maybe, yeah, tell some of our listeners a little bit more about how you got to some of these heuristics. And you obviously built different projects and you iterated on a lot of things.

Yeah. Do you want to reference this?

Sure.

Yeah, I'll take a look at it.

And we'll put this in the show notes. But I just wanted you to have the opportunity to riff on this list because I think it's a very good list. And which one of them helped you for Lexica, if there's anything interesting?

So this list is pretty funny. It's mostly just like me yelling at myself based on all the mistakes I've made in the past and me trying to not make them again. Yeah. So the first one is like, I think the most important one is like, try to, when you're building a product, try to build the smallest possible version. And I mean, for Lexica, it was literally a, literally one screen in a React app with a Postgres database and it just showed you like images. And I don't even know if the first version had search. I like, I think it did, but I'm not sure. Like, I think it was really just like a grid of images that were randomized. But yeah, don't build the absolute smallest thing that can be considered a useful application and ship it. For Lexica, that was, it helps me write better prompts. It's pretty useful. It's not that useful, but it's good enough. Don't fall into the trap of intellectual indulgence with over-engineering. I think that's a pretty important one for myself and also anyone working on new things. There's oftentimes you fall into the trap of thinking you need to add more and more things when in reality, the moment it's useful, you should probably get in the hands of your users and they'll set the roadmap for you. I know this has been said millions of times prior, but just, I think it's really, really important. And I think if I'd spent like 2 months working on Lexica, adding a bunch of features, it wouldn't have been anywhere as popular as it was if I had just released the really, really boiled-down version alongside the Stable Diffusion release. Yeah, and then there are a few more, like product development doesn't start until you launch. Think of your initial product as a means to get your users to talk to you. It's also related to the first point where you really just want people using something as quickly as you can get that to happen. And then a few more are pretty interesting. Create a product people love before you focus on growth. If your users are spontaneously telling other people to use your product, then you've built something people love. So this is pretty—

It sounds like you've internalized Paul Graham's stuff a lot.

Yeah.

Because I think he said, he said stuff like that.

A lot of these are just probably me taking notes from like books I found really interesting or like PG essays that were really like relevant at the time and then just trying to not forget them. I should probably read this list again. There's like some pretty personalized advice for me in here.

Oh yeah.

One of my favorite ones is, um, don't worry if what you're building doesn't sound like a business. Nobody thought Facebook would be a $500 billion company. It's easy to come up with a business model once you've made something people want. You can even make pretty web forms and turn that into a 200-person company. And then if you click the link, it's to LinkedIn for Typeform, which is now, uh, I think they're like an 800-person company or something like that. So they've grown quite a bit. There you go. Yeah, pretty web forms are a pretty good business, even though it doesn't sound like it.

Yeah, it's worth $1 billion.

One way I would like to tie that to the history of Lexica, which we didn't go over, which was just walk us through Aperture V1, V2, V3, which you just released last week, and how maybe some of those principles helped you in that journey.

Yeah, so V1 was us trying to create a very photorealistic version of our model. of Stable Diffusion. V1 actually didn't turn out to be that popular. It turns out people loved not generating—

Your marketing tweets were popular.

They were quite popular. So I think at the time, you couldn't get Stable Diffusion to generate photorealistic images that were consistent with your prompt that well. It was more so like you were sampling from this distribution of images, and you could slightly pick where you sampled from using your prompt. This was mostly just because the CLIP text encoder is not the best text encoder. If you use a real language model like T5, you get much better results. Like the T5 XXL model is like 100 times larger than the CLIP text encoder for Stable Diffusion 1.5. So you could kind of steer it into like the general direction, but for more complex prompts, it just didn't work. So a lot of our users actually complained that they preferred 1.5, the Stable Diffusion 1.5 model over the Aperture model. And it was just because a lot of people were using it to create like art and like really weird abstract looking pictures that Didn't really work well with the photorealistic model trained solely on images. And then for V2, we kind of took that into consideration and then just trained it more on a lot of the art images on Lexica. So we took a lot of images that were on Lexica that were art, used that to train aesthetic models that ranked art really well, and then filtered larger datasets to train V2. And then V3 is kind of just like an improved version of that with much more data. I'm really glad we didn't spend too much time on V1. I think we spent about 1 month working on it, which is a lot of time, but a lot of the things we learned were useful for training future versions.

How do you version them? Where do you decide, okay, this is v2, this is v3?

The versions are kind of weird where you can't really use semantic versions because if you have a small update, you usually just make that v2. Versions are kind of used for different base models, I'd say. So if each of the versions were a different base model, But we've done fine-tunes of the same version and then just release an update without incrementing the version. But I think when there's a clear change between running the same prompt on a model and you get a different image, that should probably be a different version.

So the startup manual was the more, you can actually do these things today to make it better. And then you have a whole future page that has tips from what the Siri successor is going to be like to Why everyone's genome should be sequenced. There's a lot of cool stuff in there. Why we need to develop stimulants with shorter half-lives so that we can sleep better. Maybe talk a bit about when you're a founder, you need to be focused, right? So sometimes there's a lot of things you cannot be building, and I feel like this page is a bit of a collection of these.

Yeah.

Are there any of these things that you're like, if I were not building Lexica today, this is a very interesting thing?

Oh man. Yeah. There's a ton of things that I want to build. I mean, off the top of my head, the most exciting one would be like better tools for language models. And I mean, not tools that help us use language models, but rather tools for the language models themselves. So things like giving them access to browsers, giving them access to like payments and credit cards, giving them access to like—

Credit cards. Geez.

Giving them things like access to like real-world robots. So like, it'd be cool if you could have like a Boston Dynamics Spot powered by like a language model reasoning module, and you could like do things for you, like go and pick up your order. stuff like that, uh, entirely autonomously given like high-level commands. That'd be like number one thing if I wasn't working on Lexica. And then there's some other interesting things with like genomics I find really cool. Like there's some pretty cool things you can do with consumer genomics. So you can export your, uh, genome from 23andMe as a text file, like literally a text file of your entire genome. And there is another tool called, uh, Prometheus, uh, Prometheus, I think. Prometheus. Where you upload your 23andMe text file genome, and then they kind of map specific SNPs that you have in your genome to studies that have been done on those SNPs, SNPs. And it tells you really, really useful things about yourself. Like, for example, I have the SNP that, uh, for this thing called, uh, delayed sleep phase disorder, which makes me go to sleep about 3 hours later than the general population. So like, I used to always be a night owl and I never knew why. But after using Prometheus, and it pretty much tells you, oh, you have this specific genome for, or specific SNP for DSPS, it's like in a really tiny percentage of the population. And it's like something you should probably know about. And there's a bunch of other things. It tells you your likelihood for getting certain diseases, for certain cancers, oftentimes like even weird personality traits. There's one for like, I have one of the SNPs for like increased risk-taking and optimism, which is pretty weird. That's an actual thing. Um, like, I don't know how, how—

This is the founder gene. You should sequence everybody.

It's pretty cool. And it's like the, it's like $10 for Prometheus and like $70 for 23andMe. And it like explains to you how your body works and like the things that are different from you or different from the general population.

Wow.

Highly recommend everyone do it. Um, like if you're, if you're concerned about privacy, Just like purchase the 23andMe kit with like a fake name. You don't have to use your real name. I didn't use my real name.

It's just my genes. The worst you can do is clone me. It ties in with what you're talking about with, you know, we want the future to be like this and like people are building uninspired B2B SaaS apps. And you and I had an exchange about this. How can we get more people to believe they can do great things?

That's a good question. And I like a lot of the things I've been working on with GPT-3 has been trying to solve this, is get people to think about more interesting ideas.

Yeah.

I don't really know. I think one is just, the low effort version of this is just put out really compelling demos and get people inspired. And then the higher effort version is actually build the products yourself and get people to realize this is even possible in the first place. I think the Baby AGI project and the GPT agent projects on GitHub are practice today, they're not super useful, but I think they're doing an excellent job of getting people incredibly inspired for what can be possible with language models as agents. And also the Stanford paper where they had like the mini version of Sims. Yeah, that one was incredible. That was awesome.

It was adorable. Did you see the part where they invented day drinking?

Oh, they did? Yeah.

They were like, you're not supposed to go to these bars in the afternoon, but they were like, we want to go anyway.

Nice. That's awesome. Yeah, I think we need more stuff like that. That one paper is probably going to inspire a whole bunch of teams to work on stuff similar to that. And that's great.

I can't wait for NPCs to actually be something that you talk to in a game and have their own lives and you can check in and they would have their own personalities as well.

Yeah, I was so kind of off topic, but I was playing The Last of Us Part 2 and the NPCs in that game are really, really good where if you point a gun at them, they'll beg for their life and like, please, I have a family. And when you kill people in the game, they're like, oh my God, you shot Alice. They're just NPCs, but they like refer to each other by their names and like they plead for their lives. And this is just using like regular conditional rules on NPC behavior. Like imagine how much better it'd be if it was like a small GPT-4 agent running in every NPC and they had like agency to make decisions and like plead for their lives. And I don't know, you feel way more guilty playing that game.

I'm scared it's going to be too good. I played a lot of our show Fallout. So I feel like if the NPCs were a lot better, you would spend a lot more time playing the game.

Yeah.

Let's jump into lightning round. First question is your favorite AI product.

Favorite AI product. The one I use the most is probably ChatGPT. The one I'm most excited about is, it's actually a company in AI Grants. They're working on a version of VS Code that's entirely AI powered where Cursor. Yeah, cursor where you could like give it a prompt and like iterate on your code, not by writing code, but rather by just like describing the changes you want to make. And it's tightly integrated into the editor itself. So it's not just another plugin.

Would you, as a founder of a low-code prompting-to-code company that pivoted, would you advise them to explore some things or stay away from some things? Like, what's your learning there that you would give to them?

I would focus on like one specific type of code. So if I'm building a low-code tool, I would try to not focus too much on appealing to Whereas if I was building an alternative to VS Code, I would focus solely on developers. Um, so in that, I think they're doing a pretty good job focusing on developers.

Are you using Cursor right now?

I've used it a bit. I haven't converted fully, but I'm, I really want to.

Okay.

It's getting better really, really fast.

Yeah.

Um, I, I can see myself switching over sometime this year if they continue improving it.

Hot tip for, for ChatGPT. People always say, you know, they love ChatGPT. Biggest upgrade to my life right now is the iForza menu bar app I found on GitHub. And now I just have it running in a menu bar app and I just do Command+Shift+G and it pops it up as like a single use thing. And there's no latency because it just always is live and I just type, type in the thing I want and then it just goes away after I'm done.

Wow, that's cool.

Big upgrade.

I'm going to install that.

That's cool. Second question. What is something you thought would take much longer, but it's already here? Like what, what's your acceleration update?

Ooh. Um, it would take much longer, but it's already here.

This is your question.

Yeah, I know. I wasn't prepared. Um, So I think it would probably be kind of, I would say, text-to-video. Yeah.

What's going on with that?

I think within this year, uh, by the end of this year, we'll have like the jump between like the original DALL-E 1 to like something like Midjourney. Like we're going to see that leap in text-to-video within the span of this year. Um, it's not already here yet. So I guess the thing that surprised me the most was probably like the multimodality of GPT-4. And the fact that it can technically see things, which is pretty insane.

Is text-to-video something that Aperture would be interested in?

It's something we're thinking about, but it's still pretty early.

There was one project with hand animation with human poses. It was also coming out of Facebook. I thought that was a very nice way to accomplish text-to-video while having a high degree of control. I forget the name of that project. I think it was Draw Anything.

Yeah, sounds familiar. Well, you already answered the, a year from now, what will people be most surprised by? And maybe the usual request for startup, what's one thing you will pay for if someone built it?

Ooh, one thing I would pay for if someone built it. So many things, honestly. I would probably really like, I really want people to build more tools for language models, like useful tools, give them access to Chrome, And I want to be able to give it a task and then just, it goes off and like spins up 100 agents that perform that task. And like, sure, like 80 of them might fail, but like 20 of them might kind of succeed. That's all you really need. And they're agents, you can spin up thousands of them. It doesn't really matter. Like the law of large numbers is on your side. So that'd be, I would pay a lot of money for that, even if it was capable of only doing really basic tasks, like, like signing up for a SaaS tool and like booking a call or something. If you could do even more things where it could handle the email thread and get the person on the other end to do something where I don't even have to book the demo, they just give me access to it, that'd be great. Yeah, more really weird language model tools would be really fun.

Are ChatGPT plugins a step in the right direction or are you envisioning something else?

I think ChatGPT plugins are great, but they seem to only have read-only access right now. I also want them to have— I want these theoretical agents to have write access to the world too. So they should be able to like perform actions on web browsers, have their own like email inbox, have their own credit card with their own balance, like take it, send emails to people that might be useful in achieving their goal, ask them for help, be able to like sign up and register for accounts on tools and services, and be able to like use graphical user interfaces really, really well. And then also like phone home if they need help.

You just want virtual employees. You want to give them a Brex card, right?

I wouldn't be surprised if a year from now there was BrexGPT, where it's like Brex cards for your GPT agents.

I mean, okay, I'm excited by this. I kind of want to build it.

You should.

Yeah.

Well, just to wrap up, we always have one big takeaway for people to display on a signboard for everyone to see. What is the big message to everybody?

Yeah, I think the big message to everybody is you might think that a lot of the times the ideas you have have already been done by someone, and that may be the case, but a lot of the times the ideas you have are actually pretty unique and no one's ever tried them before. So if you have weird and interesting ideas, you should actually go out and just do them and make the thing and then share that with the world. Um, 'cause I feel like we need more people building weird ideas and less people building like Better GPT search for your documentation.

There are like 10 of those in the recent OSU patch. Well, thank you so much. You've been hugely inspiring and excited to see where Lexica goes next.

Appreciate it. Thanks for having me.

Bye, guys. Bye.