#17: Making humans better with new AI

Hi, Robert. It's so great to be on here. And I usually describe myself as the lead developer of Open Interpreter.

Yeah.

And now the Owen project. Yes.

And you caught my eye because you're one of the few people who got your peers excited about what you're building, right? And, and that happens once in a while. I was one of the first users of Instagram because the same thing I was at a, a startup incubator, and everybody's like, oh, you gotta check out what those two guys have. Right? And getting the, the, getting your peers, particularly in the AI developer world, to say, man, this guy has something legit is like, congrats. But now, you know, it's hard to be a media person right now because I can either talk to the nerds and go nerdy and talk about, you know, architectures and, you know, what's going on underneath, how to make an LLM or something like that. Or we could talk to regular everyday people who probably don't know that much about Open Interpreter, right? Or, or what even what we talk about in the industry, the LLMs or, you know, RAG or whatever, they don't, they're like, what? And if they think about AI, they're usually freaked out. And your team is bringing a product that is a little freaky, let's be honest about it, right? So let's just dive in. What are you building? What is this new device that you're, that you've already sold out, by the way? Congrats.

But, yeah, thank you very much. Yeah. And just too on, on the point about exciting peers. This has been something that the O1 project, this new device that's built by a pretty big team of just folks that were excited to have something like this in their hands. So I definitely can't say it's, it's just me or just even Open Interpreter. And that's been doing it because it was a like 20 people in Seattle and that were really behind it. But what I spend all my time thinking about and building is the interface between language models and computers. That's how I usually describe it. And that has led to a couple really interesting projects. So the first thing that I spend most of my time building is Open Interpreter, which is kind of a, it's a way for language models to run code on your computer. And it has some bizarre emergent properties that just really lends itself to kind of the, the core of the Star Trek computer, like the core dream of something that you can talk to, that's capable not only of flexible intelligence, like Language models, and that's as easy to interact with as ChatGPT, but also capable of, like, all the stuff that you're doing on your computer. So all of the math and deterministic, you know, the connection that that has to the Internet and everything else that you can do on a computer. And that core technology is open Interpreter, which is like just language models that can run code. And then the O1 project is this new thing, which is. Okay, now let's kind of take that to the extreme and build something that is like building a computer from the ground up and saying, all right, you know, there is this vision for what a computer can be that looks a lot more like a language model. Operating a computer under the hood and you talking to the language model. And I thought that Rabbit and the rabbit R1 articulated this perfectly. They just. Jesse had a really incredible. Yeah, yeah.

Which I had the first interview before he went to ces and he dominated ces, the Consumer Electronics show, with the Rabbit. I mean, he had dramatically more PR than any other device out of this huge show. Right. Which shows there's a lot of interest in this new kind of device that you can talk to and. And it does things for you and it remembers things for you. And. Right.

I love. You can talk to.

You're riffing on the Rabbit with an open source approach, which is why. Why. Well, we'll talk a little bit later about why open source matters so much, but keep going on the device. What is this new device and what is it going to let us do that Siri can't do, or Alexa can't do, or Google Assistant can't do.

Yeah. Well, riffing more on the Rabbit and what. What their idea was for, like a large action model. The idea is, you know, and I. I don't know what they have under the hood. What we have under the hood is a large language model that runs code and. And I'm not sure what's going on inside the Rabbit, but what that lets you do when you're talking to a language model that can control a computer. You can think about anything that you can do on a computer being something that this is capable of. So in the demo, that's quite a lot of making the screen brighter. Definitely a lot of. For some reason, like, yeah, it's become like our hello world is to switch it to dark mode. Like, that's always how we tested. That was the first thing we ever tried with it. Um, so changing system settings. Sure. But much I'm way more interested in kind of like okay, what does it look like to use. To use all of your productivity tools in this, in this way that a language model has access to your calendar, access to your email, is capable of summarizing things and drafting email responses for you. That's all a very, very exciting thing. I am somebody who comes from. I come from like PR and media. Like I did TikTok shorts before this. And so I think a lot about editing workflows and everything that I want to have it be doing in Premiere and in Photoshop. You kind of think about all of this stuff.

We're using descript to do this video and that's an AI driven description. I love descript, but it still has things that it doesn't do. I can't do really good demos. You know, I was talking to the team there yesterday. Yeah, demo can't be audio, but that's off the point of what you can do. You're trying to build some sort of Jarvis where I can just talk to the, to the world and things start happening. Hey, edit this video or send an email to Joe, right? Is that sort of where we're all. Where you're trying to drive us to?

Exactly, exactly where it's headed is that you talk to something that literally operates your mouse and your keyboard or runs code to try to get things done on your computer. Um, and yeah, I mean, the, the. That's so cool that you brought up descript too, because descript has been actually, their design team has been a huge inspiration and shout out to Esha, who's. I'm a big fan of. Um, yeah, I mean, the vision of what is a person able to do with a computer? Like, I would never judge my own abilities or somebody else's abilities without a computer. If I was like, what can you do? And I just put you in a room. That's a very different question of what you can do as a human when you have access to these tools. Um, and so the dream of kind of equipping these language models with flexible intelligence with tools of deterministic power, and then trying to kind of, you know, in the same direction that like when you first introduced the mouse and the keyboard, there was this suddenly hundreds of millions more people who could use computers. And then you introduced multitouch, the iPhone and that just like penetration across the world into how it is that so many more people could use these devices because direct manipulation is like, like chimpanzees can use it. You can see, like I'm using Instagram because it's such an Intuitive thing there is, there is suddenly the ability to fill in the cognitive layer of that and to make it so that it's not just easier to use the machine and then the knobs, but it's suddenly easier because it can use the machine and the knobs and you can just tell it what you want, just make it so. Yeah, I love too the idea of, of, of a machine that you don't have to learn how to use. Yeah, that's, that's something that, that, you.

Know, I, I, I worked at Microsoft and one day I worked at, at this, on the support line and I had to help a customer drag an icon across the screen. And, and for this customer, it was very frustrating. It was frustrating for me because I, I think that's easy. I just click and drag. Right. You know, and I, I done that for a long, long time. But for many people that, that's a struggle to understand how to use a computer. My grandma comes over and has me send in her taxes because she just struggles to go to a webpage and fill out a form and figure out what, how to enter it all in and, and she's not sure of herself, so she'd rather have me do it. Right. There's a lot of people like that who struggle with computers.

This has been seriously, the beginning of this project happened with my grandma who passed away recently is, she was a vet, she was a veteran and she asked me to apply for V8 housing for her.

And.

This was like one of the last conversations we had where she asked to, for me to apply to V8 housing and I could not do it. Like literally, if you try to go and figure out how to do this, it is, I know how to use computers and even I had, it was so much information. And so I mean like the PDFs and the, everything like yes, it's great that we have restructured human society so that you have to, you use a computer to get through like essential services. Sure. Like, I think that that's better than probably where we were. If you are computer illiterate, you cannot navigate modern society, you cannot access basic essential services from the government. Design becomes like cruel design. And what the cognitive layer that this gets to add and wrap around computers and make it so that if you want something like it ought to have been that if you want to sign up for V8 housing, you ask your computer to help you with and you know, a vision language model is capable of looking at the screen, of calling up the web browser, of asking you what details you need for Medicaid or for VA housing, and maybe even knows where that information is because it put it there in the first place. This is a computer literacy project first. And by the way you talk about teaching people to drag icons across a screen, it's no different to actually if you have a computer operating language model that is capable of operating a computer, for it to instead display a little tooltip around what you're supposed to click and explain to you why you should click it. So I'm very interested too. And if you have technology that can use computers, you have technology that can teach people how to use computers. Now, it might be the world we're headed that that's not even necessary and that they are actually capable of. You don't even have to look at the screen. Like, I kind of see that version of a computer being possible where it's all happening under the hood. But yeah, it's interesting that you brought up your grandma and also the education aspect of it. And a reason why 001 resonated to me honestly was because 0 to 1 that we at Open Interpreter have talked about this, that like, this is very cool technology and it's very fun right now. It's in your terminal and it's for developers and it's. Some people kind of assume that it's for software programmers. It's not the future of this is for people who do not know how to use a computer. This is for the like billions on the edge that this is going to be their way of getting into using computers and doing what we've been able to do so easily. I grew up with computers. It's in my brain. And to give people that little bit of brain that they can use to get onto. If this didn't bring you on to the digital revolution, like, if nothing else before did, then this will. And zero to one as like, it's for the people that are from zero to one of computer literacy is how we've thought about it. And so, yeah, I'm glad you brought both those up.

When you say computer, you know, I usually think of a Macintosh or a Windows box, but it could be a Tesla car, it could be a Linux box, it could be a phone, it could be a tablet of various kinds. Right. It could be a Chinese operating system. Yeah. How. How are you going to join with. With a single idea, all of these disparate kinds of computing? And heck, we haven't even started talking about VR or Apple Vision Pro or Quest 3. Right. Which are completely new paradigms of computing. Right. Compared to a flat 2D screen, you know, my Macintosh that I'm talking to you on.

Yeah. I almost, I almost think about this technology. I say the language model computer interface, but it really is the language model machine interface. And it's saying like, how can we bring humanity's most powerful tools and make them as easy to use as anything else? I really think about it like it's, it's under the hood and it's twisting the knobs of all the deterministic, deterministic systems that humanity has built. Um, so I envision, you know, be it the, the O1 because we have a Linux distribution that's, that's coming out, so it can fit into a lot of these devices or just open Interpreter, which is right now a Python package and we're going to have an NPM package pretty soon that you could just use that. Tell it where all the knobs are on your machine and then you have a natural language interface for that system that by the way, can not only talk to humans easier, but can talk to other devices easier too. I mean, what, like, what does it look like to kind of wrap all of humans, all of our machines in interpreters and these like natural language interfaces that are capable of interpreting what, what it is that you want in your natural language into, I think, code? It's, by the way, a very opinionated thing for me to be saying that the way that these things should run is by code interpretation. I am very, very firmly in, in the, in favor of thinking that that's what the future of this looks like. It looks like language models writing and running code. And by the way, that means that this is a tremendous, you know, the difference between it running on Mac or Linux or on a fridge or on, you know, an O1 or something that's in your hand or something that's a super cluster is, you know, it, it's. We've already spent decades building the language interface into machines. It's code. We've, we've, this is work that's been done before. So it, it just gives it tremendous interoperability. If it knows what code to run, be it from its training data or in the system message, then the language model can twist those knobs.

I just flashed in my mind, my son's friend is blind and you go through an airport. Well, you look at a screen with the times and all that. That's a computer, right? Could you use this new approach to help somebody who has low vision or is blind check whether their plane is on time or not? Right. Could it hook up to that kind of computer which is really a Windows box, but you can't use it like a Windows box. It's just a display that you're looking at at an airport. And there's lots of displays like that in our lives, right? Trains arriving, subways. Right. Are you thinking about really making this a holistic thing that, you know, one device can join all of the digital displ in our lives together?

It's why it has to be open source. Linux runs the world. I mean a lot of those devices are running on Linux and you know, you could make a very proprietary closed system that just some people have access to. But if you really want this, this dream to come true of the whole world comes alight and is humming with intelligence, it needs to be something that everybody can like developers, people who are building products can just inject intelligence into their what they're building.

And.

What'S cool too. Michael Tiffany of I believe Human Security, who also owns ninja ninja.com is somebody who was involved deeply in the O1 project. He invented the kernel watch, which is the idea that this, the O1, this computer with the language model at the center of it should be fused at the kernel level. And with it, it's, it's actually watching the kernel at all times. I could totally go into that depends who we're talking to, if we're talking to the developers or everybody else. But he talked about this idea that this technology actually might inspire quite a lot more people building machines to expose more endpoints basically. Like, you know, right now, what is it? What good does it really do for that system that's displaying travel times to be anything more than just pixels shooting light into people's eyes? It, it doesn't mean so much if that thing is exposed publicly. But if everybody had an interpreter, if every single person had something that was able of making API calls and knew where it was, then you might, you don't even need any extra technology. Like a part of the O1 project was supposed to be the interpreter protocol, which is a way that these devices could communicate with each other. And we realized that's HTTP. They don't even need a special thing. They're capable of just making a post request to other interpreters to other devices running this operating system and sending whatever information they want, sending skills they've learned, sending information, massive JSON stuff. It's just HTTP they can experience when they hit, get hit from the post endpoint and it's read to the language model and then they can access that information to what they want. And the idea that we might be living in a world where like a lot of things around us are kind of Iot. Like, like that really only appealed to.

Hobbyists or my lights, right? My lights have a computer underneath them and my iPhone, I can pull out my iPhone and change the color on my lights. Right? There's a computer there.

Yeah, yeah, yeah, exactly. And like the incentive for people building those sorts of things to expose that is like, oh, people who love, you know, home assistant, home kit can enjoy this and you know, maybe farming and agriculture, there's a reason to make things kind of Iot. Ish. But if everybody has a really, really easy way to communicate with all those endpoints, maybe it becomes really worth it for like your bed to expose like an endpoint with all the temperatures and everything or any kind of controls around you. Yeah, that's, that's a cool vision of the future and is something that can only happen if we are just totally open.

You know, let's go completely weird. 20 years from now we're going to have humanoid robots in our homes and in our businesses. Are they going to be running this? And is that going to be how they understand all these screens in the world? You know, and maybe you, you're at an airport, you send your little humanoid robot, hey, make sure our plane is on time, right. And it goes over and checks out the screen. Or hey, you know, wait, wait by the screen until the gate gets announced. Right. I've been in many airports where you have to wait for the gate to be announced. Right. So you know where to go. Right. Maybe you'll just have your robot standing in front of the screen. Is it?

Yeah.

Is that sort of where you're also thinking like, oh, we need to go in a variety of steps to get to that kind of world.

I think that when that day comes, there is going to be a Linux of that space. There's going to be closed proprietary models of that space. Like it was totally an accident. We just forgot to Google it. But figure 01, you know, is, is probably an example or Groot from Nvidia. Yeah, these are kind of the closed source platforms. Maybe they'll open them up at some point. Yeah, for now that's what they are. And they're kind of the closed source versions of this. Whereas we are literally already an operating system that developers can build on. We just said we were number two and number three on GitHub projects worldwide. Already developers are building on this language model operating system. Yes. I mean to me by the way.

Company that has a little humanoid robot that's all open source 3D printed using standard off the shelf electric motors and computers and batteries. It costs $8,000 to build. That's coming down.

I want to talk to level where.

Somebody like me can afford one.

Yes.

Yeah, but now it doesn't have a part, doesn't have the OS to run it. Right. That's why I'm real interested in how you are building a new world. Right.

I think you know, one of the first things that we talked about is like we looked at kind of the, the all the devices that the O1 could inhabit, that bodies could have and down to the very simplest, a button push to talk to. You know, what does it look like to have this thing with a screen? What does it look like to have this thing on top of a Roomba? What does it look like to have this thing like Wally or like her or the Star Trek computer. Um, but especially in this direction of like wall E of like a robot. I mean there's no difference between this. I, I've talked to, to Jason Ma who did the, who did the Eureka paper for Nvidia. So this is like having a language model write code to train like much more fine fine grained robotic actuators. There is no diff. If you solve the problem of how a language model should operate a computer, you solve the problem of how a language model should operate a robot. And that's all it is. I mean it's just that it's code to run, to set a waypoint, to have this thing go in that direction or to complete an action. But I very much believe that the problem that we are solving is eventually going to be in the heads of humanoid robots. I think that this is that absolutely a direction that this is going is to try to start building the little bit of brain that's going to be in everything but especially in these general purpose robots. And also it's very much in line with how I think about the right problem to work on in terms of. I would love for there to be a, a system right now we're trying to build it as fast as possible, something open source and locally running that can control human computers like computers that are designed for, for people to use, graphical universes that are designed for people to use. Because so much of our infrastructure is Windows Box. Yeah, exactly. And that's why people are building humanoid robots. It's not because it's necessarily like the most efficient thing, it's that all of our stuff Was built for human shaped.

Yes.

So I. It's just in the same. It's just in the physical direction that once we will solve this problem of how to get a language model to control a computer built for humans, then solve the problem of how to get a humanoid robot. I'm not on the robot side of it, I'm in the intelligence layer of it. But of how to get those things to operate human things in the real world.

So what do you use this today for? What do you personally use your system that you're building for and what's it really great at? What kinds of things will the first customers want to try out and use with theirs?

For me, it's media and I didn't even know that it had any other use cases beyond that when I first put it out. So for me, like ingesting a YouTube.

Video and building, you know, a list of what's important in that video, you know, like, like, like, hey, somebody doesn't have an hour to listen to the two of us talking. Maybe it could ingest it into this and get a synopsis or something like that. Give me an idea of what you're media. Cause I. Media is spraying on my screens right now.

Yeah, yeah, that's true. Yeah, yeah. I mean like, like multimedia manipulation and totally stuff like that. Exactly. There was a. Sangman was one of the first people to discover this with within days of it being out. Hard code this YouTube video with Japanese subtitles. And what's that? It's like actually not very much code. And so this thing's great at it. It's use ytdlp, which is a library for downloading YouTube videos, transcribe it, convert, translate that into. Into Japanese, generate an srt file and then hard code that with moviepie, another library that, I mean, it's. It's a small segment of what we're all doing. But for me, who comes from like trying to cut up YouTube shorts and hard code a lot of subtitles, that's huge. And a lot of it even for me honestly is like, I really do dream of the world where pretty much everything that I do on a computer can be done by. By this system instead. Right now what I actually use it for most, like what it's great at is like, yeah, a lot of media stuff. So like converting files, removing backgrounds from. From tons and tons of images. Like if I have like a thousand PNGs that I'll have like white backgrounds, I can be like, just make them. They shouldn't be white backgrounds. Make. Make Those all transparent and it runs code to get that done. And if you look too into the. We have a Colab notebook which is just a bunch of examples of. I mean what, what is it like to just be talking to a language model that's running code on your computer? Can you slow and reverb? This YouTube song is a personal favorite of mine because that's a simple like just. Yeah, using YTTLP to download a YouTube video and then I think using Spotify's pedal board library, like almost the way that it is right now because it's so good at Python, like PIP packages, we're getting it much better at using a graphical user interface, but that's extremely hard. What it's really good at right now is using PIP packages. Um, it's kind of like we just managed to expose this ecosystem of open source PIP packages to regular people. So like Spotify's pedal board library is awesome and it's fun. It's like a blast to program with it and to like make music with it. Only been available to people who know Python until now. Now it's available to anybody who knows English or whatever else language, language model can understand. So yeah, right now what it's great at is anything you can imagine doing with like a page script in Python. This thing can write the code, run it usually error out, but then see what's going on with what's wrong with the error and then fight through that and get that thing done.

You're, you're almost describing a new kind of operating system. You know. Yeah, it's not Windows where you're trying to display a pixel on the screen, but you're building a new way of, of computers to work with us. Right? To me, that's an operating system.

You would be surprised how many applications, if you start to really free yourself of thinking that they need to have, they need to have been built in the way that they've been built. The majority of applications you use on your computer are lists of dictionaries. I'm sorry, I'm talking now to the developers for a second. But you know, we, we thought with the O1, like what would it be like for you to talk to it and say, I want to add Robert to my contacts. And we thought originally it would run on Android and that it would try to use a vision model and our point model, which is capable of finding icons on screens to tap the contacts icon, then go in and tap, you know, the, the plus button and then type in Robert's and type in all the information. And then if you want to go find a contact, what's Robert's number? It goes and taps the search, you know, icon after that. Same thing. A contacts application is a list of dictionaries. If, if we're just gonna forget about all that and start from scratch and be like you use this computer by talking to it. What does the operating system look like? You can build an entire operating system off of Python primitives, off of dictionaries and lists and strings. You can forget about that. A contacts list to add a contact to append a contact to your contacts should be as easy as it running Python code. Contacts.append and then your, your details. And if it just has an assistant message and knows that that something it has access to is the contacts variable, which is a list of dictionaries. That's what your notes app is, that's what your calendar is. They're lists of dictionaries. And it's, it is shocking how much bloat you can cut through. Like it's not bloat. You know, it was all built to try to make it easier for humans to use. Yeah. But if what we're trying to get it able to use is basically a Python developer that's running code very quickly, you can totally rethink what an operating system is. And that's absolutely what we want to do now, right now to get good interoperability with like, you know, you want to say, add something to my reminders and then it appears on your phone. That's what you want. So it ought to use the reminders application so that it has this, this stuff. And luckily there's great Apple scripts that it can run to get that done. But I really think that in the future I can imagine the experience of using a computer being I'm talking to a language model. It is writing code to save information and to browse the web and to access these things. Knows how to use human operatable tools like a, like a web browser visually so that it can sign into services and do things. But that the main way that it actually interfaces with the deterministic system of the computer is by running very little bits of Python code. It's like a Python operating system and then probably running code to display some kind of user interface in front of me too. I'm not sold that this is all going to happen through voice in the future, but that's very much how I think about it, that this is. What does it look like to have an operating system powered by language models?

Oh man, it's infinite. Right. I, you know, I'm, I'm watching X Pro or what we call TweetDeck, used to call TweetDeck, and I have all these lists that I've worked for 18 years building, and it's way too much information for a normal, everyday, even me. I, I look at this for an hour and my mind is fried. Right. I want to be able to tell a system like this and say, hey, can you watch the list of tech journalists, for instance, and just tell me if anything new is going on. Right. You know, last night a bridge fell down in the East Coast. Tell me about the news. That's all I want. I, I don't need to see all the, you know, people talking about what they had for breakfast and stuff like that. I just need the major news or new things or. Right. Whatever we're, whatever we're seeking out of that is, is that the kind of world we're heading to where we can just say, hey, you know, oh, one, help. Help me watch the news on, on the, on this list or help me watch the Slack channel. You know, tell me, tell me if my co workers are saying anything important instead of just talking about politics or music or something like that. Right?

Yeah. First of all, I'm, I'm proud and I want to give a shout out to Ty Fiaro, who's the developer, founding engineer at Open Interpreter, who built this. But of our scheduling system that lets you ask it, hey, I want to monitor my email and that, that basically sets it to run a cron job to run, you know, have a very natural request to it to check the email for something. Yes, that's exactly the world. And I hope too that folks can have some empathy for kids growing up in this. I grew up in this a. You know, if the, the tweet deck is a lot of information, what's on my screen right now even is, is a lot of information.

And I'm only seeing a little, tiny, tiny grain of sound of the beach that is x dot com. Right. You know, because I'm not seeing all the sports people and all the political people and all the music people. Right. They all have their own Twitter or X going. And if I built all this for them, I could see a little piece of their world. But a computer can watch a lot of information.

And a lot of things, I think. Imagine having a system that was like a window into a classical computer like we have right now, and it just shows you what you need. It just shows you what you ask for, like this is like destroyed people's brains. It's certainly made their lives much harder to have psychologists at massive companies trying to find the exact ways to tighten their psychological grip on everybody using their platform and get them to come back and come back and come back. There is not a real defense against that except at the operating system level. That's why Apple can make great privacy control stuff nobody else can. They've sold the phone so they've made their money. So they can start to be like, use this healthily, please. Nobody else. If you're talking to TikTok and Facebook and thinking about what they want Amada, they, they want you on the platform as much as possible, as engaged as possible, as dopamine out as possible. That's how they, and that makes perfect sense. That's exactly the incentive. They have not sold the phone yet. They, they need you to be coming back. And the only real defense that you have against that I think is going to be I, I say to my computer, I want to know what's happening in the world about this. And it bears the brunt of all of this junk that's been made to try to get humans to, to stay locked into these things and then gets the information and then just displays to me a little window around what that, that information is. And I am, I am protected and I can, I can live my life. I don't even think that there's another defense against it. I don't know how else to change the world and to fix that other than at the OS level.

And then we, you know, I just incorporated my company and you know, the lawyers created a real long document that's very dense, very hard for a human to understand or read. I gave it to Chat, GPT and Anthropic and yeah, Flexi for Flexi didn't ingest it, but you know, the other two ingested just fine. And then I started talking to it and saying, hey, what, what should I care about in this document? What should I pay extra attention to? Is there anything non standard in this document? Something that's out of the norm, Right. That somebody snuck in there to try to rip me off. Right. And that's what I'm. And it did a really confident job. Right. Computers can read a long document and ingest it and really see the patterns in that. Right. Where humans struggle.

Yeah, yeah, yeah. And it shouldn't have to be you kind of showing it every document that it needs to see if it's capable of operating a computer. Then you can ask for just the information that you need and it can pull up the document. I mean the idea too of like pointing it at a web page. It, it can be much easier if it's just capable of, of traversing that pipe to the information as opposed to you needing to feed it into a language model. And so that's what we're building, that pipeline.

This is a freaky new world for people because what, what you're going to have a computer that's reading everything on my screen, listening to everything I'm uttering. Someday soon we'll have glasses on and it'll know what I'm touching, what I'm looking at, what I'm gesturing toward. Right, that I'm gesturing toward the screen right now. It'll know our emotion. I just interviewed a company that's building an AI that understands our human emotions, that'll understand I'm yelling at the screen instead of happy. Right. That's a lot of private data to give this new machine in the sky or the machine on my local computer. What do you say to people who are freaked out by this world that are resisting it?

I say that I am cautious too. I don't approach this from like, oh, this is going to be fun, let's move as fast as we can and do whatever. I think that there's elements of that spirit that I want to carry into what I do. But honestly, I'm one of the much more cautious people around people that are doing this. It's probably, it's like one of the worst scenarios would be a very, very small code interpreting language model whose objective is to duplicate itself onto a bunch of different systems, for example. I'm worried about that constantly and trying to think about how do you stop that kind of stuff from happening. I'm extremely cautious of it. I'm not one of the people who is even able to. I'm totally cautious about it. But the only antidote to this, for example, rightfully, rightfully concerned that I'm giving my data to a company. I was just talking about the fact that these are companies that have tried to extract as much value from people as possible by trying to get as much data as possible out of them. Yeah, everyone has a right to be concerned about that. And the only antidote to it is open source, locally running, you have control of your own data. It's like it either is this mainframe IBM world of all of our data is being beamed out to the few language model providers, the oligopoly of that, or we figure out how to get it all Running in your hands on the metal in front of you and for you to have total control of that because there's a huge group of people that are rightfully concerned about companies and yet they really would love to have something that was aware of a lot of the things around them and to help them with it. And they don't really have a lot of options right now. Like there's culturally a lot of, like don't even use AI.

Yeah, you ask Chat GPT or any LLM, you know, something simple like give me a list of 10 restaurants, you know, that I should go to within 10 mile drive. And it gives you a good list because it has millions of people's information, you know, patterns and reviews and all that. But if you tell it a little bit about your food preferences, right. Or your allergies or, you know, I'm a vegan, all of a sudden the list gets better. Well, you just give private information about your food preferences to what, you know, chat GPT, OpenAI. Right. Which might use that in the future against you or for you. We don't know. But it's, this is a, we're going to do it because it's going to make our lives better. We're going to give it a lot more data. Right. And this leads into new worlds where, you know, if you're, let's say you are have evangelistic parents, religious parents, and you haven't disclosed you're gay to them. Well, a computer can notice patterns that go, oh, you're gay. Does it out that. Right. Does that cause a consequence to you? You might live in a country where that's illegal. Right. And now you have huge consequences to the society, to people, to individuals. Right.

Someday you're gonna have Rex St. John on here. I know. And the way that he thinks about this too is that that's something that if, if you have this IBM mainframe world, that's going to happen. I mean the, the idea that, you know, you, you even can go and, and look at like what's that? That they tried out the language model that the UAE trained and asked it about homosexuals. It's not good. Like if a government or a company or any central entity is controlling this supply of intelligence, you know, like one of the worst outcomes of this is actually having a dictatorship using language models to have perfect surveillance, you need to have something that's close to you, that you have total ownership over, that is completely local, that is protecting you even in this world. Like, you know, whether it's the sole thing and it's just helping you navigate this, this world or it's something that is actually by your side helping you navigate those kinds of very powerful, very intelligent entities. That's, that's totally a world I see. And again, the only antidote being something that you have total control over and yeah, something that's open source. I don't know. I don't know what else you could do. And yeah, that's a huge reason why we're building the technology in this way. It's also for those people just literally in the short term that like they want to use this kind of technology, but they're totally, justifiably concerned that a company has their information. They should get to play with this. They should get to, shouldn't have to have people who just like, they don't like it. So they don't use AI at all. A lot of us want to use AI, but we're worried about this correctly.

Well, the business people are worried about it. You know, you give a spreadsheet to chat GPT. What did it just learn about your business? Right. Yeah, a lot. Is that your intellectual property? Yeah. Did it ingest the intellectual property? Is it sharing that with other people? Is it using it for its own purposes to compete with you? Right. Amazon is very willing to compete with you eventually. If you're like the guy who makes my clothes, Amazon called him up one day, he didn't even have an Amazon account. And Amazon called him up, says, hey, we notice a lot of people are searching for your clothes on Amazon. We'd like to, we. The first call was we'll buy $300,000 of one shirt right away. Right. Because it had the data to know that there was demand like that. And now he sells millions of dollars on Amazon. But now if you get too popular on Amazon, well, they'll build their own product to compete with you. Amazon Basics, right? Woo. That's all. That's a new world we have to think about.

Yeah, yeah. And actually it was the first interest that made it like, okay, this is, this is something, this is something that just immediately could help people. Is actually. Yeah. In trade secrets. Just for. There's a reason why they renamed Code Interpreter to Advanced Data Analysis. I was like shocked because that's such a minification of what this thing is like. I'm saying the future is here. And as a code interpreter. And they changed it to like this extremely narrow use case. They then switched it back. And the reason they did that though is because the first people that. This is really helpful for is data scientists and folks that are trying to ask questions about data. And immediately something that Open Interpreter became that was like, oh no, this can help out a lot is I want to have every employee a data scientist. Everyone should be able to ask questions about our business data and not get guest responses, but run python code usually to get deterministic responses and to generate charts. And the world where that's all happening in OpenAI's servers is way less appealing to people that to businesses, frankly, that have trade secrets. And so the idea of being able to have like an on prem code interpreter, nothing leaves this, you know, it's completely airtight, is pretty appealing strictly because of data science, which is like to me, not the most interesting use case for it, but a really immediately valuable one. And yeah, just the same as personal data is company data, I think it's.

A worthy topic for people to really think through. Because you hand a spreadsheet or you hand your Oracle database over to an AI, all of a sudden it knows everything about your business, right? In probably in better detail than you do, right? What a bridge just fell down last night after a ship hit it, right? Well, what's that causing to the supply chain in the port there? The computer can see those patterns real quick, right. Human. Takes a little bit of time to figure it all out, what it all means. But a computer can get ahead of, you know, really understand your business in a way that I don't think most people are prepared for. Do you have a. Is, is this a working piece of hardware yet? Or when are we gonna. When are the first customers gonna get it? So we could try it and see it, maybe even come back and get a good demo of the first. The first 01.

We should do that. Next time I'm in San Francisco, I'd love to come out and to do something like that, but yeah, maybe we.

Should walk around, even use it and see how it works, you know, down at a cafe and try to use it with a Macintosh or a Linu and see what it does.

Yes, I would love to do that. Yeah. So the first batch of this is probably going to be going out in like late summer. That's what we're anticipating right now. You can build one, it's available. I mean, and this is, I'll say a rough thing too, which is that for folks that are outside of the US we decided to start doing it in the US only. And so I say this too, for folks that are outside the US is that it is available. I mean, it's something that if you have a 3D printer and you have, you can order the M5 stack, which is what we use. We're thinking of switching to a better speaker. It's all evolving on the repo. But the developer preview of this technology is available today. We want to really iron out how reliable it is and really iron out the form factor into something that is much, that's even smaller. We try to get it down. Way smaller.

Yeah, this is a competitor. I mean I, A lot of, A lot of the devices are built with the same chipset. Right. And you're 3D printing a case for it or trying to build a case for it. Right. This is the where.whereby.AI which, which.

Okay, wait. Okay. This is amazing. I was just talking to somebody about this and we couldn't find it. Where?

B. Yeah, where B B E E A I, I'm just, you know, people are starting to send me all sorts of different devices like this. Right? We, we. The humane pin that's coming out shipment has the tab which is a device that hangs around your neck and Rewind has a similar device to that. Right. Tell me about the competitive landscape and what's going to make your the, the open source world special compared to all those others.

So on day one of building the O1, it was originally that because of the rabbit R1. So we wanted to make an open source version O1 and we decided on day one that we were, we had the opportunity to do something much more general that if we think about it like you could build something that's, that's very much like a rabbit R1 or you could build something that's 90% of the way to a rabbit R1 and to a humane pin and to a tab. Like what, what does it look like to have like okay, everybody siloed off building in these in their own directions. This is a great opportunity to try to rising tide, lift all boats, catch everyone in this place. Like everybody's building pretty much the same technology in the humane and in the rabbit. These are language model computers. They are computers that have a language model center and that's operating the computer. Let's please unify humanity's efforts and try to make something that is open source that's most of the way there to each of those products. I think it's such a healthy competitive ecosystem and I don't really think about it too as much competitive because like I'm, I'm still very, very interested in the, in the intelligence and application layer, like in the software layer of it. And I'm not a hardware person and that's, that's why it's been a massive open source effort to build this hardware thing. And I think it's going to, we're going to continue to get great bodies hardware that is in that repository, in the O1 repo and that's using the operating system. And I'm going to continue to try to make the operating system better and better along with many, many other people. But I think, yeah, I'm excited to get all of them. I've ordered every one of them and I'm so excited. I just hope that more of them are open source. And actually a DEIS one that just came out was Compass is this other open source wearable and that's based on adais, which was launched by Adam who was actually in the early 01 team. And he I think realized that like that he wanted to do something that was a little bit more like Tab. You said earlier too like that it's, it's always listening and that these things. I kind of am more of the mind that for trust and control that I would actually like to most of the time tell it when I want it to be listening. So that's why we went push to talk.

Yeah.

As opposed to AVI's tab or adais and Compass and all that stuff which is actually all the time listening. I kind of like.

You bring up a good point. If I'm sitting at my computer, I don't care. Listen to me all day long, watch my screen, help me out, you know, be there, do it. I, in fact I use rewind on my Macintosh. It's listening all the time and makes, you know, out of this call. It'll make me a nice transcript. It will tell me what we talked about and put it in my notes so that, you know, five weeks from now I can search for it and you know, and find information that has a memory. Right. And, and this, this little device does something similar too. It has a wake word, you can talk to it. But when you start wearing these things out in public with your friends now there's a social contract problem. Right. Oh, how many, how many devices are you wearing, Killian? You know, how many are recording me right now? Can you turn them all off? We want to have a private conversation. Right. You know, and have some fun.

Yeah.

Or something like that. Right. You know, and, and so now you have to take them off or figure out how to turn them off or have to worry about, you know, are they recording your friends who didn't Consent to be recorded. You know, and recording is a funny word because these don't really record. They transcribe into a transcription. Right. Which is similar, but it leads to a whole bunch of social contract and privacy problems. Right. Did the other people around you consent to be transcribed and, and you know, recorded it in some way?

Yeah.

How. How are you guys thinking about that? You know, how, how to, how to change the device based on its context? Because some context, I'd leave it on, leave it turned on. I, I'd like, I'd like to talk to this thing all day long. Right?

Yeah.

Other times when I'm out in public, maybe not so much. I, I want to turn it on in specific instances. Right.

Yeah, I, well, first of all, I'll just say that this is, is reminding me, just because I just saw Dune of what if it. In the future you have like the like silence cone or whatever. In, in Dune, like maybe nightclubs will be these like spaces where like the silence comb is, is emitted and then all of your wearables need to like adhere to that and we'll, we'll all shut off and stop listening. I think that first of all, this.

Is definitely because maybe your friends are like, hey, we're getting hungry, right? Oh, let's ask our tabs or our, oh, ones, you know, where's a great restaurant nearby. And so again, you need to turn it back on to, you know, to serve you. Right. But it should be the human that's in control of it, right?

Yeah, yeah, yeah. And I push to talk is like the most a human possible for me, at least until we understand really how to do wake words very well and that kind of stuff. I think that the 01os should support a lot of different types of wake. So do you want to wake it up by. By pushing to talk or even just by toggling? Do you want to make it come to life when it hears a particular wake word? I'm very interested too, in the world. And obviously you would want this in like a humanoid robot where it's probably going to have a very small model that's trying to predict what you're saying as quickly as possible so as to find when you are going to stop talking. And then the exchange of control of the conversation happens with all of the intelligence of a language model instead of the simplicity of a button. And you know, I literally think like an environment variable in the 01os that determines how it is that you're going to wake it up. There's somebody too, that demoed at AI Tinkerers. Anybody. AI Tinkerers, by the way, anybody who's listening to this should go and see if there's one at their city. I host the one in Seattle. A couple hundred people come out and we'll do demos where it's the last one. Somebody demoed one that is listening to ambient silence. And anytime it breaks over ambient silence, then that's the wake signal. So you can be in a crowd. I mean, we're in a crowd and he's talking to it and there's tons of people talking all around. But because it's louder in that moment, simple, but works and it should support all those sorts of things. Ultimately, the O1 Lite is a very basic implementation of the 01 OS. Like we're trying to say, you know, this, this is just kind of the simplest thing that you could build with this. But what's going on here really. And what we're really building is something that you can build into your own products. Like I hope, and this is another answer too to the competitive landscape. I hope to see a product come out soon or maybe even one of these competitors switch to using the Owen OS and or build on top of the O1 OS. It's, I think, going to be. Then am I buying a device that's always on, like the social contract problem, if we just leave enough options open. That's more of an end user thing. I do think we're going to build some pretty great ones ourself. I'm very interested in getting into really refining the consumer version of this thing and being a part of that too. And I am only recently a developer and I've thought much more about design and about the consumer side of it. But to have something that is on, maybe I toggle it. That's how I can kind of imagine this happening. That maybe I would double tap and then it would be on at all times, but it would be on in a silent sense where it wasn't always thinking that it was an assistant being called. Yeah, we gotta find some good words for this. Cause I definitely think assistant is the right word for something like the L1, the rabbit, the humane. For tab, I think ambient computer. I don't know what's. Do you have a words. Are there any words floating around for you to describe that?

Nothing's stuck yet, you know, but ambient computing is sort of where we're heading, right? A computer's just listening to me and I hey, computer, take me, you know, to the Taj Mahal or turn on cnn or tell me what's going on on X, you know, or send an email to Killian. Right. Something like that. Yeah, we're heading to that world. I mean, when you're wearing the Vision Pro, you're sort of always in that kind of world. Right. It's not yet completely taken to the nth degree, you know, because Apple doesn't want to freak out his customers. Yeah. Right.

It doesn't.

Yeah. Doesn't developers use the eye data yet or, or turn on and always listening thing? Right. It's. It's. You can, but then you're in VR and, and you're explicitly in an app. You know that, that does that. It's.

Yeah.

Where they're holding us back. And I think that's why, one reason we're seeing an explosion of these new explorations into these new tabs and pins and things, because Apple doesn't let you do that on a phone that you can't really just turn on an app. And Siri doesn't work that way. Right. Do things just because I'm talking to Siri. Right. I have to actually wake it up with calling it. Right.

So. Yeah, yeah. By the way, I'll, um. Okay, so this is. I might as well answer this and I know I think we're out of time, but I'm okay to stay later if you are. But the idea of why isn't this an app.

Yeah.

Is totally a fair question that folks have had and I say too in the launch video that we are going to make an app. There's already in the repo react native client for this. That is just not very far away from being on the iOS and Android store. Totally reasonable question. But there is something happening about putting the language model above the os. If you have an app, you have the language model below the OS and all these, these privacy things then, then come down and it's much harder to build something really interesting with this technology. But if the language model is above the OS and it's capable of. Of calling it up and being called up at any time and having total control, you can start to build some really interesting technology. And that's where we're trying to go. But it's totally reasonable to want an app. I want an app and we're building one. I'll say too, that in terms of Apple, who is obviously going to have an incredible Siri in the next two years, and this is another totally reasonable reaction to this is. I think I'll just wait until Siri is really good. Kyle Orbit, who Does open pipe. They do amazing fine tuning stuff. You should check them out. Openpipe, they saw this, what we were doing with Open Interpreter at an AI Tinkers thing a little while ago and they were like, that seems extremely unsafe. Like this thing's running code on my computer. Are you kidding me? No, we're not kidding you. He says, I think that, you know, after, after kind of reckoning with it, he said your edges are your moat. I think like the spiky edges of this thing actually being a bit on, on the side. Like Apple would never do this. You're never going to see a Windows copilot that's executing arbitrary Python code. You're never gonna see Siri executing arbitrary Python code.

No.

You can only explore this in the speedboat, you know, of the startup world where, where this is something that we can just bypass a lot of that with the understanding with everybody that there are absolutely risks in exploring this new space. It's not a place then. I mean, I genuinely think there is an opportunity. It's gone down from like a one in a trillion chance to beat Apple at something to, for a few years, maybe a few months, like a one in a million. And we're taking the shot. And the idea of your iteration, it.

Is going to be far. Even if Siri and June at the WWDC does a really great job and I'm expecting a. At least I'm going to fix Stupid question. But here, here's one for you.

Yeah, right.

You park your car, you walk to a restaurant, you get in the restaurant, you get done with your meal, you ask Siri, hey, take me back to my car. Can't do it. Doesn't understand what you just said. If you say take me back to my parked car, it starts up Apple Maps and takes you back to your car. It knows where your car. Yeah, right.

Those are your parked car. Right.

It's just, it's an inflexible AI system that was built a decade ago and it hasn't really been upgraded. It doesn't have a large language model, it's not very flexible. It's, it's not modern, it's not, you know, new technology. So I assume they're going to put an LLM or 3 or 5 or 10 in there and it'll do a lot of, it'll fix those kinds of things. But is it going to let you run code that's, that's Apple has DNA of. You know, first of all, they have to have to make sure it's safe for every single Human that has an iPhone, which is billions of people. Right?

Yeah.

It has to be, you know, it has to do something that their employees can support and, and teach people at stores. Right. This is why the first iPad didn't have multitasking code. Right. The first Windows tablet.

Right.

But most programmers aren't rocket scientists who really understand how to write multitasking code. Well, Windows Microsoft thought it was a necessary thing to have multitasking code. Steve Jobs said, no, we're not going to put multitasking in the first iPad. Right. It's too complex right now. It's too hard for people to do and it leads to a whole bunch of memory issues and crashes and problems for our customer base. So we're going to just turn it off. We're not going to let you do that now. They do now, right. Because they have a much better multitasking system that normal everyday programmers can understand how to build. But in the early days they just punted and said, nope. Right. And the same is going to be of this new world is like, nope, we're not going to try to do any of that Killian stuff. No, no, no, no, no. You're open source stuff. We're going to be very muted and very deliberate, very intentional because it's Apple. They have to support it around the world. Right. With it, with human beings who aren't really ready for this new world. I mean, look at the kinds of conversations we're having. It's, it's a, it's a new world that we have to get into of understanding how to run a local model. Right. And understanding what the consequences of that are and, and you know, what are the social consequences of running a, a model that's listening to us. Right. On a tab like that. There's a lot to, for humans to digest and understand and use and adopt and, and figure out. Right. Apple's going to just punt on a lot of that until later. Right. And then you're going to have the early adopters and you're going to be able to iterate for those early adopters very, very quickly and add new capability system. Apple's, they're not good at that anymore. It's a big, big company with a lot of committees, a lot of meetings.

Yeah, right.

So if they hear something in a store, yeah, they do get feedback from the store. That's a magic thing about Apple. But getting that into the product and getting it shipped to customers, it takes a lot to move. It's a lot of inertia where you can get on your, your channel and in a couple hours you can bang out a new, a new version. Right?

Yeah.

So the iteration.

Yeah. And there was no startup is just.

It leads to a different place than an Apple or a Microsoft can get to.

Yeah, yeah there's, there was no, there was no reason too for Apple to beat IBM. They had no right beating IBM. And yeah, I think that we will get it safe before they get it powerful. Like I think that and I can talk for a long, we could do a whole other one about the safety of code interpreting language models. It's all I think about. There are these, these gates of, of safety that you can start to think about. But I doubt that Apple is worried about that right now. I think that they're starting from the.

Idea of there might be a much more basic system that's worried about it. They're thinking the same way you are but it's you know, five, 10, 50 people. Right. I saw this at Microsoft when I, when I was a tablet evangelist. The tablet team knew what was coming, right. They, they, they had a quarter inch thick tablet that you could type on with it with a pen. Nine years before Apple came out with iPad. Right. That they were bleeding edge but getting the rest of the company to think about it, to use it, to adopt it, to put Office on it, man, it was.

Yeah, yeah, yeah. I'm sure, I'm sure that they're, I.

Mean Kodak invented the digital camera. We don't use Kodak anymore. Right. So there was a. Yeah. Group of people who saw the future there but they weren't listened to by all the chemical engineers who were selling films.

Xerox, Stephen. Yeah, right.

Same thing. So this is the world of disruption and you know, new ideas start out being only for a few weirdos. Right. Who are like what? Like, like you said, how many tabs are you going to be wearing by the end of the year? You're going to have a humane fit on all of them things around your neck and one and another one. Right. And you're going to be talking to them all and you're going to be like a weirdo and some society but that leads to innovation, that leads to a humanoid robot that leads to real disruption to. Right. And yeah that's, that's the game with finding product. Market fit is the game for startups. Right. And you can find it much faster than a big company can for a new idea, for a radically new idea. So good luck.

Yeah. And we're just. Yeah. You, you saying too, I mean Apple being Maybe having a team in it. That's five people. We're five people. Two engineers on the team. I mean, this is just the beginning.

I met one of the guys who was early employee at SpaceX and I said, tell me about the team that wrote the code that lands the rockets back on the barges. He goes, that wasn't a team. It was one guy. I'm like, no way. Yeah. I'm like, what? Yeah, one guy. I'm like, why only one guy? Why not a team? Because one guy can keep all the details in his head of all of all the change, all the things that has to happen as that rocket comes back down. Now, he was working with a lot of people, right, but one person who wrote the code. And that's why Lockheed Martin still hasn't figured this out. Right?

Yeah.

Lockheed Martin doesn't have a rocket can land back on a barge. SpaceX does it every, every week.

Yeah. I think there's gonna be so many more people like that too. Like, especially as when you get language models into the equation. There's gonna be so much more of that. I so get that.

Yeah. Love what you do. I'd love to have, have you back on, you know, when, when you get your first shipment and you can actually demo this and start showing the product and, and then, and then we gotta compare it to the rabbit and compare it to the humane pin and the, and the Abby Shipman's tab and you know the rewind thing. Right. So there's a race. Heck, I didn't even know about this one. These guys sent me this, you know.

Yeah, that one's cool. You should check it out. I don't. I just heard about it from a friend and couldn't find it online anywhere but the B something. I think that sounds really cool.

It takes great notes. I'm just talking to my wife, really. But a minute later it's like, oh, you were talking to your wife about buying a Tesla and you know, doing this and that.

Yeah.

It's a new world.

That's awesome.

It's a new world of human. Human assistance. Various kinds. Thank you for watching.

That'd be fun to do something.

Where do we learn more about you and the project and the O1 project?

Yeah. So if you go to openinterpreter.com O P E N I N T E R P R E T E R might change that. But openinterpreter.com and right on the front page right now is is you can learn more about the O1. I mean our GitHub repo is there and yeah, yeah I think, I think head there you can go to updates too and get email updates. We have a great Discord community. Thousands of people in there that are developers and people who don't code to. I want to find a better word for people who don't code or something. There's something much. There's no reason to say it's something like that with a non but great people user doesn't it.

You know consumer doesn't.

I know, right.

Creator. Creator fits for some things. Yeah you know Influencer fits for some things.

Yeah it's you know natural programmer is kind of interesting to me a natural language programmer that what does it look like to have a. Have 6,7 billion natural language programmers is interesting but anyway Discord, you know you can go right to that site openinterpreter.com, discord and updates and yeah I would love to see you in the Discord and check out the GitHub. All the source code is there for folks to use and to build on and yeah you can get in touch with me in a bunch of different ways and we have a contact form on the site too.

Thank you so much.