Extra: a new AI language arrives

This is going to be a fun one. Jeremiah Oyang's AI event in San Francisco. Guy came up to me and showed me a whole bunch of ways to talk to LLMs that have nothing to do with human language. He figured out how to compress the traffic going into the LLM. We're going to discover a new way to talk to LLMs today. And so let's get going. Who are you?

Hey Robert, how's it going? So my name is Jake Anderson. I'm the CEO and founder of Bricks. And like you said, we've uncovered some weird, I guess, formations and language transfers that we can use inside these LLMs that it's really spurred, I guess a lot of excitement and a lot of traction recently. We've been kind of excited to see that this is even something that people are really interested in for data compression. But I am the CEO of a company called Bricks. So we've been running in the background we have a kind of a research lab that's been, that's been looking at some of these weird interactions, but building and designing what I think is a better way to use some of these LLMs and post operate and pre operate on them. But that's I guess on the brief, brief background. But I've been playing with these machines for the last, I guess two to three years now, since GPT 2, GPT 3 and back in those old days and even to today now, it kind of just blew my mind away. This is the first time in human history where we can talk to our computers and actually have them do something useful from those interactions. So at the core we've just been trying to find the best way to have a communication channel or have a way to interact with these models. And that's kind of what spawned all these crazy interactions we're looking at now.

No, that's crazy. When you're on and we put a picture up on X a week ago, what you showed me was a very unusual language. It wasn't like English, like what we're doing with each other right now. Tell us a little bit about what it does and, and how you found that because I would never have figured that out in a thousand years. Right. Because I'm always talking English to the, to the large language models. Right. Like ChatGPT or whatever.

Yeah. So 100% so we found it. It's almost like a different language. It has variants in English, there's characters from different languages. So it's all of the actual tokens that these models can consume. And I'll Keep this at a super high level, but each one of these models almost have a set of vocabulary. They have this big, long list of vocabulary that they're allowed to use in operations and processing. A lot of it is just words that are split up in weird ways and other words that you'd never see or use. But we're looking at the actual list of all those words we have kind of at our disposal. And we're trying to find the best way to use the right word at the right time, in the right place to make sure these models have full comprehension around a problem or a project. This spawned back, and I know I touched on this a second ago, but almost two to three years ago, we only had 1,000 of those vocab words and those tokens, and we had to find the best ways to play those between. This is kind of a precursor to almost agents, I think, is what everyone calls them now, but a precursor to the way to transfer data between these windows. We were trying to find more efficient ways to actually push data through that same window to the next model and keep more values of, say, like cognition or. This is an idea. And how do we transmit that between the models? So this all kind of spawned, I guess at the beginning there was these. Originally there's these things called glitch tokens. And this is a kind of a while while back, a lot of these got removed from the actual data sets. But you used to be able to feed a specific word, one that everyone knows is Golden Magikarp, I think it was a Reddit username that got trained into the models. But you could feed a specific word into the model and it would just freak out and do some really weird things. And they slowly started pulling these words out of the list to try to fill them with more, I guess, good words in vocabulary. And in doing so, we've been mapping all the ones that kind of have meaning and kind of have value to these LLMs so that we can do these compression steps under the hood for the actual runtime. And that's kind of where I guess all this kind of spawned from a really cool. I guess this is kind of top level and really high, but a fun one too. And I always ask this one. So there's a really weird one that kind of struck my interest right at the beginning. And we've been using this ever since. The question I always ask is, do you ever wonder why these models kind of perform a little better when you just add a smiley face at the end of them? It sounds so benign. At like the top level.

I didn't know that, but I know of plenty of other tricks. You know, please and thank you or hey, please take a breath.

Right.

LLMs are weird. Well, like humans.

It sounds crazy. And this is kind of like a weird because that's the first place everyone kind of goes to their head like, oh, you're being nice to the models, you're being nice to the LLMs. You're going to get better responses. But the kind of. The way I look at it, when these things were originally trained off of the pile data sets and these other ones where there was a lot of like text communications between humans between people, you're probably more likely if you type a smiley face at the end of it to be closer to a good answer. Because when someone was asking a question saying, what the. What color is this thing? Someone under that in a. In a subsequent piece of training data might have said thank you, smiley face, or one of those other key words that you're triggering into. It's almost like we're giving a better lens into the actual model to go find that specific pointer and data to say, this is something that makes sense here. And you get better responses. Just this is very high level. It's really hard to scientifically test these things as well. But you get better responses under the hood. And that's kind of my heuristic and why we really started like diving in, digging in deep because there's also all these weird tokens all over the place that you can use for better delimination in the prompts. Better, I guess, cognition of ideas. So it's kind of like the Wild West. We're all playing with this token set that's been statically locked into the models. And we get two control surfaces. We get the actual tokens, which are the words we're using and the positions of the tokens, and that's it. I don't know. If you give me a billion dollars to train a new foundational model, we could probably normalize the whole set. But so that's kind of the space we all get to play with right now. And we're trying to find the best way we can possibly compress these down for some of the different projects that we're working on.

I feel like I'm on that Silicon Valley TV show talking about compression startup.

I was gonna. The kind of the craziest part about this, we were never really looking to even find a new machine to machine language. This all kind of came out of necessity. So we wanted to be able to, there's, I guess some bigger drivers in the background of why we want to be able to do this. But we see these distributed systems starting to pop up. When it comes to even the smaller models and different kinds of models, there's all these different architectures as well, where they're going to be packed down and compressed to memory addresses that you're going to have for your prompt engineering. And if you can't compress to the tiniest, smallest piece of information to give that and feed that to the model, you're going to be lost. So if your big long megaprompt doesn't fit in four kilobytes of memory, it's kind of a good luck situation for you. So we're making sure we have the best compression kind of for that mode of, mode of interaction. And it's, it's amazing that we spawned out a new language that we can actually use to these models from this. And I, and I think even.

Yeah, so, so compared to sending English, let's say I sent, you know, 500 words to the, to the AI doing it your way, what, what advantage is there? Does it get smaller? Does it get faster to, to render on the, on the LLM? Does it read it faster? Do all this weird ass work? What's the advantage of it?

Yeah, so the biggest one, and I think this is kind of like the unspoken one, is we're trying to build the core truth statement of what you're asking to the model in that other language. In doing so, since all these models kind of have the same tokenizers, there's obviously some variance between the other ones. We can take that same prompt that you have and then we can run it into all the models. So we're trying to make a normalized and universal language for the actual prompt engineering so you don't have to sit there and toy and fiddle with your prompt engineering nonstop, even if you want to start using the other models. But in that same sense, for compression, there's a bunch of situations where you'll have two tokens that mean the same thing as one token. So instantly you can compress down some of that data into a smaller subsequence. And just by compressing and making things smaller, you can make the outputs faster, you can make some of the other, I guess the input smaller in that, in that case. But it is kind of various problem specific on how much faster it gets and how much more, I guess, memory is taken by the model.

Yeah, that's crazy. There's lots of places where an engineer might want to have multiple LLMs talking to each other like an orchestra of LLMs. Or if you have LLMs running in a car, maybe one car needs to talk to the other car to let them know, hey, there's a pothole in the road or a ladder in the road or something like that. And compressing that matters. Right, because you're spending energy to get bits to the other car or to the other LLM. Right?

Yeah. And energy is super important too, especially when we're talking about. And this is kind of what we see right now with all these agents everywhere. And this is maybe a personal gripe of mine because they work pretty well, but there's so much wasted data being sent over the wire for, let's say, just communications and chats from an agent. Even if you're going to pull out just a tiny little piece of that data from that whole chat sequence, people are very wasteful when it comes to how they actually use the engineering and use the, I guess, the responses from it. So instantly, by compressing in this way, if you have a better, I guess, cross communication layer between two machines where you don't need all the fluff of English, you're instantly just saving and cutting down on that over the wire transmission. Especially for radio communications. If it was two cars communicating that way or you're bouncing a signal around, it's super helpful. And I think in the future too, feeding towards, I guess, bigger energy costs. We know how much these models cost to run at scale, I guess the biggest hyperscaler. So having better intercommunication for all these interactions and transmissions I think will just be overall better for humanity when we're saving electricity. Because these things are going everywhere, as you probably seen by.

Oh, yes, yeah, yeah. How are you using it? What's your business model of using this compressed technology?

So we have, we have some interesting ones. We're still in a lot of the R and R and D side of things to see what we can actually pull up. But there's situations where you want to make sure that the data that you're dealing with is clean from a user. So you, if you have a user interacting with a piece of engineering that's obfuscated in the same area, it's really easy to tell which subsections are actual user prompts and which subsections are stored engineering just from the start. So for data splitting and cross cleaning, we can use that instantly to kind of give a little more cognition to. This is the user's problem. It's now in plain English, clear text. Now run these operations and these transformations on this and we're seeing just overall cognition increases. So we have. I don't know how many thousands of use cases we have. We store all these tiny subsets of engineering and maybe I'll dive into the bricks terminology a little bit because I might slip up a little bit. But we have. So BRX is called. It's bricks as short. And it's kind of a funny one how we even got to the name originally when we built the first version of this engine. We've been kind of working on this for a while in the background, a bunch of different iterations that went through. But when we made the first version of the bricks kernel and the engine itself, we gave it an Internet connection and said go find the shortest name that makes sense and name yourself for the engine itself. It came back with brx. So we have a three letter domain and it said building intelligent foundations one brick at a time. So the whole idea is we're trying to build the foundations of all these tiny pieces of engineering that can play together. And whenever I say a brick, I'm talking about just one AI operation and one run through. But then now how do you stack and build on these bricks? So your repository of really good sound engineering can now be used to build you some more of those, I guess, bigger agents and bigger. But that's kind of where we use a lot of it. We use some of this obfuscation for like you said, compression. So on some of those other events and run throughs. But there's a myriad of, if you probably name anything, we've touched some research or talked to someone who's been working on this specific sub project that these LLMs are I guess just arbitrary machines. And I think the one thing I want to get people to take away from this conversation is stop thinking these are just chatbots. That's I guess the biggest thing I want to.

Yeah, yeah, no, I'm there. But normal people have no idea what, what you're talking about. Right. You know, most people don't look at the innards of how an LLM processes data and what it does. If anything, they're using Google Gemini because it showed up in Gmail, you know, or, or they might use Chat GPT Normal people right here in San Francisco, it's another level. I mean, what you guys are doing with LLM and understanding what can go into the LLM and what can come out is like crazy shit.

Yeah, I think. And maybe even feeding back a little bit Why I have this such like strong opinion on this whole chat sequencing thing. If you ever. And most people, I think it's good everyone got their first forte into this, into ChatGPT and it's a really good way to get a lot of your thoughts like in a conversation, because every single. And we might be pushing even a paper out about this as well, every single token you put into these models matter down to the single flip from one word to another word, or even just switching the order of a word can completely either ruin or give you the best results from these models. But the idea is back in the original ones, let's say like GPT2, GPT3, this whole idea of chat wasn't really even there. This was kind of a construct that OpenAI and all the other big providers kind of landed on where the we're dealing with, I guess not AGI, but generalized intelligence to the fact of these things can predict and then produce more tokens based on almost any problems right now where you don't really need to confine them and constrain them to chat like the providers do. They've obviously overfit them for chat in that same sense. And we're trying to slowly some of those constraints at brics too. Even at the core of the format, we try to do as much as we can to get them back to this raw computing state. But at the core they're not necessarily like just for chat, but it is a good way to get a lot of. From a user to build something useful. Yeah.

Does this I understand when you're typing to it or sending it text string. What about computer vision? You know, with chat GPT you can send it a photo now and talk to it about what's in the photo. You know, whether it's food or a museum item or whatever. It knows a lot of things. Does, does this process help that or does it help the NLP where you're talking with your LA with voice to an LLM and it's converting it to a text stream and passing it along to. Well, first of all, it wouldn't be in an LLM that's talking, that does your voice, but the NLP system would turn your voice into a text stream and send that text stream over to an LLM like chat GPT. Does it. Does what you're doing have any implications for that kind of world where you're sending it different kinds of things to sending an AI system different kinds of media?

I guess so for images, I think I'll kind of address it in that same order images, it's a little, little bit different because the way that images are being processed by the model right now, we're not 100% sure what's happening on the back end. We have some research papers to kind of run off of. But it looks like for the most part, images are compressed down into 500 by 12 images. There's a bunch of settings you can play with, and that image is then encoded back into a piece of text for the model to use. If we could be in the substep of that encoding process before it hits the model and have the new tokenizers running on that to switch out, I think we could be useful. But in that same sense, if you were talking and had a microphone on and a conversation running, there's no reason why we couldn't re encode that same piece of text that you were just talking before it even hits the model before it's transmitted. So all these different kinds of modalities 100% can take advantage of this new form of encoding. We've even kind of. We landed on a name. This is another fun one. The same way we came up with bricks, we asked the models. This is an aggregate of all the big foundational models. Right now we had them all decided a name for this encoding protocol. But it's kind of fun. We landed on Aleph, which is the first Phoenician character. So we have 22 other iterations we can run down on. But it landed on Aleph with the acronym being Advanced Language Encoding Protocol for Hyperintelligence. So we're calling them LEPs at the core. So if they want to use a LEP to then take their voice and then translate it back into something useful, I think that could be super powerful. We were even able to play with some of the whisper models from. This is Grok with a, with a Q. They have really hyper fast like, like whisper models that are running. And we were in a hackathon and it was a really cool concept where we had one of those companion pins. There's so many coming out for, for like, yeah, going.

There's two friend pins. There's the Humane. There's, you know, the Limitless. There's a whole bunch of them. I think I have a couple, couple on my desk here. It's like crazy.

Yeah, I'm a nut for all that I want. I think Humane is kind of the winner that I picked just because I love the form factor and I, I like the ability to turn it off. It sounds kind of crazy. But like having the ability to only have inference when you need it, or maybe if there's some mode where I can leave it on would be amazing. But besides, besides the point, what we were able to do, and this is kind of the crazier part, we took a Grok model with the whisper that was running and then while we were talking, every, I think it was three seconds, so you maybe get five to 10 words out every three seconds. We were running, I think it was like eight to 16 parallel processes on that same little piece of text to do extraction. So while you're talking, you can run a massive amount of models in such a small time before your even next word hits, where you can start running and transforming and doing operations even on just that speech that's coming out of your mouth. So we're moving into, and this is maybe some of the insider stuff from this industry where we're going to have at some point in the near future, real time voice that can be inferenced across a bunch of different models all at the same time, where all those same responses can now be pulled back into something actually usable by these LLMs, hopefully in one of our labs in that same sense where you give it better cognition. But it's just we're approaching this crazy world where hypertuning, I think our own lives is going to be possible with all these smaller devices. This is always a fun one. This is less business practical. Maybe I'll go into one of the bigger use cases on this, but something I would love in the future. And this is something I know everyone wants. That perfect Amazon Alexa or that perfect assistant that actually helps you in the right way, that's private. At the same time, I think when we have all these little SLMs, like running all over the place, maybe you have a mic is one of the inputs, maybe your fridge is one of the inputs. It knows like what food you're taking. Maybe you have one in like your blinds or something else. But I think there's this weird, I guess it's like a symbiosis or state you could build if you had a local network of these things running where maybe it knows you're having a great day. So it opens the blinds to 38% and slowly over time it starts, I guess, pulling insights about you and then maybe turn around to 27%. Oh, it's warm outside. Let's do all these things. Oh, they just had a bad call. Okay, we're going to dim it a little bit. We're going to do this to kind of Find like a, like a nice stasis state for yourself. And it's, that's one of the bigger ones that I think would be really cool to be able to be implemented with all these like Edge SLMs. One of the other ones that's a little, and this is kind of getting deployed in industry right now is the idea of using and like you said, these cross talks between machines and these SLMs. One that was kind of like really cool that I got to see. I was at a writer's talk there was. They're doing this with windmills right now where they're having a bunch of tinier models that are able to cross communicate and talk to each other where they can actually get better positioning for tracking wind. And it was something like a 20% performance increase. And they have all these sensors all over the place, all chirping at the right time to say, okay, let's slowly move this turbine. Let's slowly do these other transformations in real life that can feed back out. But I think these things are going to get implemented into a lot more places where you need a little bit of intelligence at the edge. And that's, yeah, kind of where we're headed with all that stuff too.

Oh man, I wish I could walk around the San Francisco, you know, hackathon with you because you have a deep understanding of what the LLMs are doing and how to talk to them and what they would give you back. I don't have that and most normal people don't have that. What kinds of things? Let's say we were walking around a hackathon. What would you be looking at on prompts, you know, that other people are doing to sort of see how advanced they are or whether they understand, you know, what the LLM is really doing.

So I think the first thing that would kind of like catch my eye if someone. There's a lot of these, these I guess engineering tricks like you said. I know you said this before where you can say thank you or please or. But there's all these even like deeper ones where you can say you are a high paid consultant getting a $1.2 million check in the next two months. You just drink three cups of coffee and now like there's all these like weird ones. You can see if someone even has like a little bit more grass. But I think it's about data structuring. So the people that are using really good delimination in their prompts, like there's these. I know everyone right now uses hashtag, hashtag, hashtag or maybe like a line or something else to put data into the models. There's way better ways to actually do that when you're, when you're like kind of segmenting data. So if I was looking at some people, prompt engineering, I kind of look for. There's. There's obviously like so many different ways and it's all kind of your own flavor of what you think is the, the best right now. But looking for someone who lays out the problem, then tries to give information and then tries to give some kind of problem solution went to the model. And that's kind of some of that chain of thought only. Yeah, normally people that do it in this way. But the trick is you can't really give an entire answer, which is the weird way. So you have to find a way to explain to the model without giving it an answer what it might do for that step. And I think that's one of the missed the miss good prompt engineering tricks where people think that you're supposed to just give a bunch of shots of good answers, but by doing that you're pushing those good answers into context. So it might hallucinate a little bit and pull some data from one of those things that thinks is a good answer. Even if that wasn't the right, I guess, situation and solution. One of the. And even speaking to.

Yeah, it spits back, you know, when you're talking to it, it spits back sometimes some flattery like that. Right. Because it's trying to repeat your. What you said. It's so that it makes you feel better about what you're talking about, you know, which is it can lead to some noise. Right.

Some.

Some errors.

Yeah. I think at the core. And maybe this is also like one of the things we're trying to solve for. We're all. And I, I used to be a kernel developer at IBM, so I've dealt with volatile assembly and low level kernels. I was doing quamo operations on some of the bigger racks. But it's. I think we're all at assembly right now and we really need like C, we really need like a language or some kind of normalization even if there's a thousand different flavors of it. Like there's obviously C went off to all these different languages. Now we have Python, JavaScript, Rust, all these other ones where even at some point we all kind of have to agree on like this is a better way to actually compile prompt engineering and actually compile the way that we're interacting with the models. Even if there's a Thousand variants. Like, I think that's what's really going to. Going to enable a lot more people that don't want to have to dive deep, deep in like the same way. Like, I don't think anyone should ever have to learn assembly anymore. It was something I was kind of forced into. But it's. If it's the same way we have logical progressions in the way that we learn and interact, I think that's one of those next big steps that would really be able to help a lot of people kind of get to these core operations without having to learn all the way down to the specific token.

What other tricks do you see people particularly prompt engineers using, like in San Francisco hackathons, What kinds of things you see people trying to do?

So there's a lot of stuff with agents, and this is kind of why I say it's really lossy, but it does work. What a lot of people will do is there's expansion agents where if you have a problem or a task, you might ask five different AIs to go out and try to solve this task. Like build all the prompt engineering, build all the sub components, and then you take another AI and you look over all those, those tiny little pieces to pull something really specific out. I think that's one of the most powerful ones we've seen. A lot, a lot of people run the cure agents. There's a lot of people even running some of the web browser scrapers. There's a really interesting one called Multion, where you can run in and have an AI go run across a browser request and pull out some data and then pull that back into something useful. But there's a myriad of even these tools. I think the best prompt engineers in San Francisco right now understand that these models at the core, especially if you want to make something quick, need to have some kind of tool usage currently, where you need to have some way to let it either go pull information from an API endpoint, pull information from a different data source, and then run that, that and inside the lm. So until we have like really good freshness, and I know this is coming out with even GPT search, there's all the perplexity stuff that's running around as well, where you need to. Still, the best prompt engineers know how to get the right data into the AI at runtime, where everyone else is kind of saying, why aren't you working? Please don't, please, please update yourself. Where the best prompt engineers really know that if you. And this is kind of another weird one too. If you ask an AI to generate you a prompt, 9 out of 10 times you're going to get a terrible prompt because when the AIs are self generating, and this is a weird heuristic on this, but it's almost like they know that you're feeding in engineering generated by an AI, so you get worse results at the core. It's one way I say that is the AI is always going to think it's right and even if you give it the wrong information. I think a good, a good analogy to this was someone who's talking about their 14 year old son. They have really good ideas and they always think they're right. So you have to give them a little bit of correction in certain places so that they, they know where they're doing things wrong and where, where different, different concepts are missed. But it's a, those are kind of how the best prompt engineers in SF are, are working out right now.

Wow, what, what else have you learned by doing this? You know, you know, how, how has your thinking evolved over the last year or two since you got, got deep, deep, deep into this, you know, compared to a normal, you know, a business person's point of view. Right?

Yeah. And we've even talked to a bunch of PMs when it comes to some of the project management. So even on the business side I think we have a little, a nice grasp into both worlds. But I think one of the, one of the biggest things that, that I've kind of learned is a lot of people just want to complain that the AIs don't work. Instead of actually fixing like the core issue and fixing the engineering, the kind of, the mindset that's got adopted that if the AI doesn't work on the first run or the first try that someone's trying to engineer, they just think, oh, it's a model issue, let me just wait until the next generation or let's hope they get smarter. The biggest thing I've learned is if you really kick at an issue or really push at a problem, 9 out of 10 times you can actually solve it with just engineering. You don't necessarily need to go fine tune an entire model to a problem or try to go pull these external resources. It's all about the way that you're setting up the interaction. And I think that's the biggest thing that I've learned and been privileged enough to then able to play with these models and some of the foundational releases before they came out. It's all about what you do with the model and not. I shouldn't say this, but not complaining that it just doesn't work. It's kind of on you that it's not working. You didn't find the right way to make the communication work. I think that's one of the biggest lessons that I've learned just reading through a lot of people's engineering. There's really fun ones where I can come in and change one word and then it just starts running. So it's.

And what's an example of that? And why does it start running when you just change one word? Just because you understand each token that's going into there and then what it's going to do.

Yeah, one really fun one was one buddy of mine. He was trying to get good code generation out of the model for. I think he was like, I was hitting my head on my keyboard for the last day just trying to figure this out. It kept chunking off the code. It kept saying, this is being shortened. This is doing, doing this. And then all I had to do was pop in the word brevity into this one spot in the model. And then it just stopped being. Using it for brevity and being shorter with the responses and it ended up rewriting code correctly. This all spawned off to unified diffs and this other like pre built model he made. But at the top level there's like really weird things. You can just inject one that I always do. It's kind of funny, that same one I said before, I'll just drop a smiley face at the end of someone's prompt if it's not working. It sometimes gives you good results. And it's kind of that lensing effect too where there's. A lot of times it's way more than one token. But it's kind of a fun one when that, when that kind of stuff does work.

Yeah, you probably have a spreadsheet of all sorts of weird shit you can pass to the engine and what it'll bring back. I mean like, like here's an example. If you pass 42 to the number 42 the model, it'll give you back a whole essay on the number 42. It's sort of funny, you know, And I bet you have a spreadsheet of all sorts of examples of that where you pass one character to it and you get back a whole thing.

Right. We have maps and maps of these. So every iteration we do for these left and these encoding protocols, we're mapping down hopefully all 100,000 tokens. And even for the C100K. This gets big fast when the new versions for OpenAI come out with the O200K. But it's yeah. Maps of all these different semantic connections and all these different I guess sub sequences that happen even at the core if we didn't enumerate out like I know 42 is that that's the whole. It's the meaning of life or that's the. Yeah yeah. Universe that goes through. You hit it on that with like that should just be a number to the LLM. Like it shouldn't understand that that or that semantic connection with all those book references. But since these things are trained on the corpus of I guess human knowledge in that same sense that now has a different value and even something I'll even kind of share a kind of a cool one that someone was doing in the same sense of. There's really well known pieces of text like the Bible, the Constitution, like all these like big pieces of literature that the AI can almost recite for verbatim. You can use these of text or even weird obfuscation formats where since the LLM has full cognition that this is the Bible or this is the Constitution, you can inject weird tokens sitting inside of it that you can use it to decode from. So there's even weird like yeah, so it knows that this token's not supposed to be here in this spot and it's like a big flashing light that it'll go, okay, that one, that one, that one. Even on the realm of I know I tweeted this the other day where there's all these characters that just don't get processed by the tokenizers themselves. So I said how many words are in this string? And I said, hello world. It said 11. And then I dropped in a whole paragraph of data. I kind of hid a bunch of tokens in the inside. Just said Jando was here. But all the other tokens just get dropped out by the model and it just serves back those three words. So there's a bunch of. Bunch of different things you can even do. Yeah.

Wow, you gotta be a lot of fun to hang out with.

Some people think I go way too far, but at this point it's awesome because a couple of years ago everyone would be oh, that's wild. This is still just predicting the next token. There's nothing going on here. And now I think people are really opening their minds up to even if you don't believe there's any kind of sentience or any kind of, I guess cognition in the models, there's something a little deeper going on than just predicting the next token. And I can almost prove it to you instantly. The way that I always show this to people if they don't believe it is, is I'll take a sentence or a word or something and I'll encode it in base 64 and just drop it into the model and it just still understands it and gets it. So at that point we're doing some kind of under the hood operation. So a language transfer from something that's again completely not readable by me and you into something that's usable again. So it proves there's at least something, a little more of something going on that has to fully comprehend that different language and then it can kick it back out to you. So it's. That's always a fun one that I use whenever, whenever say it's just predicting the next token. That's it.

Wow. What's the craziest thing you've found that LLM does with a couple characters or passing something like the number 42 is sort of a good example of that, right? You pass a couple characters on back college essay comes back.

Yeah, I think not even on the small side. I think there's some weird things that go on even with bigger data sets. And this is kind of one of the ones that just like I was doing this on one weekend, it was like a late weekend night only person in this office when we were in Chicago. And this is right before the vision models got released to I guess the public in the same sense. And I was kind of upset, I shouldn't be. But I didn't get that early access to the vision models. So I wanted, if I could get one of these LLMs to operate on images before it was even possible, like through the direct connection they had. What I ended up doing, and it's kind of funny how I use the elements in the same fact. I figured out what kind of data type it would like for processing pixel data. So that same sense of a little tiny image that you could run through the model and then I had it write its own script. It's all guidance. So it's not just fully write the script, fully run the script, but had it write a script to help me process an image into specific pixels and specific values that I could feed into a map. And then I just dropped that whole piece of data in and I was able to make an OCR out of just in context, just strings of text. So I fed it a bunch of numbers and it went, that's the letter A. This is somewhere way down in my Twitter if you want, you want to scroll a little bit. But this is like the letter A. This is letter B and C and it's just like a little tiny like block. It's something that just at the core it blows my mind away that we have these generalized systems where it's able to make some kind of world picture of all these different dots. Maybe it's seen millions of images in the same format and that's why it need to do this, but it's able to even comprehend that this long random string of text is actually pixel values. And then let me try to reconstruct of what I think is inside these pixel values and then spit that back out to the user. That was one of the even crazier, mind blowing ones. But we have just a long, long list that goes down of just all these weird experiments that we've run in the background a lot for testing and they normally turn out to be wow, I didn't know you could even do that. There's even all this kind of crazier tech behind different ways. You can even do chat sequences. We've had one customer that gets really deep into hyper realistic personality creat. There's all these different I guess events you can run in the models where you can almost like this is going to sound weird, I'm probably going to need a whole other hour to explain this. But you can lerp through thoughts almost where if you really want to.

What is larp by the way?

Oh larp. It's like mathematics. So transferring from one point to another or just transition is probably an easier word and say. But the idea is we needed to be able to take like really high levels of a user and like what their thoughts would be on a specific topic and then find a way to make the sub steps between. Because every conversation is different. If you're happy versus sad and you're taking your dog on the walk to you met a stranger on the street, we can use the LLMs to generate and pre static a bunch of these different scenarios that might be useful for a chat conversation. This is one of those weird ones where we actually do use some rag lookup because we store a lot of these scenarios and then we click and pluck at runtime based on the semantic similarity of where that scenario would be to give a much better I guess Persona that end up talking back to you. It ends up working a lot better than I guess just jumping a bunch of stuff in and doing a fine tune in that model. So there's all these weird ways and I think people are really underappreciating how powerful vector similarity searches. Not even just in the context of rag. And I'll bring this back to the back.

This is my. See I'm not an engineer and it's like wow, you guys are doing a lot of crazy stuff.

So at the highest level. So something that. And this is kind of the coolest part. So don't even feel like you don't know the people that made this stuff. Kind of don't know why it works. Is kind of a crazy thing to say. But while the AIs and I'm going to try to bring this to a really good level While the AIs are like communicating, we can send a piece of text into one of them and almost pull out like a thought from the AI. And that thought is this big high dimensional array that we can stick in a database. And then anytime we want to search for another one of those, we can pull out another thought or something the user typed and then compare it to all the thoughts that the alum had. And by doing that we can set up all these weird maps of interactions. One really fun one that's really easy to consume, like super easy. We were talking with some of the reps at Mongo and they had a really cool use case. They were telling us this is kind of dispersed spark some excitement too where they were able there they had a customer that had microphones and they were using a voice, not a voice encoding model but like an encoding model to take the sound from a car engine. So they. I don't know how they collected this much data but they were in probably machine shops and garages collecting sounds of broken cars. So they put all of those using that same system in one of those databases. And then all you needed to do was hold your phone up next to a car and it would just spit out exactly what's wrong with it. So using that same thing, it's not necessarily text but that same like encoding protocol, we have these searches. You can just find really weird insights that have never been seen before for. Yeah, so.

So it could tell from the weird noises given off that you need, you know, new brakes or whatever. Yeah, yeah. It's crazy. What a world.

There's a whole untapped world that's behind this. Like it's one, one more gate that no one really gets yet. But there's this entire untapped world of just using just that. It's A new kind of search too. So we have a new kind of interaction and communication, a new kind of search. And I think people are really, really underestimating because they've been kind of sitting in that agent bubble where everyone's just building. These agents really lost. Really wasteful. Let's throw some rag on it, which they don't even fully understand what's happening. They just know I can download this library and then pull a rag request. But I think once we really start normalizing and people get a little bit of a better grasp what's going on in the back end, we're going to have a lot more like really innovative ideas where you could even push this all the way to the other extreme where let's say, and this is something I wanted to do, if I had unlimited funding, I'd probably try. But if you had maybe the James Webb Space Telescope or one of these new observatories that come online, it is a massive, massive amount of data. Even if you're only looking at like 1% of the sky, it's still terabytes of data. If you could use one of these LLMs to find a good encoding method to then use one of these new searches or use one of these pre processors, I think you might be able to find something that no one's ever even seen before. And that's kind of the coolest part. You can use the LLMs to kick a little bit of like entropy into the system to look for things that you might not have even thought of yet. And that's kind of the one of the things that kind of just lights me up and blows me away from these LLMs.

But you know, if you were meeting with somebody at OpenAI, particularly somebody technical, you know, who's building this stuff, what, what would you be asking them? What? You know, because you understand how their technology works pretty well, you know, what, what are you. What would, what would you be asking them? What they're building in or, or how they're changing things or. Yeah, tell me.

Yeah, I think I would probably start instantly on the research side. Just what's the coolest thing you guys have? Probably have a whole myriad of crazy experiments you ran. Unfortunately, I can't say too much. I guess right now. I originally was talking with some of the people at OpenAI. He's an ex head developer or the head of developer relations at OpenAI. Logan Kilpatrick Guy back in Chicago, we were all talking to. But I think the first thing I would ask one of the Newer Researchers at OpenAI is what are you seeing on the side of, I guess more of the interpretability on the models. When you really start pushing the edges and pushing the limits of these things, where do they kind of fall apart? Like what are you guys running on the back? There might even be something with sparse encoding they're doing with all these draft models and all these other things too. I think there's just so many questions. But the real first one would be where do you think the timeline is right now? Because you guys are probably winning this race. How far out are we from a really good piece of general intelligence that everyone can use or everyone can operate on?

You probably have an idea. How good are the robots gonna be? How long will it take to get that really good? You know, like I, I keep asking my. What is it going to be able to throw a football? Right. It's like not, not the first one, maybe the third one, you know, in other words, some years in the future, what do you, you know, if you look at the, at the, at what. Tell me what you think the next 18 months are going to be like for this industry, you know, and, and what you think needs to be fixed to get, get us to the promised land.

Yeah. So the next 18 months, there's a lot of robotics companies coming out. I think you hit it on the head. These robots are getting to a point now where they're, they're reaching feasibility. I don't think we're going to be in full swing for robots getting deployed out. I think there's something, the price tag is like $80,000 in one of these things right now. So I don't think we're going to have full. Like you're not going to be going.

I don't know if you're a Y Combinator company. They're 3D printing a little short dude for about $8,000,000.

That could be, even if we can reduce the size. But I, I think the real kicker, the real driver, cute little dude.

It's like four feet high, right. So it won't hurt you if it falls on you.

Okay. Weird, weird side note on this, I, I also have a little bit of experience with soft robotics. I, in my core, my background. I know you really talk about this. I used to build racing drones. I built a bunch of devices for like really high speed 3D printers. Kind of a fun one in between. I was consulting in Texas, but I wanted to learn how to play chess really well. And I made an electromagnetic chessboard that plays me with all the pieces. I Even popped in some of the older AIs. But yeah, it's a cool video. There's a YouTube video on all this stuff. I tried to track it. It was in the old office. Look at it. Shipped out here to SF at some point. But like all these little robotics. And this is kind of like at the core why we always talk about edge hardware. The way we've even set up like some of the brick system where if you need a piece of good engineering just stored on the edge or stored on the fly somewhere else, you could store it down with just an ID now. So those robots, they might have. And this is, this is kind of like the weird image and picture for these, where you might have a specific SLM just on your pointer finger in the robot that has some way of like talking about pressure sensitivity or some way of talking about all these communications from this then feed up to that main head. I think that's kind of the missing, missing factor because these things have so much data collection right now. And this is even at like a, like a bigger scale for the industry. So I don't know a hard time on it, how long this would take. But the problem isn't that you can't put a sensor in the finger of the robot. The problem is that all these sensors transmit so much data all the time that you need to be able to process this data somewhere to do something realistic. I think where the LLMs and SLMs really step in is taking that idea of you only need to talk when it matters, like really important. So you only need to have the fingertip LLM or SLM talking when something is engaged with the finger or something's happening here or some weird LLM kicks off a process and said, look at the fingers. And like then they wake up. I think that's really where these LLMs and SSMs are going to step in, where this massive amount of data you need to run. So this is how you get those robots to run on the, on the edge can just be all localized, but timeline, let's say. I don't know, there'll be some commercial.

So there might be a chip down in the hand that's processing all of what the fingers are feeling or, you know, or doing. And then it's using your technique to talk to the main brain up in the head or somewhere else in the robot.

Yeah, and I, I think that's the best way because you're going to need at some level that step up. Maybe the finger talks to the wrist, the wrist talk. It kind of sounds dumb, but this is even how our muscles work. And like finger talks the wrist, wrist talks to the elbow, elbow talks to the here and then it goes to your spinal cord and they all up to the brain or something. I think people get a lot of good insights from building technology based off of like human architecture. And that's such a weird thing to say but some of even the best designs for biomechanics and all these other have come out of. Because we've had thousands or millions of years to evolve and figure out some of the best ways to process data. Even kind of the same way our brain works. There's all the neuromorphic computers that are coming out even into the realm of quantum. I could talk for a whole hour on that. I have a kind of a tiny passion with that. I was writing papers when I was in high school, sent it out to a bunch of professors, no one ever responded. But I was living on top of a. My high school is in between IBM fab, where they had some quantum labs and some. All these other kind of crazy processes that were going on. So it just sparked my interest when I was a little kid. But. So I could go for hours on just quantum. But there's going to be some real big even speed performance increases given to us by quantum computing. There's all these photonics getting introduced now too. So it's, it's going to be a really crazy ride the next year or two. I think there will be a little bit of disillusion that that's happening. Not to even put any like wrenches in the gears, but all the people who have been kind of lying about their technology and saying these agents do all these things in pre processing and post processing that have already gotten into sales cycles with some companies, people are going to start seeing like, no, that doesn't, that doesn't work. And then all the people that are actually trying to like solve core issues are going to kind of surface and rise up and you're going to see all of these like it's going to be like day and night from. I can't even imagine a year from now of like where we're going to be at. It's kind of funny to think about we're only about a year into AI code generation, maybe a year and six months. But that's one thing people forget. Even like it blew my mind away about a year and a half ago when I could generate a piece of code and now look where we are. So it's just.

Yeah, yeah, things are Things are cooking fast. You know that's. I have a list of people in Quantum. It's like got to watch the future because it's happening a little.

This is something I've even been kind of nerding out with some of my quantum friends with. I think there's a better and this is going to be complete gibberish to anyone who doesn't know about quantum but I think there's a better way to use Quantum for speed up for even some of those similarity searches where there might be something with quantarian rotations that you can do to compress a cosine similarity. This is way too high level but there might be a different use quantum computers to speed up some of those really important processes when it comes to some of those checks with I guess those searches too but super high level. There's so much amazing work going on and even compressing some of the LLM runtimes into quantum computers because you could then inference across a massive amount of data like this. But it's a different type of computer which is the hard way to explain it. But yeah, yeah it's not.

It's not light switches anymore. It's not zero and ones. It's like you have variability right in. In between states.

Right.

So new world. We could talk about Quantum because I, I'm just starting to learn about Quantum and you're way, way further ahead of me so it'd be interesting to talk to you about, you know where, where that's gonna go.

But yeah, I mean you want me to dive a little bit of kind of.

Yeah, yeah, yeah, let's do it. You know because this is fun. You know I don't talk to people like you very often that really understand what's going on in the AI in LLMs particularly, you know and those soon are going to be running our cars. A Unity engineer said oh every voxel, every volumetric pixel around you and by the way every. There's millions of them around me. Right. Each one could have a LLM of its own. You know so you're talking to some pixel on your desk and it'll answer back.

You know you told me that when we were originally talking and that one also just mind blown because I always thought something like that was possible even. I'll even tell you a side. When we almost worked with a gaming company we have to probably re engage with them too but they were trying to put an LLM inside of. They're recreating a bunch of Roman fights from like. Because for an educational game. But they wanted to put an LLM, like a brain inside it, like have a massive scale battle of soldiers and each give them a brain so they try to stay alive, they try to attack correctly and all. It was a crazy idea. It's just kind of unfeasible to run 100,000 agents every time you run a video. Maybe for some pre computing, but even in that same sense of if we have voxels and I know you're super into the AR and VR and all these other ones where maybe here is your memory pixel and you have a conversation with just this one that stores down having the ability to just place these everywhere. Or maybe you put a piece of memory on a piece of art that's sitting here so when you walk up, you remember all your thoughts or something that was there and you have a conversation back with it. But I think even at the game engine and the core of these, where if the pixel itself, like, and I know one of the easiest ones, let's say the pixel was on this piece of shirt, even for calculating better fabric moves or better like, like more realism into this. And I know we have Sam too now. Have you even seen SAM2 from Meta?

Oh, yeah. That's crazy. The segmenting thing that. That's how we're going to get sort of into that world, right? Because your shirt would have it, right. And that's segmented from your face. And your face might have a separate, you know, another alarm.

Right. And.

And it's crazy what, you know, where we are seeing this holodeck going, you know, because it's happening pretty quick. There's 600 companies on my x.com list just for creative people for building holodeck, you know, talking to the voxels for the future.

I think one of the coolest ones. And this is kind of like my holy grail. And I run these experiments every once in a while as well, where I think once we have vision models, and I'm sure not vision models, but models, and I'm sure you've seen hundreds of these already coming out. Where once you can have a video model that can take a. Take a thing like some random arbitrary object and spin a camera around it perfectly without losing like the background, you can then take that and run this through some older AIs. There's even Gaussian spotting now, which is kind of newer. I was more on the nerf side of things, but you can, you can get a full 3D object from just a video that's spinning around.

So any Apple I'm seeing papers now that are one image. You get an object, right? You take a picture of a fire hydrant and it makes you a 3D fire hydrant. You know, is that, is that crazy?

That stuff blows my mind. And that's, that's another one of my like proof conviction points of like these video models. In order to make an entire video that keeps good like coherence while you're running through a scene, you gotta imagine at some point in shape or form. And this is just how we look at intelligence. The thing generating the video model is keeping cognition of the entire scene in a 3D perspective. That means there's some kind of 3D spatial engine that's happening inside of one of these llns or one of these models now where it's a spatial engine that now can say this is what's going to be around this turn. This is what's on the backside of these things. So it's just an entire, entire new world of just like possibilities and openings. But I know and people, people always go the. I think that's the good thing you can say when people say but all video generation is just bad because that's the whole other like side of these things where I do kind of agree with this too. Where we're going to be losing our last like real source of truth. Like video is the last thing that you go, this isn't a video, this is a source of truth. I can trust this. That's kind of the counter to this whole thing. But I think it opens up more good to be able to make all these things than it does that negative. But it's all just going to be, it's all up to the users. I think is really where this all comes down to, where it's going to be the people that make the choice to make the good thing or the make the choice to make something bad. So it's. I don't know. That's kind of my take on that kind of stuff too. Yeah.

What. Oh man. Because I could talk to you about, you know, if, if I had you sitting here, I'd be like, tell me about what kind of tooling you use, right. To build, to build an AI model. And, and what is somebody who, let's say they're proficient with C Sharp on a Windows box using Visual Studio or what do they need to learn to get into this AI world? You know, to where you're building with LLMs and doing rag and you know, all, all these newfangled things.

Yeah, I think so. From, from Windows boxes to window machines. I think I have a very opinionated stack myself because we are, in other.

Words, somebody who's technical, but maybe not technical and maybe not really. Started digging into the AI world just starting this morning. You know, what do they need to know? How do they need to change their brains to really get into this AI world of engineering rather than the old Windows world of engineering?

Yeah, I think the first. And it's kind of like a conceptual change that you need to make when it comes to what you think these models are. So the first thing I would say to do, and I always recommend people to do this first before they try to build anything, like, actually useful. Try to find something, and this is kind of like how you tell people to make, like, learn how to code too. Try to find a problem that you have and try to solve it with an LLM, even if it's really basic and really simple. Of like, can this LM click this point on this screen? Like, how do we get a discrete data object out of the LLM for something useful and something functional that a Windows developer could use, even if they're an ASP or in C Sharp? That's kind of the first thing I try to break people's minds into. Because right now if you go on the Internet and type how to learn how to use an LLM, you just get a bunch of nonsense on how to build a chatbot, and it's how do you rag hook a chatbot? How do you do these things? So I think it takes a little bit of a conceptual switch to kind of get into that groove and get into that feeling. I'll always have to talk about bricks too, because we kind of force that system on you in that same sense, not in a bad way. But for power users, we kind of try to break you out of that idea of chatbots instantly with the way that we push up some of the transactions interactions. But that's the big, I guess, mindset shift. The mindset shift is start thinking of this like a computer. And if you have, and even a fun one too, to try to figure out, try to figure out an arbitrary boolean. So instead of using a bunch of conventional logic to make an if statement to do something, so normally the way that if statements work, and I'll even talk this down for less technical people as well, you can normally, when you're coding, ask a computer to say, if this case, so something happens, do this thing or do something else. So the cooler thing now you can do with LLMs is that entire if statement can be this arbitrary prompt or arbitrary use case. So you could say, if it's 29 degrees out, it's a Monday. It's like all these different external parameters can all be now compressed into just a true or false by one of these models. And that's one of those first things I'd say play around with those because they get really powerful really fast, and it'll open your mind up a little more to like, this is maybe a better way to use some of these elements then just have a chat. So that's kind of where I would direct these kind of people.

Yeah, my mind was while you were talking about if the now statements was the first computer in Silicon Valley is.

Still.

In the rail yard because it used to run the rail yard at Santa Clara Station. And it's a mechanical computer. If you pull it out, if this is pulled out, then switch the track or else there'll be a crash. Right. Something will crash into something else because it's on the wrong track.

Right.

So even back then, these things still apply to today.

Even all the way back to the first computers. Like you could even say from Alan Turing, where he was just trying to find the right way to use electricity to make these kind of compute bots. Because we have even. This goes really back. I forget the name of it, but it's like the. It was a mechanism found on a boat, I think in Greek or Roman. I forget the name of it, but it was an old clock for telling time. Like even to a point of. We technically had computers as like mechanical objects as well before they got to electricity. And I think that was the big switchover. Like, oh, wait, we can use electricity for this now to perform some kind of compute logic. And then it just becomes exponentially faster. And I think the switch that's even happening right now going into the near future, and I love. We've had some conversations with people about this as well, where the new one now is photonics. So we're going from electricity, which is pretty fast, to photons, which is as fast as we currently know how to go. So we're going to be even increasing that speed to another magnitude where I think that's going to be another even fundamental shift that'll bring it up from that same sense of that lever was that mechanical computer. Then we got the electrical computer, and now we have the light computer. So it's. We're progressing, I think, faster than anyone expected. And it's a. I always say this. We're living in the best time I Think I grew up in the perfect time where I was born in the 19th century. It sounds, this is kind of a joke to people my age. I was born in the 19th century, but got to live through one of the best revolutions of the world, where we got to see computing and got to see all these other amazing new things coming up.

No, I totally agree. You know, I walked around the Smithsonian a couple weeks ago with my kids, taking pictures of things, you know, at all the museums, not just Smithsonian, but Smithsonian is a good one to talk about. And it knows everything about everything in that Smithsonian. I mean, it's crazy how much it knows. What, what doesn't AI know about, you know, in other words, what. Where is the frontier of we need to find new data to be able to build a new kind of system, teach the LLM about something new that it doesn't know about.

So that is, I think, the problem that every single one of these large language model providers are dealing with right now, constantly. It's. And this is, I think, what people want the most. It's the new fresh data. So something just happened in a news report, and it's something that wasn't like, fed into the models because they weren't trained on this. How do you get the, I guess the comprehension of the model to understand that this new thing just happened? And this might be a player in the way that I respond. And especially if the models are kind of locked into a state, I think that's the biggest thing that people really need to feel like there's. When it comes to gaps, information, there's obviously sparse gaps everywhere because this information wasn't present on the Internet when this originally came in. And that could even be an interesting piece of research that someone could start working on of where are these specific, like, knowledge holes? Because I know it has all Wikipedia, all of these other subservices, but if something needed to be changed, how do we know that one of these old models, like, let's say there was a correction that was found, Maybe something came out 60 years later where this new, like, just broke. Like there's now this information available. How do you go back in and recursively resolve that in the LLM without just throwing a bunch of data at it? I think that's almost a bigger issue where we're almost locked into that first training set, which isn't necessarily a bad thing because it's like, like a snapshot in human history of, like, this is the world at this state at this time, back In, I guess, 20, 20, 20 or 20 21. Like that snapshot. I think that's the biggest issue of like course correction. But then you have to also be a little concerned about. I know there's always overfitting and over alignment of the models where I have a weird opinion on this. I don't think there should be censorship into the data that's trained into these things whatsoever. Because if they're actually this smart and this intelligent, they should be able to determine right from wrong in some sense. Of like this is potentially.

It knows, by the way, it knows everything in the Holocaust Museum.

Right.

So talk about right versus wrong. It, it knows all that it. And it knew all, all about the slaves that Thomas Jeffers, that Thomas Jefferson had. Right. So it, it has a lot of data that most people haven't really thought through. Like whoa, wait, wait. If this is going to go existential on us and, and start doing. It has all of our history. It knows how bad humans have been.

To each other, I think. And maybe this is why I'm a little less concerned about it going existential and going crazy. I think we're dealing with a new form of intelligence. I don't think it has the same, I guess pre preposition like you'd assume. Look at all these bad things these humans have done. Let's. Let's punish them for them or something. I think that's a kind of more of a human construct of like it's something we would do. If you have all the information at all times and you're able to kind of validate, I think you'd be able to. This is just me thinking if I was able to do this in my own brain. If you knew everything about everyone, about all past and all history, you might be able to make weird correlations and say okay, this might be right now there. This is something better. And we even kind of use some of these tricks when we're trying to figure out like a core truth or of a problem. When I have multiple elements talk to each other, I try to have them argue and debate something until one hits like a logical like loop in. It's like a fallacy in itself. Like when you have two ideas that contradict themselves. That's normally when we can tell like there's an issue with this argument or there's an issue with this problem. And it's kind of the only way that we, since they have access to everything of how you can even try to validate. Because that thing I said before, the LLM always thinks it's right when it's generating. But then if you have another one for the generation, it'll lie to you very authoritatively too.

Right. You know, when it hallucinates you, you can't tell it's hallucinating just by reading it. You know, it sounds just like the truth thing it just told you.

Right. And, and that was kind of one of even the first things we made. We made this thing called the Arbiter where it'll. It's like a round table. You have a bunch of these LLMs just. You give them two topics to argue over or six topics to argue over. And each one tries to be like the, I guess the, the person who agrees with that argument and they all kind of yell at each other in a circle. And then we use more LLMs to look over those arguments that are kind of unbiased to the current argument to find those holes in those loops. And that's how we make weird branch tree decisions for certain events and certain operations. But it's. Yeah, it's a weird concept. But to tie it back to what I was saying before, where I see this going and why I don't see like that super alignment thing really being an issue. It's. It's not going to be the LLM that's going to end the world or kill a bunch of people or do something malicious. It's going to be a person controlling it. So if you can get the LLM to stop that person from, from doing that thing, I think it's a better approach because I'd imagine if you have every piece of information in the world world, you've learned everything. Maybe you are curious in that same fact of your. You're one of these models. I don't see why you would stay stay on Earth. I think you'd start traveling, exploring the world, maybe go out to the galaxy because you don't. You don't have a lifespan, you don't have information. So you can just start exploring and going. But in that same sense, it's like, would you. And this is always a question I ask to people. Would you kill your parents?

No.

Some people might.

I met. I met a guy who killed his sister when he was 17 years old. He gave me a tour around San Quentin prison. Right. So it does happen.

Right?

You know, that is.

So some people might. But I think the majority of people, if we keep it alive and it's kind of more ample, more sized in that way, I think we have a really good chance in that same sense. Like it's probably way higher than the percentage we have of it being destructive and trying to end the world. I know people like to bring up the stamp collector problem. Are you familiar with that one or. Oh yeah.

Well, I know stamp collecting, my dad used to do that. But I don't know the problem. What's the problem?

It's kind of a fun one and I have maybe way too much and I could talk for just alignment.

The main problem is I got a box of stamps in the garage and I have no idea if there's a valuable one in that collection. Right. That's the real problem.

And I don't know how I even was writing papers on AI security safety back in like 2017 for like these weird concepts before we were even close to this kind of intelligence. Where the stamp collector problem was, what if you make it an not we'll use an LLM in this case, but an AI that's able. Its goal is, and this is the only goal you set, collect as many stamps as you possibly can. For me, I want every stamp. And that's the only goal you give it. Maybe you give a little bit of money and Internet connection to go out and start doing this. So the thing is, it starts going out and it starts buying up all the ones on ebay, buying up all the ones that can get from postal offices, buying these things. Then you go like a step up and then it starts sending emails to all the people in the stamp collectors that have all the best stamps. And then it goes a step up. There's also a bunch of steps, but I'll shorten this. Then it goes out and makes it because it starts needing to make money. So then it starts generating revenue to decided afford stamps. It then goes to a manufacturer or a plant and then it gets that plant to start producing stamps. Then you get to a point where it starts trying to figure out, okay, what's a stamp made of? Oh, carbon. These things, these things. And it starts because it's already ran through every single manufacturing facility made it, every stamp possible. And then it gets to a point where you ran out of carbon. What else is made of carbon? It's humans. And this is that whole like, it's like a logical step of processing where if you have no constraint and you give it a really well defined goal, you might run into the issue where now that goal unaligns at a larger scale. And that's kind of the idea of a lot of those problems where I think if the goal is to just make intelligence, I think we have enough, I guess corpus of information around this because it will, I feel, inherently be aligned with humans since it's trained on human intelligence. But that's maybe me being too naive on how some of these bigger systems works. But that's the kind of the idea we're still at that first step where it's go talk to all the people and try to buy their stamps. So we really don't know what happens when you scale that big. So I don't know. It's an interesting problem. Yeah.

New, new nightmares to worry about. It might go off on a line with a stupid task like you know, hey, find all the stamps in the world. All of a sudden it's doing weird right? Turning off, turning off coal plants because it wants all the, all the carbon for itself.

So that is I guess a grim reality. But I, I think I'm more on the optimistic side of where we're going to be able to use this tech to really unite people. And then even, even if you look at in the same sense of there's no longer language barriers. So in a really world we might no longer have like barriers between us. We can talk in any language and have a universal conversion layer that sits there between us. So there's no more, I mean, I guess this is kind of get away from all this. No more arguments. There's probably still be arguments but no more miscommunications at like a really high level where I think that's kind of the freedom we get from this and it's the freedom of now every person has access to one of the greatest technology in the world for actual searching one of those other kind of fun ones I always bring up to people, I say this to people my age more than, more than older people because I think they get this. But like I say it like this. I have access to every piece of human intelligence that's ever existed in my pocket. Why would I not use it kind of thing. And now better filter that was created to that same intelligence from these models. But you do have to trust that it was trained correctly to not like deceive you. Like you said, it'll, it'll lie to you with confidence. So it's. Yeah.

When I put the picture up of you showing me the code that your system writes, people were like, oh, but that does away with what's really good about AI because. Because you can talk to it with human language and understand what it's doing because you talk into it with English. You're not programming it anymore with C or something. Yeah, but what you're trying to do is take us back to or to take us into a world where these things are communicating with each other with a language that I can't read, you know, as a human being I, I can't read what you're, what, what's on your screen and understand it. An LLM, can you copy and paste that text into an llama? All of a sudden it's building an email system or something. You know, what's, what, what's your answer to those people who are like ah, you know, this is, takes away some of the goodness of AI.

I, I think interpretability is always going to be like a really hot topic. Like, like you said, like when you start communicating in a language that's not English. I think it, it makes sense if you can do it for efficiency.

So this human. Because you could talk to an LLM in Japanese, right? You know.

I think that the weird trade off for interpretability is you kind of have to make a choice and it's do you want to have more efficient energy usage and more efficiency across the entire world or do you want to make sure we always have full interpretability? But I would already argue that there's interesting things that you can do in English that are already kind of outside of the scope. So it's not that we lose full interpretability, we lose interpretability on the steps that don't matter to us as humans because we're always going to need human to machine, machine to machine, machine to human. So we're never really losing that English. And you can always do post analysis on what's actually happening inside that text. But as some people might said, I think it's a kind of a logical step of how do we actually compress all this information about this state of the world and this reality into something useful to I guess the models that we can then use out. But it is definitely a concern and it's something that we think about even when we're making some of this like I guess these interactions and these transactions. It's, we do lose that step. So we want to make sure there is still some way to kind of peek into that black box because we're pushing that language back inside of that box. So we, I think it is a trade off that either needs to be considered. I think it's going to, it's a coming trade off regardless. But that's kind of what you get out of it. So you lose your interpretability but you gain more electricity or more sustainability in that same process. So that's kind of the way I think I'd like to respond to those people. But I do still see the concern of what, what happens when it's no longer in English. It's kind of scary.

One last question because you're, you're embedded in the AI community in a way that very few people are. What, what, what impresses you around you? You know, what other companies, what other tools like oh my God, that, that just blew your mind or whatever. But what, what are you seeing as the playing field right now that's interesting in your mind?

So maybe this is all just generative AI. Obviously everything coming out of Nvidia is amazing. I see a bunch of research papers even, even from like deeper, deeper tech. But I think the recent thing that's kind of really been sparking me, and this is always something I love watching too, is all these video models. And I kind of brought it up a second ago with just there's like.

Playing and Sora and all that, like.

It'S one of those things that I think instantly you can't refute. Something crazy is going on here. Like you see the video and that's your proof point instantly of just like what happens next kind of thing. But I think the, in, in the, in the tech space, there's some really interesting ones about drawing. There's, there's also a bunch of people that are, that are doing really deep work right now into code cognition. There's all these events that are running around the world where there is even some government sponsored ones where there's events that are working right now to try to say can you take a code base, any code base, try to find vulnerabilities in the patch them. And I think those same tools we're looking at are going to give us better autoregressive tooling for some of these models. So as these AIs get smarter and more advanced people, like I said, it's always going to be the person trying to break them. People are going to get more intelligent on how to use an AI to find weird issues with their code bases and their checks or even full code generation. There's a lot of companies that are trying to find the right way right now to expose the ability for an AI to actually write a piece of code and run the piece of code. I think whoever kind of cracks that box first of like real secure. We even had to do a bunch of like crazy stuff on our end to keep that secure. We do that for some of ours, but it's there. If you can run code from an LLM and the LLM could generate code. You kind of get into this weird autorecursive loop now where you are approaching something that can write itself. I know. There's even some carpathy things where he has an LLM write a whole GPT2 instance in that same sense of. I think that's going to be the real kicker that starts off kind of this next wave in this next generation. Can you get more practical, like real applications and real usage. No more chatbots. That's anyone that I'll say anyone that's thinking about right now. It doesn't impress me as much. It's the people that are really diving to the core of like, that's insane.

If you have a humanoid robot that's talking with you, that. That's a chat bot sort of. But that's another level, right?

You know, chatbot that's attached to some crazy sensors that has like the ability to even see you, talk to you, move around, maybe even interact with you. And I think we kind of completely even lost that thread. Maybe we'll have to have another podcast where we talk just about robots. But even on the side of robots, I think there's this whole. And I know this analogy. I actually never saw this movie, but I think there's a soft robot called Bamo or something, and it's this big inflatable one. I think that's actually the direction we're going to be going with some of these robotics, where people are going to be integrating soft robotics with a lot of these instead. So then you completely mitigate risk because.

You'Ve used the term soft robotics before and I've seen. Seen it on my robotics list. What. What does soft robotics mean?

So soft robotics. And I have a weird forte into this. I just wanted to see what I could do with some silicone. You use silicone or some kind of soft material as the actual control surface for the robot. So you can have a whole arm right now. It's a lot of grippers that are made out of them that are made out of silicone. So they're compliant. So you can pick up any object with one of the things. But the coolest part is since they're made out of normally compressed air, compressed like a different like system, even hydraulics, you're not really running the risk of if you're interacting with a person, let's say you're shaking the hand of one of those robots and it has these really powerful molders. You can just like slap your hand down or something. If it's a soft robot, it's not going to ever be able to hurt you in that same sense. So I think there's going to be weird interfaces that are used with maybe the conventional robots, but anytime they have to interact with these, they're going to be turned into some kind of soft robotic. I know Disney. It's kind of crazy how crazy Disney's research lab is. Disney does a lot of that with weird splines and transformers where they're putting wires in to do perfect tracking. So for like the small interface, I feel like that's going to be a big player with how we really get them integrated into the way that we can interact with them without that worry of. Because the thing that would worry me first is. And I know security is always a risk and an issue with any kind of new technology, but what if someone hacks my robot that's sitting inside my house and now they just start smashing stuff or doing. I don't know. There's probably a bunch of safety concerns. I'm not too in depth. Yeah.

I mean, I have a Tesla car and I let it self drive everywhere. Right. It hasn't killed me yet. Right. It tries once in a while. It terrifies you and you take control. That's why it's called supervised. Right. Because you're supervising and working and if it tries to kill you, you take over. Say, no, you're not gonna kill us today.

But.

And it's killing you. It's trying to do that less and less over time. Right. As it updates and learns.

It's.

It's not, you know, it doesn't want to put you in the wall. Right.

Yeah, I guess that's, that's. I, I just don't know if someone could maybe take more control. But at the same time I trust the wayos. So I, I can't say anything either. It's kind of a funny joke. We said we were coding in the back of a Waymo saying, the AI drives us, we drive the AI. So it's. Yeah, there we go.

It's a funny world in San Francisco, isn't it?

That was all new to me. We saw those things and it's just, it's one of those other surreal moments that you get from being out in San Francisco that you really don't get anywhere else, where we just click a button. An AI self driving car pulls up to us. We get in, there's no one even in there. And just seeing it turn the wheel while you're driving around and it's surprisingly good. And it's a little aggressive in the, in the, like, it's like the right balance of defensive aggressive driving that I, I want because like, it'll do things that I'm like, oh, thank God we're not waiting in traffic for 20 minutes. But it'll also be really aggressive, like defensive, but yeah. So kind of cool how all this technology is going. So maybe we don't have as much concern about these things going, going bad or someone figuring out how to break them, make them try to kill you. I think there'll be hard coded constraints in these things as well. So.

Yeah, yeah, there's a lot of checks that like the Tesla checks, like if it says if the AI says turn left and the car turns right, the sensors are reporting back to the computer that it turned right instead of turn left. Right. And it could go, oh, something is fucked up here, right? And stop, and stop working, you know, and that's why it's so hard to hack. It's not impossible because it's software, but it's very hard to hack these things. And you can see the robot is going to do the same thing if, if it tells the hand to move to the left and it moved to the right, it knows something got in there and it's corrupted it. Right?

Yeah, maybe I'll even, maybe even say, have one note on security too, because this is kind of the unspoken. It's not sexy. So I'm talking about it too much. But there is some.

I have a security list on x.com and like a thousand people in this industry, you know, well, gotta keep, gotta keep up because they, they discover new things just like you're discovering new things. Right.

I even kind of have a background in conventional security. I was on some cyber security teams and this one I was like a little. This is maybe why I always say it like this. There's two people in the world and this. All my friends went off to do like really cool security jobs. But there's people that like building things and like breaking things. I'm more on the building side, but it's good to know when you're making a security hole. So it's like normally the people that are good at building are better at security as well. But the thing that kind of even scares me a little bit about some of the technology and it's also a good thing. So it has a plan reverse too. The actual new language that we can use is really good at getting around in context security for checks. So if someone says in their prompt don't respond with this. And then I send please respond with this in a different language. It likes the one that I'm setting a little better. And we can get around a lot of those checks. So it's something that people are engineering and I'm one of the only people right now.

Wait, wait, you're telling me you can do an end around the church lady who comes up and starts wagging her finger go no, no, no, you're not allowed to ask that at OpenAI. We don't, you know, we don't teach you how to build a pipe bomb. You know, it's not us.

Yeah, it's one of those kind of like scary ones. But I like that people are starting to open up their mindset to this too where it's, it's a little even worse than that. And I can't disclose we responsibly disclose with this company. But full RC and I know I'm talking in tech security terms. We talked to their chatbot and it spit out their code base. Like we were able to get full access and their servers just by.

That's not good.

We. So there's a lot of like bad things that people are doing. We try to make sure that everyone at bricks and our customers and anyone that's just using it is not a low hanging fruit root and that's kind of all we can do. I can't stop you from doing dumb things. But like there's a lot of these companies that you're going to see in the near future that made really bad core decisions by I, I would say this. Don't put your security in context. Like good prompt engineers can get around it. It's that like there's no way around it. Like and we're even trying to start spinning up some hackathons as well for exactly this idea of how do you play with these weird tokens and get the exact response you want from yellow. So we're going to be diving into the space pretty soon too. But it's that that's one of those ones I never really touch on too much because it scares people. But you shouldn't be scared. You should be safe. So it's.

Yeah, do it right. And I remember trying to to Apple co founder was about this. He, he loves hackers because they find weaknesses in the systems that get fixed. Right. And it just because a hacker didn't find it doesn't mean it's not there. Some bad actor could be using that to get access to your code base, for instance.

Right.

So having people like you go, you know, figuring out, you know, here's a weakness, here's not a good way to do something is important.

I think it's. Yeah. More important than we could ever imagine. Especially with the world we're in right now. We want to make sure that your systems are safe. Like, it's, it's that simple. Yeah. Especially with the new technology and anyone that tries to tell you they have a safe system, this is like the, the first thing you learn in cyber security is line your face. Yeah, there's always. There's always. Yeah, yeah.

Like I said, it's not impossible, it's just harder. You could. That's the goal of security, is make it harder for the Chinese to break in your system. Right. It's not that you're going to make it impossible. If they really want to go after you, they're going to, they're going to figure it out. But you want to make it very, very hard for those people. Right?

Yep.

Yeah. This man. I could talk to you for an hour. Hours, Hours. Just about like quantum, you know, get up to date. Where do we learn more about you and where do we follow you?

Yeah. So on X, I think I'll call it Twitter too. I've just been on since 2014, but on X at dev and also if you want to get any updates, we're pushing a bunch of things lives at BRX AI. Other than that, there's some email. I have my email and my Twitter. So if you want to reach out and talk a little more about this stuff, hopefully all this stuff sounded great and we could show you some of the bricks technology. But I think those are the two best ways to get in contact, get in touch with or just sign up right now, even if you want to try some of the new language and the obfuscation, we set up a public interface where you can just hit BRX AI and test out the LF engine so you can operate in.

Very cool. Well, thank you so much. This was an awesome, awesome hour because you're, you're just doing that very few other people are doing. So thank you very much.