Founder Eric Steinberger on Magic’s Counterintuitive Approach to Pursuing AGI

The thing that remains to be solved is general domain long horizon reliability. And I think you need inference time compute, test time compute for that. When you try to prove a new theorem in math, or when you're writing a large software program, or when you're writing an essay of reasonable complexity, you usually wouldn't write a token by token. You'd want to think quite hard about some of those tokens and finding ways to spend not 1x or 2x or 10x but a millionx. The resources on that token in a productive way I think is really important. That is probably the last big problem.

Hi and welcome to Training Data. I'm delighted to share today's episode with Eric Steinberger, founder and CEO of Magic. Eric has an epic backstory as a researcher, having caught the attention of Noam Brown and becoming one of his research collaborators while still a student in high school. Eric is known for his exquisite research taste as well as his big ambition to build an AI software engineer. We're excited to ask Eric about what it takes to build a full stack company in AI, his ambitions for MaGIC, and what separates a good AI researcher from a legendary one. Eric, welcome to the show. Thank you so much for joining us.

Thank you for having me, Sanjaya.

Okay, so let's start with who's Eric? You're a Vienna born wunderkind whose early passion for math turned into a, I think what you described as a full fledged obsession with AI by age 14. Take us back to age 14. Eric, what were you up to? How did you become so obsessed with AI?

Thank you, Sonia. I think I just had my midlife crisis when I was 14 and I just was just looking for something meaningful to do. Spent about a year looking at physics, math, bio, medicine, just anything really that seemed valuable to the world and at some point bumped into just simply the idea of AI. It hadn't sort of occurred to me until then. And if you could just build a system, a computer system that could do all this other stuff for me, great, I don't have to decide. So it felt like my decision paralysis was sort of resolved. And it was this weird moment where I could just see the next 30 years of my life unfold in front of me. And I was like, okay, this is clearly what's going to happen. I have to do this. It was quite nice. I like predictability, so it was great to know what the world will look like.

And you started loving math. Why AI then?

I think I'm naturally attracted to math. It's just what my brain sort of gravitates to AI just seems useful. The thing that's most important to me is just what is useful for humanity and the world. And math is nice, but not useful. At some point, like kind of, you know, 17 dimensional spheres are probably not going to be the best career choice if you want to be useful. So it seemed like something that I could get good at, but also just the most important thing ever. And so it was a very clear choice. AI just like it was clear 10 years ago, it's just, it wasn't close and now it's close and clear.

Can you tell the story of how you got to fair? I think it is such an epic story.

Sure. I mean so when I started at 14, I had, I didn't really know how to program. I didn't get into programming out of curiosity about computers. I just wanted to solve AI basically. So after a couple years of just warming up on my own, I reached out to one of David Silver's PhD students who's the AlphaGo DeepMind co founder. And this PhD student, I guess at that point he was a graduate and sort of worked at DeepMind. I asked him if he could spend a year with me just every two weeks bashing my work, sort of trying to do so sort of like super speed up mini kind of PhD like experience where I could just learn how to do research. And I sent him this like giant email. You could print it out. I don't know how many pages it would be, but it'd be a lot of pages where I was basically just saying like, I want to build this algorithm you made in your PhD, so I want to beat this algorithm you made in your PhD here. Like here's a list of 10 ideas. I don't know if they're going to work and I think I need your help to figure that out. And then over a year we eventually got there and he was like, his name is Johannes. Johannes was kind enough to just bash me every two weeks roughly. And yeah, it was, it was brutal, dude. Like just like, because he, I was like, he hold me to the standard, you know, and I was like, like don't be nice just because I've been.

You were in high school?

Yeah, I was in high school. And, and then when we were done, I just graduated high school when, when I finished. So this is this when I finished the project that I was trying to get to with, with this. And then Noam Brown, who is obviously one of the best RL researchers in the world, reached out because he had worked on something similar it turns out. And we sort of had some ideas that were very similar and some ideas that were a little different. And so we just both published this and he reached out and then I got to work with no. 1 Brown for two years, which was great. And then that continued and so I got back for another year.

You were a high schooler? He was no. 1 Brown.

Well, I mean he published a paper called Deep Counterfactual Regret Minimization and I published a paper called Single Deep Counterfactual Regret Minimization and mine beat hiss by a little bit.

And so you want no.1 Brown as a high schooler?

I think I just graduated and I also. It took him like three months to write this paper and it took me a couple of years, but it. Yeah, I mean slightly. I'm sure he like would have come up with this the next day, but like sort of the gap between the two things, but it was, it was just. Yeah, it was like obsession is the right word. I just, I do things like 100% and yeah, so that was a lot of fun. Kept working in RL with Noam Brown for a while and yeah, so that's how I got to fair. Noam Brown worked at FAIR at the time and he reached out. I was actually, I was at university then and basically just worked part time as a researcher at FAIR while studying and anyway, so that was it.

That's awesome.

It was a lot of fun. Gnome is great. Like the Brainstorm ping pong sessions with Noam Brown. Dude, there's like nothing like this where you would just think there's like this problem and it's sort of like, you know, maybe you would like start a six month research. Noam and I would get on a call and we just discuss it and it's done. That was so great.

I love that. I love that. What makes him so great as a researcher?

I think it's a number of things as a researcher. Generally more from a meta level. He is fantastic at picking the right problems and then spending a long time just grinding to make it better and better and better and better. So he's very good at the whole compounding thing in research. Also making bets that aren't obviously the right bets when he makes them because he makes them earlier, I suppose, and making mistakes differently. So he's generally very good at picking problems and then attacking them consistently. He's also just very smart. I guess that helps. He works really hard. He used to do 100 hour weeks during his PhD. I don't know if he still does them. But he used to, he used to work really, really hard during his PhD.

I imagine he still is. Okay, so Noam arranged for you to become a researcher at fair while you were still a university student.

That was in my first semester, I think, or something.

So you were juggling that. You were juggling being a collaborator to Noam at fair, and then you became obsessed with yet another problem, climate change. And I actually started an NGO that is incredibly popular, Climate science. So you just didn't have enough on your plate?

No, it was actually too crazy. That's what I dropped out. I was like, this is crazy. This is too much. I can do two things. I cannot do three was like my conclusion after, after doing like, I did three months of that. That was terrible. That was awful. Doing like all three things because it's just like you just can't do well at three things. I mean, Elon can, but maybe I'll learn it in 10 years. But I couldn't at the time, so I dropped out at the time. But yeah, so I started ngo. I generally, I just think like charity stuff is awesome and hugely under appreciated. Like, it's, it's sort of like super high status to start a startup, but I think it should be like equally cool to start a charity. You're like helping the world in other ways. And so yeah, I mean, we started it as a. It's a nonprofit, but we started it like a startup. People were working insanely hard. We had clear objectives. It was, you know, a software product. Effectively, it was much more similar to a startup with the exception that there was no money in, no money out. Effectively. It's just very weird. But yeah, it's mostly volunteer driven or I guess is. I just no longer run it. But yeah, it was an interesting experience. You'd think I would learn transfer a lot from running a quote unquote company to running a quote unquote company now. But they are so different that I could. There was like no transfer at all between climate science and magic. Like a thousand volunteers, 20 hardcore engineers, no money at all. I can't tell you how much we raised because it's not enough yet. But you know, Giant, it's like completely different in every imaginable way. But yeah, it was a lot of fun.

So, Eric, Climate Science became an incredibly successful nonprofit. It wasn't just any nonprofit. What made you decide to kind of hand over the reins and hand over the torch on that and go start accompanying AI?

I just thought AGI was further away when we started it at all. I would never have started anything else if I thought AGI was so close. And once I realized it is, there was just like no other. I mean, my initial thing was always AI. That's what I did as a kid. I care about various issues in the world, but none of them are my unique calling in any way. I just, you know, hopefully be in a position to donate a bunch of money and whatever. But the thing I care about fundamentally is AGI. And it was like, oh, damn it. This is not 20 years away. So I have been running around with this AGI to do list, which is somewhat of a meme internally, because it's sort of like we're just going through it and we're trying to fix all these problems. You have seen it? Yes, we showed it to you. And I've been running around with a version of this. There's actually like 2017 or so. I was still in high school. I got. I don't know why, but some conference invited me to present my AGI to the list. It was wrong at the time. I was also sure it was wrong. But at some point there was one thing I just couldn't at all figure out. And I don't like blue sky research in the sense of like, just staring at a wall and trying to, like, figure out what the right question is. I really like to have the question and then look for the right answer when starting an intense project, because you need to know which direction you run in to really plan for it. And many things seemed clear, but it seemed completely unclear how to make these models reason in the general domain. And that became more clear with language models, especially code models. When I saw just some of the early results in this basic, I was like, okay, I know all the stuff from the RL world. I have a bunch of other thoughts. This seems great. We should just take LMS and make them do the RL stuff. It's a very simple kind of proposal, but I think that's sort of where, I mean, it makes a lot of sense. RL has been doing this for 10 years. It works in restricted domains. If you can make something work in 20 restricted domains and you have something else that works in a general domain, if you can combine them, maybe you get both the X and the Y axis and then you have your beautiful top right corner of the matrix, it seemed pursuable. And when something as important as AGI becomes an actually executable to do list, obviously there are still things to figure out, details of the algorithms. How do you make it efficient? Et cetera, et cetera. It's not like we do everything at all. Many, many things to figure out. But the direction was clear, so seemed like the right moment.

Okay, we're going to circle back to your AGI to do list later because I'm curious about it.

Sure.

I want to brag about you for.

Minutes because it might be wrong still, I don't know until we have AGI. It is a hypothetical AGI to do list, but we're trying.

I think the research field is tracking pretty closely to your to do list. I want to brag about you for a minute. I think you've been incredibly humble about your background, but as a high school student, you did catch Noam Brown's eye. And as you know, as one of his colleagues at fair, you became one of his top collaborators. Not even just one of one of many because there's such talented people that work there, but you're one of his top collaborators. And you know, when I speak to folks that know you, they just say extraordinary things about your capabilities as a researcher, your creativity, your work ethic. As far as I can tell, you work non stop. I think you texted me at 2am in preparation for this podcast, so I think it's safe.

Hope I didn't wake you up.

No, no. Thank you. Silent Month. Anyways, I think it's safe to say that you are one of the brightest minds of the current research generation already and will certainly be one of the legends that people talk about for the next decade. And so with that in mind, I'd love to ask you some questions of advice for aspiring researchers. And so maybe first off, you did it all from a very untraditional background. How did you do it? And do you think that what advice would you give to others in your shoes?

I can only really speak for the sort of profile of goals and person I am. I think I was lucky in the sense that I knew very, very early with 14, as we said, exactly what I wanted to do with my life. I had no doubt at all. And uncertainty can be paralyzing to a lot of people. I also had a very clear sense that I did not at all have a plan B. Like there was no other path in life that I would have been even like remotely above the neutral line on it had to be build AGI. Everything else is completely irrelevant. So, you know, I understand for many people, you know, a well paying job at Google is a great, great achievement. I mean, I would, if it's on AGI, it's fine, but you get What I mean, like, I just knew that there was nothing else I could do and like be fulfilled in life and look back when I'm 90 and be happy. So in a way like burning the boats very, very early gives you the opportunity to just be like, do things that you'd otherwise do 10 years later. Which again, even like I sucked at the beginning. It took me two months to understand the first paper. I tried to understand like I was terrible at programming for a long time. But when you're a teenager, you're like a decent researcher. You don't have to be great. Like you know, that gets you things, like a great mentor who then bashes you for a year, which was very, very helpful. And you know, then you get better and you're still young so you're sort of, your brain shapes more easily maybe, I don't know. So I feel like I benefited a lot from being early. But within that I'd say just like go for the end goal immediately. Doing anything, sort of like, oh, I'm going to do a PhD because I need a PhD to get it done. That's all bullshit. Like you don't. It's just completely bullshit. The other thing is like writing five page emails to people actually works. Writing like I get a lot of these like two paragraph meh things now and grateful I get emails. But I understand now why people think this stuff doesn't work. It certainly does when you're like here is how I'm going to beat your algorithm. Please help me. Five pages. At least in my experience, every single time anyone I want help from in this way was very helpful. So I suppose be proactive in seeking like the best people in the world to in a time efficient manner just distill their brain into yours and show them that you can make use of that. If you tell someone who's very good effectively, hey, I'm going to make good use of this. If you want to coach someone, I would love to be that person. They'll usually do it. They won't do it for 10 people, but if they do it for one or two, that's enough. You just have to win that seat, I guess. So that's been really helpful in my experience also just not shying away from learning new things. Again, I didn't get into programming because I'm curious about computers. I'm not very curious about computers. I just like AI and that computers are the thing that are necessary. So it's fun. I enjoy programming now, it's great. But I wouldn't have Gotten into it, I think if it wasn't for AI, but still you get into it. So don't be shy. We interview a lot of people who don't know how you'd implement an LLM and it's kind of crazy to me if you're a researcher and you couldn't implement charting or whatever, it's just insane. So really understanding the whole stack going down to. But sort of not bottom up, really top down, like, here's the thing I care about. This is the problem I want to solve. Okay, what do I need, what do I need, what do I need? And then all the way down. There are much more competent people at kernel programming and hardware design or whatever than I could ever dream up to be. But I understand enough of it to do better work at the top of the stack than I could if I didn't. So I think fundamentally you need to understand the domain you work in. It's also really good to just read everything I used to read. I don't know, I don't have a precise number, but just every paper. Every paper I would see, basically. And eventually you get so fast at it that that's feasible and you build a database in your head of like, oh, this is similar to this thing. But this was sort of my eye opening moment is. Or Bill Gates has to sit here like, oh yeah, if you learn enough things, they're all similar to each other. So it's not linear, it gets easier. And at that point I was like, I should read every paper. So thanks for the advice, Bill. Obviously this was like through a video. I never met him. But so I just started reading every paper. And that's really, really helpful because a lot of the best ideas that we had that work really well now at Magic were enabled by random things that are like, oh, it would never work without this random thing that I would have to have come up with in tandem. But because I have this database in my head, I can go like, oh yeah, like this. And then so often one good idea is enabled by three other ideas that others have come up with. And so it's always just like this composition of stuff. So having a large database is really helpful. Yeah. And then just never stop. Like never never stop. It takes like ages to do good stuff, to do good work. And at any point there was actually one moment with Johannes, the Deep mind research scientist who mentored me for a year in high school, where we had a version of the algorithm that wasn't very good. It was all right. And we're thinking Like, I should republish this. Like, we were both not really happy about it and he was like, close to giving up on me. It was like, well, you know, like, maybe this is just not gonna work. I wouldn't want to publish this. And so I was like, dude, fuck you. I'm just gonna get this done. And then we got it done like a month or two later. And so I think I remember going on a walk after this and just being like, can I do this? I don't know if I can do this, but there is no other option, so I just better get it done. And then I went back home and I started programming again. It was still, like, sad that day, but the next day was fine again. It just keep going. So I think you have to. I think that was a pretty formative experience because I actually wasn't sure if I could do it. And then we just did it like, super soon after. So I really haven't felt that insane level of, like, doubt and pressure since then, which has sort of enabled. I think it's actually beneficial. You have to be realistic, but you don't want to. If you stop you. Yeah, I mean, so anyway, so I think those would be the main things. Also be, like, really fucking honest about what you suck at to yourself, because otherwise you're never going to get good at it. Like, you need to search for the bad things and instead of, like, trying to. Actually, yeah, I think as a researcher, betting on your strengths is good only to the extent that you don't have necessary conditions that are completely missing. You can't bet on your strengths if they're not enabled. This is again, back to the engineering thing, for example. So, yeah, I don't know, I'm rambling, but stuff like that.

That's great. That is such a fascinating glimpse into the inner mind of what it takes to be a great researcher and behind all the glamour of training large models. And so thank you for providing that peak. And I'm really glad that you mentioned kind of reading every paper voraciously and having this database in your head, because one thing I've heard from your collaborators is that your superpower is understanding and absorbing new research. And so I'm curious, do you agree, like, do you think that is your superpower as a researcher or what? What kind of traits do you think have made you such an exceptional researcher?

So I think initially in the RL work I did, it was synthesis, where I would read every paper and I would go like, this thing plus this thing plus that thing with this Modification, I think that's what they would mean. That yes, is definitely very helpful. I think is a good way to do research. Generally there's enough work for synthesis to be a successful strategy. I guess to an extent still that I tried very hard after this is actually bringing this up, I realized this and I tried very hard to get better at leaps. Like coming up with totally alien crap that just there's no reference for it at all. And because ultimately. So if you take the transformer for example, right, like attention existed. The idea of stacking a bunch of LSTM blocks existed and you just had to remove the idea of recurrence really. And like a bunch of couple other things that were necessary, right? Residual streams, like the residual update and transformers existed from resnet. So it's like it's synthesis, but there is an amount of leap in there to make it all work. It's a little more complex than just taking components and putting them together. You need to come up with new things too, like the normalization and the D hat squared square root of the edge. Actually incorrect. But anyway, everyone now knows this, but roughly you should do some normalization. So there are some new ideas in there that really help make it work. But it's still a large amount of synthesis. So I suppose most good ideas are synthesis, but there are always some in the best ideas. There are some leaps and I'm trying to get better at those. But still it's mostly, I guess like take five things and throw away the stuff that doesn't work in them. Make the things work and configure. But yeah, I think some stuff needs leaps. But yeah, I guess no, that's the recipe. Like take LLMs, make them super efficient. Block context giant, throw RL on it, make it all work together. It's still mostly synthesis. I guess you're right.

Who do you admire most in the research world and what do you think those folks superpowers are?

Shazir. No, Shazir. Yes, he. What is his superpower? I guess to an extent, synthesis. He is. I mean he's. He's just the best at synthesis. He is also great at everything in the stack. He can. Like, he has no weakness really. Like he could implement the whole thing on his own if he had to run it. He sees the future. I think in a way it's very unconstrained and I think everyone's sort of crediting a number of the labs for scaling laws. This guy made a presentation where he was zipping through essays or completions or whatever written by models of various scale. I was like this is 100 million parameter model. This is a 300 million parameter model. This is a billion parameter model. This is a 5 billion parameter model. This is on YouTube somewhere. It's hilarious. And you're like what, what if we make this bigger? He's sort of presenting it this hilarious way. And then I agree, everyone was like super scientific about it. I think gnome is generally just, if I had to put it, it's very, very intuitive. I think they're like a lot of labs and researchers are sort of. And I think this is also. This is not a bad thing. It's very good. Are very evals driven, very mechanical. Right. Like they sort of very empirical in a way. Like Noam sort of just knows. He's like ah, this would work and then it works. So I think that's a superpower that is just extremely great synthesis. Yes. The larger. He has a larger database because he's been around for so long. He just literally knows everything. I mean he invented half of the stuff that everyone's doing now. There's no one who compares. I'd say there are a number of other people, I guess just you shouldn't fail. Out of all the people who are sort of the OGs of deep learning, I think you kind of hinted deserves by far the most credit. Just because he went through all the bashing when it was like, oh, this will never work. And they're training tiny, tiny, tiny, tiny, tiny things. They're like this will never work. And he somehow stuck with it. I think that level of grit and belief in something that is now obviously working deserves a huge amount of credit. Whether Capstone lets work or not. Whatever. Yeah, he like, you know, it's incredible to come to something like the conclusions that the world is at now. And if you look at some of the older papers, a lot of the ideas that are important now were in there already. So that's important. And I think he just deserves a ton of credit. No one Brown had the army gnomes. Noam Brown.

I should name my kid Noam.

It's a very good strategy. Yeah, it's a great strategy actually. I think 100% of gnomes that are like somewhat popular and well known in the research community are great. Yeah, no, he's also amazing. I mean a number of labs were working on what he was working on during his PhD and he basically soloed the thing and was way better and way faster than labs that put 10 people, including some really famous names on it. And if you just look at the paper track record. It's like here's the rest of the field and then gnome 100x is efficiency and then here is the rest of the field. That NOM does it again and it's just consistent. I think the consistency with which he has just bashed out these 100x multipliers in RL data efficiency and computer efficiency is crazy. Yeah. So yeah, the GNOME army is pretty good.

I want to go back to this concept of leaps are still needed in research and that you still have this AGI to do list. What do you think are the most interesting unsolved problems in AI right now?

Well, so a lot of it is solved now I think and the thing that remains to be solved is general domain long horizon reliability. And I think you need inference time compute, test time compute for that. So you'd want. When you try to prove a new theorem in math or when you're writing a large software program or when you're writing an essay of reasonable complexity, you usually wouldn't write a token by token. You'd want to think quite hard about some of those tokens and finding ways to spend not 1x or 2x or 10x but a million x the resources on that token in a productive way I think is really important. That is probably the last big problem.

Fascinating. The last one. Okay.

I hope so. I think it's reasonable to think that is the last big unsolved problem. I mean look over the last few years all of this other stuff got solved like oh, can we do multimodal things? Can we do long context? Can we do it? All this is gone. Reasonably smart models, you know, they're quite efficient now in terms of cost that I mean you'd have to be a reality denier to like not see what's coming. I mean this is just a, this is like a realization to a lot of people in the LLM space. But like RL has been doing this for like ages. So it's just like so clear that you need to do that. Or I mean maybe you don't need to. Maybe you can get away without doing it, which would be insane. But if you don't need to, it will still help you a lot. Like, like it's just, it's just like do I want to spend a billion dollars on my pre training run and then like a little bit more money on inference or do I need to spend $10 billion on my pre training? And I tried for you know, like 10 million would be great but I'm going to be I'm going to be. I'm going to prefer spending one.

And is bringing like the LLM and RL worlds together. Is that like a research problem? Like there's still like, fundamental, like unsolved science problems, or is that like a, you know, we have the recipe, we just need to do it and have the compute and the data.

I think there is no public successful recipe right now. There are good ideas. Like, okay, even if you take best of N, make N large enough, it's sort of, you know, it's not terrible. Yeah, the. So there are ideas. I don't know that the final idea exists. I think there's just a lot of room up from what is currently known. But there are ideas. See, I think it's very unlikely that even if you stop progress in research, we would not at some point hit something that everyone would agree is AGI is just that I think we can do better. And maybe it couldn't solve Riemann. Right. Maybe it couldn't do all these super hard things, but it'd be pretty good. And now I'm just curious, okay, what's the actual. If we did all the things, how good will it get? I think there is research left to be done and there are a lot of ideas floating in the world now. Everyone's sort of working on this, but I don't know that the current set of ideas is even final. It'll keep moving. I think.

Let's transition to talking about magic. Maybe just what is magic? You've been very mysterious to date, so maybe just share a little bit about what you're building.

Yeah, I mean, we're trying to automate software engineering. It took us a while to figure out how to train supergiant models. That's a pretty interesting engineering challenge. I mean, fundamentally, we're trying to automate software engineering from the product side. And a subset of that is a model that can build AGI. Because if it's a great software engineer, then it should be able to do everyone's job. At magic, we can do everyone else's job. That would be a subset. So the idea is that you could use this to recursively improve alignment as well as the models themselves in a way that isn't bottlenecked by human resources. And there aren't that many Noam Shazir in the world. If I had a Noam Shazir in my computer, I could spin up a million of them and maybe alignment would just be solved. I'm saying simplifying a ton and very idealistic. In the statement. I'm happy to turn this whole thing into a scalable oversight podcast if you'd like, but the core idea is like, okay, if I could just clone what we are doing into a computer and then press yes on the money button to run a cluster to do the work we would be doing next week, that would be phenomenal. So I think we're pursuing these two things in tandem where we want to ship something that's a good AI software engineer for people to use. It's, I think, going to be one of the first domains to see higher levels of automation. And I don't like talking around. I don't think the whole assistant pitch is going to last very long. Once these models are good enough to automate, there's just no way the economy is not going to do that. And I think everyone knows this and they're just like. They just don't like talking about it. It's totally fine. We used to all be farmers. We're not farmers. We're fine. Everyone prefers this. I think we'll figure our way out in the economy. If it produces the same or more stuff with less inputs, we should be able to figure that out. That's not a hard problem. And from economic principles, you just have to figure out distribution anyway. But that's what we're trying to do. We're trying to automate software engineering and as a part of that, automate ourselves in doing the work we want to do.

And so the reason they go after software engineering then is that is the kind of lever that allows you to automate everything else.

It's like the MVP of AGI, right? Like the minimum viable AGI, because then it creates everything else. We wouldn't train something like sora. Sora is great, you know, fantastic generic videos, awesome. It's just not interesting from an AGI perspective. If you believe that models can code themselves soon.

Totally. And so, out of all the companies that are trying to build an AI software engineer, you are probably the only one that is really taking a vertically integrated approach and training your own models. And that is either insanely brave or insanely crazy. And probably a combination of both. I'm curious, I know you love training models and so I know that's part of it, but why do you think you need to own the model to get this right? And how do you motivate yourself in kind of the David versus Goliath of knowing that OpenAI exists and has great people and cares about coding and is great at building models? Obviously. How do you Think about that entire dynamic.

I think you need, well, to build the best model. You need to build the model and we want to solve these fundamental problems. You can't rely on any. Like if the API guy solved it, then what the hell are, you know, we might as well start the company three years later. It goes to the point where we started, right? We started working on this stuff two years ago. So it took us some time to learn how to train these large models. Like it was really, I think it took OpenAI two years to get from GPT3 to GPT4 as well. And I thought we could be like much faster and this is going to be great. It's a pain. So it's definitely an engineering challenge, but it's necessary. Like, it's not like, it's not like we're doing it just because it's fun or because I like training models. It's, it's a massive financial investment that people trust us with that and it's not like it's one of those like one to one ROI investment. It's like if it's work, if it works, it's fantastic and if it doesn't work, the GPU is run and the money is gone. So like you're, you're getting a lot of people's trust doing that. It's certainly not something you should do just because it's fun and you enjoy it. Fundamentally, I think the value will accrue at, both at the, at the AGI and at the hardware level and never at the application level. There's no incentive at all to offer an API. If the API creates $100 billion company, you will just build that company internally. If OpenAI doesn't, someone else will. It's just incredibly unimaginable to me that that would be how you would build these companies in the first place. So from a business perspective, I don't think that's necessarily the right way. Maybe there's some partnership potentials you could like, oh, we'll get special access or whatever.

And then from cloud computing, right, there's been many, $10 billion, $100 billion.

I mean it's much, much, much harder to build Netflix and Airbnb and Uber than it is to build a chat interface. Like fundamentally magic is an application you press download on that we have a couple guys working on and it's just there, like it's not, you know, you can build this with like YC Pre seed money, the moats in. I guess I can just make the API twice as expensive for the next model and then launch my own product and then undercut every. It's. It's really fucked to not own the model in this domain. And in any domain that's going to generate a ton of revenue for a single company, in the case where it's distributed, maybe it's fine, but I don't think this will be. So it's necessary both for the market, which is good for us because the market is incentivized to fund folks like us, which it isn't in other domains. Like have fun writing like an email assistant. You're not going to get that funded anymore. So that's helpful. But fundamentally the reason we train our own models is because it's necessary for our mission. And I just wouldn't be interested in building like a nice little SaaS wrapper. It's just not like that's not every. That's going to happen anyway.

And I think though about competing against the 800 pound gorillas, like you've raised a lot of money, but some people have raised boatloads a lot more money.

And so as well some people have, have 100 million plus in revenue a year that they can spend. It goes beyond even the ones who could raise. Yeah, absolutely.

And so how do you, like, how do you motivate yourself to compete in that, in that reality the question is.

How much does it cost to build AGI? And not how much money can you raise? Because if you can build AGI for however much you can raise and you're having more might help you, but it won't get you there substantially sooner. Right. Like if you have all the right ideas and you get, you can build it with a certain amount of hardware. Like by definition, like, okay, if someone had like 100 times more hardware, would it be like computing that much faster or whatever. But it doesn't seem like a material advantage if your estimate for how much compute you need to build AGI is not as high as the revenue these companies could generate or the funding they raise is in fact much lower. So yeah, and I think that is the case. So it's not by any means accessible. It's very damn hard to get that much money. But it's not 100 billion. If I'm wrong, I'm wrong. And it'll be 100 billion and we will not have 100 billion and that's it. But if we can get to that point where we have AGI and a couple others have AGI and then the benefit of additional compute is there and you show an ROI it's like a reasonably even playing field in terms of additional revenue. You're going to bring AGI to the market, you're going to raise more on it. So the starting conditions of have this hardware is you need sufficient hardware, but you don't need more than sufficient. So that's a bet. That's not a. You don't know. But I think it's a bet with a high enough probability of being right that it is reasonable to compete in the space. And I think it is actually it is reasonable to think that like the ROI of having quote unquote sufficient funding might be better than the ROI of having like infinite funding early on.

And is there like an ideal for.

Investors that is not for me, for.

Mss, Is there an ideal team size for researchers? Is there a certain point at which you reach kind of like diminishing marginal returns of adding on the extra researcher?

So one of my biggest weaknesses, especially early on at Magic, was just scaling the team effectively. We were very single threaded on a very small number of people doing basically all the work. And I think we're getting better at that now. It's also you just need a certain level of maturity of your code base and of your research ideas and everything to properly segment them. So early on I would have said five for that time. Now I would say closer to 20. And I'm not including like folks working on other stuff. I'm including folks working on like the models and everything. I'd say closer to 20. I could imagine that in a few months I'll say at a slightly larger number. Especially when you get into large scale deployment, you really want to have very, very good processes around just having high reliability, availability of services that are detached from each other, et cetera, et cetera. So then you can segment even more, which obviously stuff we're working on now, but it sort of grows over time. I don't see it ever exceeding like the tens of people. And right now it's in the low tens, very low tension. But I don't know, maybe it's a skill to be able to utilize. If you're able to utilize 200 people, you're just a better CEO than I am. No, seriously, if you can, it's a good skill. And I think part of why I say a smaller number for us is that there is a ton of stuff we just don't do. If we built a video model, that would just be a separate team. They built a video model and that's more scaling. So to an extent we're More focused and that's why we're smaller. But also, if we could double the team and be twice as fast, I would do it any day.

Back in. Was it late 2022 when I first met you? At the time it was marketing assistance and email assistants were all the rage. And you were the first pitch that I heard that was AI that feels like a colleague. And I just remember that really sticking in my brain. So in some sense you've been thinking about kind of like agents, to use a buzzword longer than anyone else, maybe share your vision for that and what you think it takes to build a great agent.

Fundamentally, there are two tiers here, I guess three. One is useless, the next is assistant that you have to micromanage, and then the next is the thing that manages you basically where it's sort of more like a colleague. I think the layer where it's exactly even doesn't really exist because it's sort of this little thin point. Once the model is more competent than you are, you are there to give it guidance on what you want to be accomplished and answer clarification questions. But you'll never have to tell it like, here's a bug. I'm not saying that this is V1 of everything. I'm not saying this is V1 of our product, but fundamentally that has to be the goal. The way I feel when I talk to my best engineer, that's how I want to feel when I talk to Magic, where we have a discussion, he's almost always right and then he just writes the code and then someone else reviews it and then it works. That experience where my job is exclusively saying here's kind of what I want and then they help clarify even. Right. Like I just want here specifically that it should feel like that. And everything else doesn't matter to the user, like what tools the agent uses, how it works, does it run locally in the cloud, does it need a vm, does it have a browser? I don't care. Doesn't fucking matter. Our problem, not your problem. You care about your problems getting solved. So fundamentally that's what I think matters to customers. And everything else is dependent on the exact product shape, exact domain, exact everything. And like I'm stubborn as fuck. I just don't want to launch anything that isn't that. We will probably have to, but I just really want to get that thing like I want to talk to my computer, go and have lunch and come back and it built AGI. Like that's the. That's. That's the end Goal. Right. And they'll be, there'll be checkpoints, but. But that. Yeah, I don't think anything else matters. How you accomplish that is up to each individual company.

Yeah. How far away do you think we are from. From that? Or I guess maybe, maybe we'll break it down into a little bit more.

I mean we met in 2022. You learned how to extrapolate Eric's timelines. So maybe. Yeah, one and a half or double everything I say. But I think very soon, like very small number of years. I don't want to give a number now, but very small number.

Less than 10.

Oh, definitely less than 10. I mean way less.

Wow. Okay. Because I'm seeing some of the like the SUI agent stuff that just came out. They're like 14% on swe bench, which feels like, I mean 14%.

I just don't care about 14%. Like, I mean we, like, I don't know if 80 or 90 is good enough. Like, I think you need 99, even 96. I don't trust my computer. Like, I don't want to review the code if I have to. Like, the tier of product where I have to review the code is fundamentally different from the tier of product where I don't have to review and understand the code. And you're not talking about 95 when you don't want to review. You're talking about 99 point something. You're talking about whatever my developers accomplish plus some. Same as with self driving cars. So the difference with self driving cars is like you die if the thing crashes. And here you just have to review code so it's launchable before. But fundamentally you need way, way more. And usually the last few. Right. The nines are hard to get. So. Yeah. But no, I think you can. I, I don't know. I. I don't. People have, I mean, models have surpassed all these benchmarks. I mean just recently in a math benchmark. Right. Like way faster than even like the prediction markets assumed. And like I, I don't see that stopping. There's just too much like if, if everyone was stuck and I realized there's some perception in the public that oh, GPT4 is like, only like it's not getting much better. No.

Okay, we're gonna close out with a lightning round. One more of the answers. 1. What's your favorite AI app? Not magic.

Probably all the invisible ones still. Like my spam filter and all this stuff. The things that keep life working I think are still at the moment more useful than the sort of AGI like apps because if you took them away, life would just be awful. Like recommendation algorithms, whatever. I think that's really useful. Other than that. Yeah, I think whichever you're saying, other than, let's say other than the programming world, other than magic. I actually don't. But I'd say whichever model is currently best. It's a very boring answer, but I actually picked the spam filters, et cetera, the recommendation services.

First, what paper has been most influential to you?

I don't think this paper is relevant at all in the world anymore. But it was the first paper I ever tried to deeply understand or spent months on it and re implemented it and everything. And so it was most influential to me as a person, not so much to my current work. And the paper is called Deep Stack. It's one of those neural networks plus imperfect information game solving papers. It's reasonably complex for time. Yeah. So it's for folks who are interested. It's like nowhere near so to now, but it's sort of just an irrelevant type of algorithm. But at the current time, sorry, right now, back then it was useful. So that was very influential for me because it was just my first touch point with research really. I had no idea how to do research at all. And then I sort of just was like, I'm going to dig into this the way people like. People like hyperlink spam on Wikipedia where you like rabbit hole. I did that with this paper.

I love that. Okay, that's going to be my week in reading. Last question. What are you most excited about in AI in the next 1, 5 and 10 years?

Just what it's going to, how society is going to integrate with it. I think that's, we're getting to the point now where it's really going to impact over the next one to five years. It's really going to impact how society does stuff and beyond. Just, you know, another tab in your browser that speeds you up by some percentage on some tasks. I think it'll get much more significant in that time frame and ultimately you should like the only. I am not one of the intrinsic curiosity type of people. I know most researchers are. I really am not. I just care about the outcome and that is the outcome. So most excited for the outcome.

Eric, thank you for joining us again. Last time we recorded podcasts we weren't actually able to talk about the thing that got us so excited about Magic, which was you had shared with us your long context eval and our own kind of AI researchers had gotten really excited by what you'd accomplished on that, and that was actually what led to us investing in magic in the first place. So you just made some exciting new announcements around the eval. I was hoping you could share it with our audience.

Yeah, for sure. Thank you so much. Yeah, I mean, we've been running around with this hashes eval for a while, basically just being frustrated by the needle in hisak evaluation and everyone keeps complaining about it. And now that we've decided to announce where we're currently at in terms of our context work, instead of just blah, blah, talking about, oh, we have so many tokens of context, it felt reasonable to share the eval as well. I mean, we've used it in our fundraising, obviously, and thanks for backing us and generally just used it to guide our, our architecture development and our research. So, yeah, felt right to open source it and let others compare their architectures and their results with ours. And then it's exciting to share and thank you for having me back on to talk about it. Yep.

Thank you. Can you say a word on what's broken about needle in the haystack and what your eval does differently?

Yeah, for sure. You know, with needle in asac, basically what you're testing is like, find this weird thing, the needle, in this giant pool, pool of not weird stuff, the haystack. And so really all you need to be able to do to do this is to sort of take a little backpack and walk from the start of the context to the end of the context and find the weird thing, put it in your backpack and return it if you have to. Sort of implicit prior that this thing is weird, so you're more likely to remember it, which sort of means that you actually don't need to remember the whole context window, you don't need to know all of it. And so that allows some models, I would say, to sort of look like they're doing long context really well, when really it's not working as well. So we decided to just go the complete opposite, super hardcore mode, and just replace everything with random noise. There's no semantic information at all because it's just randomly generated letters, basically just hashes. And if you did something like needle and haystack in a pool of hashes, you really have to know the whole thing. But then what we do is we also do a hop. So it's not just you find this one thing, but you find this one thing and then you find another thing, as obviously you can keep that going. But those two dimensions, I think, really are important. Quantitative components of context There are other things you can measure much better in more domain specific evals. Of course we care a lot about code and so we look a lot at that too internally but I think from a general purpose context evaluation perspective and the reason we chose to open source diseval and only this eval is just that I think this quantifies exactly what you want to measure when you think about long context and everything else is sort of domain specific but yeah you want to be forced to remember the whole context window when you're talking about the context window otherwise is it really that big?

Yeah totally. I remember our own researchers were just blown away by the the purity of the of the eval and how well done it was and so thank you for what you're doing and thank you for open sourcing it especially in an age where long context is becoming more and more important. Congratulations.

Thank you so much for having me.

Yep, cheers of course. Thanks Eric.

Sam.