Safely Scaling Artificial General Intelligence with Reid Hoffman & Sam AltmanI | Greymatter

This episode of Gray Matter is in partnership with OpenAI, a research company whose mission is to ensure that artificial general intelligence, or AGI for short, benefits all humanity. AGI refers to highly autonomous systems that outperform humans at most economically valuable work. OpenAI co founder and CEO Sam Altman and Greylock partner and OpenAI investor and board member Reid Hoffman talked with the OpenAI team during an all hands meeting about the factors to consider when releasing AGI publicly. The societal shift that will need to happen. The meaning of consciousness and collaboration with other nations.

Today we have Reid Hoffman. This is the first board member. Come talk and answer questions about how they think about AI and OpenAI and the future of the world. Reid needs no introduction, we'll just jump right in. So I am curious how you think about what the world looks like, not even post AGI, but just as AI ramps up at this sort of narrow but strong AI over the next couple of decades.

Well, obviously the smart answer to that question depends a lot on what the speed of the ramp looks like. Because if you say, well, we think we're on exponential curves, the modeling and insect going to a person, or the notion of that the primary driving variable is the number of billions and trillions of parameters that you intersect with both larger farms, better compute infrastructure and everything else, and that all looks exponential. And you say, well, that intervening state is actually in fact more of like a lightning strike than it is a iterative thing. But on the other hand, if that progress is more kind of like interestingly linear, not so much because you don't have some exponential curves, but because the functions and the things that you're able to do have kind of more of an asymptotic vis a vis the compute infrastructure, then you have a much more detailed and rich difference. And so obviously the speed of that curve really makes a difference. And I think actually anyone who claims they absolutely know the speed of that curve is out of their mind. The people who say between those two answers, it's definitely this one or definitely this one, both of those are clearly incorrect statements. It's like we're learning and we're observing.

What would convince you that it's either going to stay in the exponential or asymptote at some point? What are the early signs of data you look for there?

It's easier on the asymptote side because what you'll see is the kinds of functions that you're trying to compute are more resilient. Like even though you went to another 10x in parameters, the actual performance increase wasn't even 10x, it was like 2x or 1.5x or something like that. If you see that, then that's a clear signal. The other one, all J curves do become S curves. If not, then the universe would be this really wackily different place. So the only real question is, where does the S curve occur? And if the S curve occurs, there's no reason to say the S curve occurs at human level intelligence, there's no reason to say it occurs below, no reason to say it occurs 2x above. It's where does that S curve actually, in fact, actually hit? And it's hard to predict where those S curves until you start seeing what the things are now. But if I were to say what would be things, that would say, well, maybe there's actually a pretty strong chance that exponential curve will continue from insect to mouse to, you know, dolphin. I don't know, maybe I'm undercutting dolphins. This kind of question of. All right, as we begin to see, we're counting on a set of both development and emergent properties from the size of the parameters, you know, one shot, learning, transfer, learning, kind of various forms of symbolic and planning reasoning. Get into it. What are the parallels for cognitive development from children? As we begin to see those major research tasks checked off and actually developed, then you say, well, the exponential curve will take us there. So it's more that. And I think the wise thing is to kind of like bet that both are possible and then continue, which is one of the reasons the OpenAI mission is so important.

Do you have a strong feeling about whether consciousness will just be some emergent property of large neural networks and learning or some fundamentally different thing that we're not even approaching?

I'm going to answer this question by going back to a little bit of my past. So I majored in this thing called symbolic systems at Stanford, which is kind of cognitive science, artificial intelligence with a focus on more of the old school AI, like, how do we encode reasoning and knowledge and so forth within a symbol system? David Rumelhart, for anyone who knew his early PDP work, was one of my two primary advisors, along with Terry Winter undergrad. And the thing that I concluded was that the problem was no one knew what thinking was. And so they were like, oh, we're going to build artificial intelligence. It's like, well, what's thinking? Well, it's processing these symbols. And I'm like, well, that doesn't seem right. And so I then thought, well, okay, maybe philosophy knew this. So I went to Oxford to study philosophy. And I realized philosophy didn't know either. And so I kind of went, all right, maybe this is not a project for my lifetime. May turn out it is a project for my lifetime. But then that was 1991, and I was like, all right, let me go do something else, like, you know, software, entrepreneurship, and these kinds of things. Now, consciousness, I have a very similar thing. People assert, this is what consciousness is. And I'm like, I don't think we have any clue. Right? Like, it's just a little bit like, for example, if you actually look at the way that we actually. You say, well, we use language very well. We're communicating. We think we're having a communication here. How do humans really use language? How does that get, you know, function as part of the biological system we're in? What is the relationship between the symbol processing and things? We don't actually know any of that. We have a bunch of interesting theories and models, but we don't know consciousness. We know even less. And so people who say, well, this is what consciousness is, and machine can certainly do it, or machine can't certainly do it. Well, first, let's start with what it is. And we know that we experience it, we know that we embody it in certain ways, but language can mislead us in the following ways, which is, it sounds like a coherent question, and yet it isn't. And so, like a classic kind of medieval example is how many angels fit on the pinhead of a needle? And you go, well, it sounds like, you know, you have these nouns and you have the verb and they put together, and it's a spatial representation that generally makes sense. And of course, the question's bonkers, Right? Well, some of the questions around consciousness may be similar because we actually just don't know what it is. And we get misled by language in some interesting ways. So it's kind of a maybe, like, whoa, maybe the earth's conscious.

Right. Let's say we never get to any of that, but we just make incredible narrow AI and sort of most human jobs from today, including ones that we think of as creative, get done better by machines. What do you think the societal shift looks like if that happens very rapidly? And how should we be thinking about our responsibilities there?

So this is one of the reasons I think this is a super important project. I've been talking with you from the very earliest days in this, and. And I think that it's obvious, like, where we play between just the development of the current techniques and essentially where we are between the linear development and just saying, look what we see down the road, line of sight from where we are on the tech development we're doing and the possibilities of AGI or even superintelligence. Then you say, all right, what we know from the line of sight is that just about every industry, maybe every industry is going to be transformed in fairly significant ways. And those transformations will. Some of the cases they're already started, other cases it'll be five to 10 years, but there's almost no industry. You think in 10 years the nature of how that industry operates won't be substantially different. Even if you say, well, we don't actually in fact have generalized robots for manufacturing, the way that it's integrated into various forms of manufacturing, quality assurance and everything else will be changing because we can iterate on the software patterns and cameras a whole lot. And obviously there's a whole bunch of different things. When the effectuators and the question of moving from simulation to reality in various different contexts, there's a general context which is kind of super hard, but the specific context will be fairly straightforward. So what that means is it'll transform everything. Now then the question comes down to when you have this kind of free floating transformation, what can happen? Andy McAfee is a friend of Sam's and mine has a pretty funny phrase which is if things keep going the way they are, the humans will rise up before the robots do. And so the question about how humans treat each other, how humans deal with this is actually non trivial. And so I think one of the things that's super important is how to have the bulk of people feel like that they have a good role in the future in front of them and, and it's something that they want to be part of and something they want to work towards. You can have obvious dystopic things like if there is one megacorp or one mega nation. Technology does tend to enable centralized power in interesting ways. That is, I think most technology tends to enable centralized power. And so then it's really important to say, well, how do you make that centralized power something that's positive for humanity? And how do you actually keep a certain amount of decentralization in, in autonomy and freedom and ability to do personal development across that. I could kind of spin lots of different, both utopic and dystopic narratives about what it could look like. But I think the really key thing, and this is part of how when you kind of looking through a fog over a minefield and saying what's the Path, what does the future look like? You say, well, what are the fixed points that change the probabilities from dystopic outcomes to utopic outcomes and then focus on those fixed points. And, and so one why OpenAI is so important is to say, all right, well we know this AI technology is going to be super important. Is there any place that takes as its primary mission humanity and the public trust as its outcome for what it will do and what it will distribute and who will empower the technology? We need to have an entity like that that would obviously be in some ways good if you had smart governments doing that too. Although where the governments will play in is kind of unclear on this front.

As a board member, under the what scenarios you'd vote to release an AGI into the world? And how would you think about our responsibilities there and knowing that it'll be safe or good or anything like that?

Well, one thing that's kind of interesting that I think we haven't really yet fully explored, partially because we're on the early side of this exponential curve, is the ability of how these decisions will be affected by introspection, being able to look under the hood. And obviously it'd be super complicated. But how could that add to your knowledge set? Because roughly speaking, I think your targets are that you'd want something that is essentially symbiotic with humanity. You want something that says helps us be better, we help it be better. There's a good healthy dynamic between it, whatever that shape is. If you said it's a complete autonomous agent, it's another form of life, roughly speaking. One of the things I've started thinking about is how would you measure, like something being a Buddhist, like how would you say, well, I think all sentient life is good. Sentient life is good. And that thought in that way probably leans towards symbiosis. And you know, one of the things that you and I have talked about is a lot of the paradigm for thinking about how to train these things right now is within competitive games. Well, maybe we can also figure out how to also add in the training paradigm, cooperative games. Because in cooperative games, and you say, well, fundamentally when we think about which human ethics we really like, we like the ones that say, I like non zero sum systems, I like systems by which we're all winning and we're all evolving and our future looks healthier together, and that's what we both want us to be and want to be. In symbiosis. It sounds vague partially because like there's just, you know billions of possibilities, what the configurations are. So it's general principles versus kind of like a heuristic.

I've got more questions which I'll get to at the end, but I want to take time to open it up now, so I'll repeat questions for the recording. How do you think about interactions with other AGI lab?

So one of the things that I was doing before OpenAI became a concept was that I had concluded that the primary drivers for the future of AI were a set of commercial labs. And what I was doing is systematically going around and meeting the leadership of all the labs and trying to get them so they'd have conversations with each other and they say, look, we should be completely collaborating on things like safety, on what the issues are. And so, yes, there's some competition, there's competition for talent, there's competition for who got the interesting result or paper first and that kind of stuff. And that'll happen. That's a human nature. But let's try to say we are in some significant ways in this mission together. So that was kind of the way that I was operating, because I think the lines of communication and the lines of where in this grand mission for humanity are we actually collaborators versus competitors is extremely important. And I more or less put a ton of time and energy into that exact effort. And so I think that's what's important in the relationship between them.

How should we test for that with other groups?

You mean, eg, do they have the same mindset?

Correct.

Well, so the way that I typically do this is that I make sure that I don't ever just rely on my own point of view. I try to get as many smart points of view as possible and try to integrate evidence across it. I don't just talk to the head of a lab, I talk to a bunch of people around it. I talk to people who are involved with it. I try to see what the result of the work that's coming out, what questions are being asked. Because usually the questions that you're asked are revelatory of what the real mindset is. For example, when I came out of the Puerto Rico conference, one of my conclusions was that we needed to be working on safety a lot more vigorously. And so as part of the whole Stanford Human Centered AI launch, which I'm the chair of the advisory board, it's a wonderful effort. It'll be collaborative. We're just having a conversation about the real interesting substance. But one of the things that the Stanford folks were really excited about, that I said, hey, look, we should do this. And what if we took the people who were serious about AGI safety and did a event at Stanford every six months? Right. This is partial answer to your question is, well, did you send the smart people who are really working on it? Did they come really thinking about it? Did they throw bold ideas on the table? Are they collaborating with that? Then that's a positive signal, right? And if they don't do that, then that's a negative signal.

So one specific thing we've done as an organization is say that if a different organization gets within two years of AGI, with three 50% probability, we'll just go help them and not have a race. If other orgs don't make a similar commitment, how do you think we should view them?

Good question. And now, of course, we're getting to questions I haven't asked before. I think it would depend a lot on what we think that the governance and output would look like. So I can imagine frameworks in which I would go, okay, I'm irritated by that, but I'm not nuclear like I think you should. I think it's a good commitment. And I can imagine frameworks where you go, nope, now is when you actually go nuclear and say, look, we have to talk about what this means for all humanity and we need to get humanity's lens and focus. What's scary to people is other people. And so, like, if people are doing the wrong things with it, that would be like a potential disaster.

Just to push a little bit harder on that, how specifically would you determine which of those two cases it was?

I would love to be able to have an answer that was completely determined. I don't think I would. I think it would be a question of judgment. One of the things I like about the commitment that you've talked about and said to OpenAI makes me proud to be part of the organization and helping because I think that's the right idea. Am I putting my power down? The right thing to think about or you want a real mission is I am a servant to the mission. So I am a servant to how humanity gets better. And if the view and the test of that organization is to say, look, we may not have agreed to do this, but we are agreeing to the way that we're operating is we are going to tangibly and provably hand over and commit to it, not just being under our own power, then I think that would be the kind of signal I'd be looking for.

So how do you see your role on our board and how do you think about your unique value in making sure we do or don't do something in the coming years?

I try to only be on boards where I think I can add kind of unique value. I haven't yet started asking for those measurements, which I do ask for when I'm doing this to make sure that I'm contributing the right way. And part of that is a combination of my own knowledge store, my own expertise, my own decisions, my own value system, the things I bring to that table. But more importantly, it's the network around me. So it's, oh, we should go talk to so and so we should build the following alliance with this individual or this organization. We should cross check this. There's this interesting work going on over here that we should either learn about or we should collaborate with or oppose if it's the wrong kind of thing. And it's that network of people and organizations and resources that I think is part of both helping the right outcomes and preventing the wrong outcomes. And so I'd say some of my contributions probably on the board is that generally speaking, people have a reasonably good sense of where my kind of ethical compass is. And the fact that if I'm going this is really important and this is something we should get behind and brings the network more to bear. And, you know, I'm not shy about if I think there's something wrong kind of talking about it and trying to do something about it. I haven't seen anything that I think is wrong in the OpenAI context. I mean, I had some discussions with people who said, well, we think open should be complete open source right now. And I was like, no, that's a bad idea from a safety perspective. And they're like, yeah, but it's not as transparent. I recognize you would like that transparency, but the transparency doesn't just go to you, it goes to everybody. And that could have bad outcomes. And so we're paying attention to the bad outcomes. I hear your point of view. And by the way, transparency is generally good value but needs to be selected. And now it's here. And so that's the kind of thing that I would say is, as a participation, is there anything you would add to the answer that you would be expecting?

No, I think that was great. Just to echo the point about ethical compass and network. That was part of why I felt so strongly about adding Reid to the board in addition to investing, as I think he sort of delivered again and again on both of those. How do you think about collaboration once this is a game with a lot of state actors involved.

I'll start with just a humorous comment because I think it's fun in this context to repeat it, which is I was asked on a stage what I thought about what the future of the free world was. And I said, well, I'm a strong supporter of the leaders of the free world, namely Trudeau and Macron. So obviously, I think my political position is not a surprise to probably almost anyone in the room. I think there's good things about the fact that US has leading progress in this, because I do think there are other nation states which would concern me a lot. Obviously, the dialogue within the US tends to be China, but I also think Russia and other places have fairly deep concern. And roughly speaking, I tend to break very tightly between an autocracy and essentially a reasonably run democracy. Again, you know, looking at ourselves is kind of like, oh, where are we on that spectrum right now? And so I tend to be very positive. I've actually had meetings with Trudeau, two meetings with Macron, basically all on artificial intelligence subjects, trying to be positive about how do you steer between utopia and dystopia? What are the kinds of things they could do both for the nation state, but also for humanity? What are the ways to look at it, what kinds of things to do? And so I think it's a positive thing to do. The real important thing is you have smart people who are listening very carefully to these things and making sure it's included. So I'm super positive on it, but it doesn't necessarily mean that you have to create a world foundation in order to make it happen.

On the general question of state actor involvement over time, do you think we are more likely to get to a good outcome if AGI comes very quickly or it takes longer, and then you definitely, you're guaranteed sort of state involvement.

I would think if I were to pick one, and like, I really try to not center too much given, like my earlier answer to consciousness, like, when you don't know, you should really just say, I don't know. Totally agree. Right. And so, but if I said, look, I'll pick one, I think my ideal answer would be call it not immediately, but soon. The reason not immediately is because I actually think our notions of how do we do safety and symbiosis are still very frail and weak, and I would like to get them better. And so I'd like to work pretty intensely on that. And then soon would be better because it obviates race conditions. Like, if I were to be able to wave A wand. That would be the segment which I think would be better.

How can the team help the board?

So one of the key things about an experienced board member is they go, I learn first, right? Like I go, okay, here's what the game looks like, here's what the organization looks like, here's what the state of thing is. And then specifically when you go do stuff, you're intervening in a way that you think you have a very high probability of actually having a net positive outcome.

What are the important things OpenAI will do differently given a primary motive to benefit humanity?

Nonprofit I'm a venture capitalist. I like profit, right. Just to be clear, I'm not anti profit. You can almost look at one of today's issues in techlash and have an immediate answer this day, which is, should the focus of AI be to essentially keep you enwrapped in as much attention to your screen as possible while you're getting agitated and angry. Because by the way, that's a positive economic result because I get a bunch of time on sign, I can sell a bunch of advertising against it, et cetera, et cetera. And the answer is of course not. And so when you're saying, hey, I'm not actually just tuning for what is my revenue growth look like and I have to maintain a 20 plus percent CAGR on a huge number and I really care about the operational property and that's a hard thing to do. So all I'm doing is tuning against that. I'm using AI techniques for it. I'm not actually thinking about my impact on humanity. And so for example, when I'm asked about which I am, probably every public thing that I'm doing is like, well, is it time for tech to be regulated? And it's a very deep, detailed kind of question to kind of go through. But what I say is we'll start with a dashboard that you'd want to be measuring it to. And so for example, if you get to social networks, you say, well, what's the things you'd want on the dashboard? And like, one of the things I think you'd want on the dashboard is let's figure out using AI in this case, a humanity positive way, which is sentiment analysis. Let's figure out how we're tuning the knobs in ways that actually lead to more fruitful conversation where people are actually understanding each other, talking to each other versus calling each other, you know, grabbing the pitchforks, right? And going for each other and have that as something on your dashboard and have the AI lined up against that when you're having product managers tune it. And that's the kind of question that could come out of a very thoughtful corporation, but certainly comes out of people who are more caring about society or humanity.

What problems are you most excited about AI solving in a post AGI world?

Well, I'm not really sure what a post AGI world looks like. There's a bunch of different ways that human nature is described. One of the ones that I like that is not frequently done is homo ludens, which is game players. But there's also homo narrativists, which is we tell ourselves stories, and then we get very tied to the stories. And so now in a post AGI world, it depends a lot on what the AGI looks like and depends a lot on where the AGI is and is playing. Now, what I would hope all the way from now through then to post is that the really key thing is trying to figure out how do we increment the likelihood of this kind of the symbiotic relationship and how do we know what's going on? Because the paradigm for these things is learning machines, and learning machines means they're dynamic. I actually listened to very smart people say, well, we'll just program the machine to not do what we don't want it to do. And you're like, well, that's not the way of working. So it's a learning paradigm and a dynamic system. So that means we have to be learning and adapting with it as we see what happens as the system evolves and grows, and then we have to be adjusting with that. And that would be roughly where I'd hope OpenAI would be, which is we're as deeply knowledgeable about these systems and possibly in creating them as anyone in the world. And our primary focus is saying, okay, these are the kinds of things that lead to the general kind of expression of what we think of as the upside of humanity's potential. Compassion, wisdom, cognition in some cases. But let's start with the value side of it. And that's the thing that I would hope OpenAI would be. Frankly, if there's other people better than us and they're better than us because they're just great, wonderful. It's great for the world. If they're better for us because we're weak, well, then let's get stronger.

What can we do to have a broader perspective that represents more humanity than just us in San Francisco?

So the fact that you're asking the question is, I think, a healthy step in where I think the right place to be is on this question, which is I tend to be not as favorable as some that say the only way that you can include a kind of call it a minority race, transgender perspective is if you have one of those people at the table at everything you're doing right, and that's the only way you could do it. And I was like, okay, that's not true. It doesn't say that it isn't useful and isn't important. But I think that all projects happen with small groups. They do actually happen with small groups who have a cluster of things around it. And if you try to go to the UN where you kind of have everyone, it doesn't operate very effectively. And you need to operate effectively. But what you need the group to be doing is saying, I am deeply worried that there are perspectives that I'm not including and I need to be figuring out how to include them. I need to be talking to people. I need to be figuring out what the concerns are, what the issues are, and I need to be including them in the small loop of what we're doing. So I think it's critical to have that as a value and as a principle and as an activity and as something you're investing in. We need to understand Africa. We need to understand emerging markets. We need to be understanding places that are have more challenges in governance and resources and how that plays into our work, given that we have global impact and perspective. But that means that what it is, is through always asking the questions and always trying to get people to contribute their insight to how we're making the decisions. And there's trade offs in these decisions too. And so what I'd say is I'm super sympathetic to the question. I think that the value and the statement activity is critical. But the usual answer is to say, like, for example, let's go pick X people to make sure that they're in the project. And you have to have one of every single variable that you think is at least one and usually more of what's important. And that's actually not the answer. Now, of course, if you say we're all white men that are all in San Francisco, we're going to have cognitive blind spots that are like a mile wide, 10 miles wide. And so you got to have more than that. It's kind of somewhere in between in terms of how you, how you play.

If we succeed in creating AGI at that point, how do you think we should use it to generate some amount of profit?

So my actual hope for the mission is that we will generate some profit even before we get to AGI on things. Because part of what makes our ability to invest the right way in the mission is to actually have a compounding loop of economics. It's one of the things that makes the commercial business stuff work. So part of the reason why I was kind of one of the central players and forces in helping the LP fund come together, it wasn't like, ooh, the chance to make money, great, always happy to make money. But it was the no, no. We need this kind of compounding loop to help fully realize the OpenAI mission. Because while humanity and our service to humanity is the absolute top goal, money helps us get there, Compute talent, all the rest of the stuff. And so a compounding loop of that really matters. So how do we take our best shot at setting that up such that the commercial loop actually is in service to the mission?

This actually gets to one of my last two questions. Do you have any strong feelings yet about how much money you think we're likely. It's a two part question, how much money we're likely to need to build? AGI given conversations you've had with me and others, this is one of those.

Questions where you say 42, I would guess it's between 5 and $50 billion.

And given that kind of a range, how will you think about what investors you would and would not support us taking that level of capital from?

There's two parts to that question, actually. One is how does taking capital change governance? And that's the one I'm most urgent on, which is governance stays in focus of service to humanity and understands the commercial loops are important for that. And by the way, people who are enabling that should win from that too. But it's the mission first, right? Like first, second, third, it's like that's really important. So it's the governance structure. And then you begin to say, well, what's the governance structure? And then how does this actor play into the governance structure? Both, sure, we set up a certain legal structure, but once you start having investors, they can start being influencers on governance separate from just the fact of whatever the legal deal was. And so you have to be concerned with that. And then as part of the mission, we have to build enormous trust. We have to have people feel like, look, you really are cross checking perspectives and thinking about it. You really are trying to do this with the mission of humanity first. And we trust that's your intent and that you're competent at doing that. So I think that the question is, where in that building of trust for also who we're allied with does the world go okay? We recognize that sometimes you make some compromises at scale and so forth, but you're clearly staying to your compass and your ethics in both the governance and who you're choosing to partner with.

Thank you all for coming, and it's a pleasure to have you on the board.

Yep. And thank you. I'm happy and proud to be part of the team.