Why Openness Fuels AI Democratization | Dreamforce 2025

So I want to talk about a vision for AI that's truly open. And what I mean by openness is, is hopefully will become clear over the next five minutes. And I think this is a really pressing question for our society. Now there's a lot of confusion about what openness and open source means. The term gets bandied around. Um, so I want to set things straight first of all, so we all know about closed models. So these are models that you can access via API or through a product, things like, you know, OpenAI, Anthropic and Google. Um, and then people talk about open weight models. So these are models where the weights of the model are released. Often people confuse these with open source models. But in support to realize when you have the weights of a model that are released, that doesn't mean any of the data or the code or the training recipe is available. It's kind of like when you just get a binary file in software so you can run it, but you actually don't know how it was built. So now true open source software means, or models means that you have the weights, but you also see the code and the data recipe. And then finally I'm going to talk about a more radical version of openness, which I call open development. And in open development you see everything and you see in particular the process by which the model was built. And importantly, because you have the process that's open, there are people in the community that can contribute. So now this is a very foreign idea, I think in AI, it's a project called Marin that we've been building to illustrate how this idea could work. So I'm going to talk through how it looks like. So we leverage a lot of the ideas behind open source software, which is the one example where we have seen truly openness thrive. And so mirroring open source Software, we're using GitHub as a way to record every single experiment that is done to build this model. So whenever there's a problem with the model or someone wants to contribute something, there's a GitHub issue that's tracked it, there's discussion that happens and then the results are posted and you can see basically at a glance what is the state of this project now. We also have a wonderful discord community that's growing where live discussions on how the project is evolving, what data sets we should be training, what evaluations results is happening in life. So this is something that anyone can just join and then finally we post our training progress life. So this is a bold thing to do because as we know, often training doesn't go as smoothly. So here's an example. We are training a 32 billion parameter model over the summer and it was getting a lot of lost spikes. So these are things that result in instabilities. But we put it out there and we actually engaged the community to help us understand what was happening. And in the end we actually got now the best open source model. And you know, this is, and we're trying to close in on, you know, the Quinn open weight model. So this is a model that you'll, you can not only used today, but also a model where you understand completely how it was built. So if you think this is interesting, we are building this in real time today. Go to marin.community and check it out. So I want to take a step back and think about openness and technology generally. So we've seen a very successful example of openness thriving and that's open source software. So it really feels like the 90s today for AI. Today in AI feels like the 90s for software where remember Microsoft was a dominant provider and GNU and Linux were kind of scrappy projects that were coming just online. And it's really incredible and inspiring to see how such a grassroots effort and now is powering our entire Internet ecosystem. And so the challenge or the ask I think for all of you is think about how can we build the Linux of AI or foundation models together. Because I think there is a better way to build AI, not as everyone being just a consumer, downstream consumer of some closed product, but rather be a participant in defining and shaping how AI evolves for years to come. So with that I will now turn to our discussion and hopefully we can dive in a little bit deeper.

Thanks. And thanks for the very brief but very insightful talk coming from academia and also now working as an industrial AI researcher. I truly appreciate the importance of your efforts that you are making to make LLM training open to everyone. Now you cover sort of the degree of openness from open development to open weight models, which determines the replicability, community innovation, then also downstream adaptation, but also has risks. So the immediate question that comes is what about the competitive advantage? Does open sourcing undermine commercial incentives? You gave an example of Google or IBM in that 2000, but the same companies are now going close. Even Meta is taking the same route.

So.

So what are your thoughts on that?

Yeah, so incentives I think is the right question to ask here. There's multiple parties here. So the first is the easy case which is that if something that were truly open AI were to exist, this would benefit the vast majority of companies and academia and researchers out there. Right. In just the same way that a startup today can go and you build completely on open source software, get something that is focus on what they need, customize the software if they need to and build you know, entire huge companies out of that that is possible through because of open source software. And I, I think right now we are still at the very beginning we're trying to figure out what the right platforms and interfaces, whether you do reinforcement learning or what what inference looks like. This is something we think a lot about together as well. But I think my key goal here is to enable builders and creators to use these tools to create much more and not be limited in a way that you would be if you just had just an API endpoint. And I think I want to think about what we're building as really a platform, not just a model. Right. We're used to thinking about models as being the primary object of interest. OpenAI releases a new model, Google releases a new model. But really we should be thinking about platforms and tools because you start out by a model but as you get deeper as your company really thinks about kind of needing more than just kind of a prompted on your solution and you're sitting on let's say tons of proprietary data and you have user feedback now you need to actually do a lot more. And openness allows you to have these tools at your fingertips and give you full control over of your model. Now I think on the other party of is now okay, who foots the bill because training frontier models is no joke as we know. This is a great question. I think in open source software there's similar costs that have. It's not cheap either. IBM back in 99 invested $1 billion in Linux. And I think there could be a similar situation where a company or better yet a set of companies actually form a consortium which to power an AI platform that is not controlled by any individual organization. And I think that would be be great because then the vast majority of companies can benefit from this very reliable source of. And we know that AI is going to be very instrumental in every aspect of applications that we build. And so having a very clean and stable way to depend on it I think is existential. I think to many companies. Yeah.

So I mean coming so you sort of gave an example of together AI. So can you please enlighten us a bit more like what are the initiatives when it comes to open sourcing or open development that are ongoing in together AI.

So yeah, so together AI provides an AI platform for companies that want to really go deep on and leverage AI and this most cutting edge technologies. So what are the pieces of, of you know, AI if you want to build AI, the at the base layer there's compute. So you know, together we have a, you know, a large fleet of, you know, know GPUs. But on top of that, you know, it's not enough just to have the hardware, you need to also be able to have reliability, ease of use and speed and cost for, you know, I think of inference as a kind of a very basic building block. Right. So inference and language model shows up in many contexts, of course when you're responding to user requests. But it also shows up if you're doing reinforcement learning training. It shows up when you're doing the evaluation. So that's a building block that together, you know, specializes in. We get, you know, really well. And then you have other things. Now you have fine tuning. So once you have inference, now you have, you know, data. Now you want to figure out how to actually improve the models and not just serve existing models, but you want to take users, you know, bespoke data and actually figure out how to improve on them. So now there's a whole set of tools including fine tuning and evals that you need to build to support that use case. So, so the way we think about together is like kind of this full stack where all the way down from the hardware to these higher level services.

Okay, yeah. So my next question is somehow related, is that when it comes to the democratization of LLMs, and that argument is often criticized, misleading because all these models you need to run is like huge in size. So you need very high like huge compute even for only the inference. And when it comes to the talents, these talents are only available in the big techs or probably in the big organization like Stanford. So what are your comments on that, like in terms of this democratization aspect of compute and talent?

Yeah, so let's take the two kind of separately. So as an educator, as a Stanford professor, I care deeply about making sure as many people understand and can use and build this technology as possible. So you know, at Stanford we've been doing a lot of things. I teach this language models from scratch class every year, all the lecture materials and homeworks online. And many people have found this to be helpful. So the first thing is education. And as part of Marin, we hope to build a set of examples and use cases to basically take the knowledge that is either implicit or hidden or non existent and just put it out into the world. I think that is openness is a powerful way to democratize through education. So I'm very passionate about that. On the accessibility in terms of resources bit of course big models you can't run on device, you can't run even on your gaming, maybe GPU as easily if you have a trillion parameter model. But this is where I think a lot of the way we think about models and platforms comes in. Because I think about models as not just. They're just like passing artifacts. Right. They're not the enduring thing because every three months we have a new model anyway. And then once you have a new model, there's a bunch of techniques to transform the model into different form factors through distillation or other forms of fine tuning. And you can think about this idea of the model. The big model is just either a teacher but also just kind of a raw asset. And you could distill it into a smaller model and in particular for a particular use case. Suppose you only cared about, you know, document classification in the financial domain. Then you probably don't need the huge model to be activated. Every time you classify a document you can actually distill into something much smaller. But you needed a big model to, to have that level of capability and generality. So this is again something that we think about together a lot which is how models can be more fluid. You have a model and now we can either quantize or distill or transform into a different model which has better cost, accuracy, trade offs for you. Yeah.

So my next question is when it comes to the startups like Mistral or together AI or even Meta with Llama, so what could be their incentives to open source this models in the light of, you know, the competitiveness. So what are the incentives that you can see?

Yeah, so the incentive question again is very important, but also you know, tricky. So you know the, the obvious thing is that, you know, businesses are businesses so there's no open releasing of a model just for pure, you know, philanthropy. But there are cases where I think incentives are quite aligned. For example, at together we want open models to exist and we will, we have contributed open model ecosystem. We have this red pajama data set and set of models from a few years ago which kicked off a bunch of things. And that data set is used a lot in other model development. And this is makes sense for us because we want there to not just be a single model provider but a plurality of different model providers. And I think in general businesses where the way you make money is orthogonal to is not based on selling model access is where you're going to see a lot of, you know, for example, Meta's case, right? They need to build the models anyway so for their internal products. So you can open up the, the models to the community and, and then as a result, you know, I guess you get publicity, you get people to build on top of these, which then further improves your the models themselves. Now where I think Meta falls short and in the Llama case is that it's actually not that easy for the community to give back. So they're actually not capturing as much of the value from open waiting than if they had or if someone had a truly open source or open development effort. So for example, Pytorch is actually open source and they get a lot of value by having a whole community of people develop on Pytorch. Whereas Llama is much more one directional.

Okay, so let's move to the next segment which is about safety and risks. Open models allow broader alignment research, fine tuning and interpretability, but they also have safety implications. So alignment research depends on openness, but also alignment enforcement depends on control. And open models can unleash new capabilities, but also can be very hard to control. So what are your thoughts on having powerful models be open given risks like defects, cyber attacks, biohazards or even backdoor attacks?

Yeah, yeah. This is something I've been thinking about quite a bit. We've published this paper on this societal benefits and risks of open foundation models. One thing to note is that often this debate between open and closed and safety is presented as a kind of one dimensional scale which I think inhibits our ability to come up with creative solutions. To give a very striking example, one thing I didn't mention in open development is that since I had only five minutes, is that open development and open wait are actually orthogonal. So let's sit on this for a moment. So just like, let's take Meta. So Meta's llama models are closed development but open weight. So they don't say anything about how the model was developed. I mean they publish a paper, but that paper doesn't have all the, certainly not details about the data and then they release the model openly. You can have open development and closed models. So Marin is actually the perfect platform where we can experiment with these things. And what I mean is that you can see all the experiments running. You can see, oh, here's a model that's training and here's the data set that was used. But you can't by default go and just download the model or the data set. And certainly for data, because of copyright issues, we don't want people just downloading the data anyway without we don't have necessarily the rights to distribute that data. Now in practice right now all the models we're training in Marin are open weight because they're still at 32B level capability. But there is a path, I think. So if Marin gets to the point where we're trailing truly frontier models like GPT6 level models, I mean first of all that would be amazing. But we don't actually have to just if, if there is evidence that releasing such models would cause the world to end, then obviously we're not going to release the model, but nonetheless we can still train and do development on them. So the other thing I think about important about safety and open development is that open development is exactly what you want for safety. Because what you're basically saying is that look, world, we are doing this thing and regulators, people in the public eye can stare at this thing and, and sort of audit the process. Right. Often when you have a closed development approach, you have no oversight of that process. And then right before release there's some short red teaming exercise that happens and then the model gets released, which is actually not very much time for society to kind of react. Whereas if you have open development, we all go in with our eyes fully open. This is what we're doing. And we can invite people to academia, your policymakers, to actually characterize and understand the risks while the model is being developed, as opposed to after the fact, in which case it might be too late. Yeah.

So coming to that. So what has been your experience so far in your open source initiative as far as the community involvement is concerned? Do you get good support from the community in terms of making alignment, research and also safe and risk free of this model?

Yeah. So this is really, I'm really just excited by the fact that the community can engage and we've gotten a few really interesting thoughts. So there's this researcher from MIT who are exploring ways of ensuring safety through data filtering. So you filter all the data that mentioned is about dangerous capabilities and then you train the model and you show that the model has the general capabilities but none of the dangerous ones, which is kind of what you want. So now they're using Marin to test out some of these ideas. And the nice thing about it is that that research, assuming it's successful, will actually inform how Marin does open development. So here we see very bottom up, you Know, improvement to the Marin, you know, model. So people are voting with their feet. You know, another example is few folks from the community were interested in a multilinguality. So there was an effort to try to gather data sets and evaluations for training a multilingual Marin model. Again, vote, people vote voting with their feet, which I think is really kind of what open source software is or open source in general is about.

Yeah, so I have a follow up to that. So if an open model causes harm, who would be the responsible people for that? Is it the downstream people or is it the upstream people like who are making applications or the people who are open sourcing it or Even making the APIs available of these models?

Yeah, so this is, this is a good question, you know, basically who is liable for damages? It's you know, obviously a general question that I think predates AI and I think in general my view on this is that having the liability be downstream is in general, as a general rule makes a lot of sense because that is where you have the context to even understand what the harm is. I think upstream there these are general purpose technologies, right. You can have a model and you can do all the safety training you want on it. But you know, people can misuse tools just like people can misuse like hammers. And so it is I think a mistake to kind of over regulate the upstream. Now that doesn't mean the upstream developers are completely, you know, have no obligations. I think they actually have a very strong obligation but that obligation is more to ensure that whatever they build is carefully documented. It's, it's that downstream users have a means to understand what they're getting because I think a lot of problematic cases are not necessarily from misuse but just from accidents where, oh, I thought this model was not going to hallucinate or it's going to be not generate toxic content and yet it or not generate vulnerable code and it happens to do it. So I think the obligations of upstream developer is to do, you know, much more, be much more transparent about how their model was developed, evaluations on the model. So at Stanford we have this project called the Foundation Models Transparency Index which evaluates different model providers based on how well they're documenting their processes. And in general open way model providers are better, but they're not always because just because you open weight release an open weight model doesn't mean that you're actually very transparent about what happened upstream.

So my last question hopefully yeah, we are just at time is about the geopolitics of AI race Should frontier models be open in the world of this strategic competition where there are concerns regarding national security and also economic competitiveness? What are your thoughts on that?

Yeah, so there are two aspects of this. One is basically competition in AI and the other is more national security risks. If you look at the current landscape, it's, you know, it's clear that, you know that open models is a, is a very important kind of play. If you are a country and you wanted to be dominant in AI, I think you have to be dominant and open. You know, way to, at least if not open source AI. As we can see right now, there is a lot of models, open models from China that are actually just getting used in the US and so I think that there's a huge gap. Even though the US still leads enclosed models, we are lagging behind in open models. And open models is actually critical from a national strategic point of view because these are the things that flourish and are adopted easily. And you can imagine that the entire world can be built on open Chinese models. And in the US we want to have open US models that the world can build on in terms of national security risks. I think this is where some of our earlier discussion around open development and you know, and you know, maybe gated or careful release of the weights is actually the right solution here. And again, I'll stress that open development is, gives you all the goodness of transparency and audibility without having the associated risks. And then it's really a case by case decision to determine whether you want to release a model weight openly. Frankly, I think all the models that exist, if they were open weight would be totally fine. I don't think this is actually that even that controversial. And I think there is sometimes a, you know, a lot of these discussions are not super grounded in the type of risks that are being, you know, mitigated. And so having better evaluations and a more rigorous understanding of what are we, what is the threat model that we're trying to defend against will help us make better case by case decisions.

So I think we have just one more minute. Maybe I'll take that opportunity to ask one more question. Okay, so now that the LLMs are becoming more like from simple chatbots to action models and also doing the autonomous event decisions and performing actions. So now I think the risks are even more with this kind of action. So what are your thoughts on how we can make these models more safe?

Yeah, so the genetic setting is both exciting because we can do so much more if you can take actions in the world, but also frankly, it can be terrifying if you are completely careless. And right now it is like the wild west. It's right early days of Internet where security was not really. It was kind of an afterthought. So I think the right way to think about this is in some ways it's not just an AI problem, it's a AI plus systems and kind of product question of how you can build these systems where you understand what the contract is, what is the permissions or security model for this agent, what kind of actions can it take? What is the privacy model? There's cases where agents are just language models can if whenever they look at text that's coming from an external source, that text can jailbreak and poison the model and make it do very weird things. And just understanding the limits of current models I think will help us build scaffolding around it. So by default I think it's fine. Sorry, not by default it's fine. There's always a way to make it fine, right? Which is like don't give the language models ability to do anything. Just assume any action that it can take, it's going to take in the worst possible way and now build so you are then secure. And then now we can try to find ways of augmenting that set. So I think a lot of this is a joint work between folks who are evaluating AI and understand what it's, what it can and can't do, along with, you know, system designers who are creating the sandboxes in which these agents live.

Okay, so I think we are just over time. And that actually concludes this session. And thanks Percy for having this nice talk and for having this nice, nice discussion. Thanks again.

Okay, thank you Sophie. Thanks everyone.