Point-Counterpoint on Open Source AI

Today on the AI Daily Brief, another argument for Open Source AI innovation The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show Notes. Hello friends. Once again, I am traveling currently and so instead of doing our long read over the weekend, we'll be doing a long read as part of the Friday show. Starting next week we will be back to normal, never fear. But this week I wanted to read a piece in the Economist called Keep the Code Behind AI Open, say two entrepreneurs. The authors include Martin Casado, a general partner at Andreessen Horowitz, and Ion Stoica, a professor of computer science at UC Berkeley. The latter is also co founder and executive chairman of Databricks and Anyscale. In the piece, they argue that open source models will power innovation without compromising security. And once again, this is regular Nathaniel, not AI Nathaniel reading this piece. No one doubts that artificial intelligence will change the world. But a doctrinal dispute continues to rage over the design of AI models, namely whether the software should be closed source or open source. In other words, whether code is proprietary or public and open to modification by anyone. Some argue that open source AI is a dead end or even worse, a threat to national security. Critics in the west have long maintained that open source models strengthen countries like China by giving away secrets, allowing them to identify and exploit vulnerabilities. We believe the opposite is true, that open source will power innovation in AI and continue to be the most secure way to develop software. This is not the first time America's tech industry and its standard setters and regulators have had to think about open source software and open standards with respect to national security. Similar discussions took place around operating systems, the Internet, and cryptography. In each case, the overwhelming consensus was that the right way forward was openness. There are several reasons why. One is that regulation hurts innovation. America leads the world in science and technology on an even playing field. It will win with one hand tied behind its back. It might well lose. That's exactly what it would do. By restricting open source AI development, a potential talent pool that once spanned the globe would be reduced to one spanning the four walls of the institution or company that developed that model. Meanwhile, the rest of the world, including America's adversaries, would continue to reap the benefits of open source and the innovation it enables. A second reason is the widely accepted view that open source makes systems safer. More users from government, industry and academia as well as hobbyists means more people analyzing code, stress testing IT in production and fixing any problems they identify. A good example in the sphere of national security is Security Enhanced Linux. It was originally developed by the America's National Security Agency as a collection of security patches for the open source Linux operating system and has been part of the official Linux distribution for more than 20 years. This learn from others approach is vastly more robust than one based on proprietary operating systems that can only be fixed by their vendors on whatever timelines they can manage. There is much discussion in Western national security circles about preventing other states from gaining access to state of the art AI technology, but restricting open source will not accomplish this goal. In the case of China, that is because the horse has bolted. China is already at the cutting edge of AI. It may well have more AI researchers than America, and it is already producing very competitive models. According to one popular system for ranking large language models, China has three of the world's top seven open source models. Some Chinese companies are also finding ways to get around export controls on GPUs. Even American companies are not easily persuaded to overlook billions in revenue. A previous attempt at prohibiting the export of high end intel chips resulted in China developing the world's fastest supercomputer using a novel internally developed computing architecture. The inability of American companies to keep proprietary infrastructure critical IP secure has a long history. Huawei, for instance, has publicly admitted to copying proprietary code from Cisco. As recently as March, the FBI apprehended a Chinese former Google engineer for allegedly stealing AI trade secrets from the company, which is renowned for its security. A question to ask is whether we want to live in a world where we understand the fundamental nature of other countries AI capabilities because they're based in part on open source technology or a world where we're trying to figure out how they work. There is no third option where China, for example, doesn't have advanced AI capabilities. The final reason to favor open source is that it drives innovation. The argument that we should move away from open source models because they cannot compete with proprietary models on performance or cost is plain wrong. Foundation models are on their way to becoming a key component of application infrastructure, and since at least the mid-1990s the majority of impactful new infrastructure technologies have been open source. There's no clear reason why AI models will be different. Today's AI is rooted in open source and open research, and the stunning advances in generative AI over the past two years with with the rise of OpenAI, Mistralanthropic and others can be largely attributed to the openness of the preceding decade. Today, many of the most advanced uses of AI are the product of developers running and fine tuning open source models. Many of the most advanced users of AI are in communities that have grown organically around open source. The die has been cast. There is, of course, room for different business and development models to thrive, and no one should take national security lightly. But restricting open source would hamstring an approach that has held its own when it comes to security while driving three decades of innovation. So a couple quick notes about this. First of all, it seems to me pretty likely that they wrote this some time ago, given that they don't make mention at all of the fact that Meta's Llama 3.1405 B and Mistral's Large 2 seem to have largely or at least close to closed the gap with closed source models. When it comes to state of the art, that certainly would impact, for example, the argument that open source is always going to be behind. I think the key implication of this piece, though, if you believe that they have their history right, is to ask the question, what would make AI specifically different? What is it about AI technology that makes open source more dangerous than other instances Today's episode is brought to you by Venice. The leading AI companies store your entire conversation history and attach it to your identity forever. That's every question you ask, every answer you receive, every image you generate, every thought you share with the machine. It's all being spied on. If you trust all the company's hackers and NSA board members that will ever have access to your AI conversations, then rejoice for you are well served. For the rest of us, Venice is an alternative. Venice is a powerful AI app for text, image and code generation that respects you as a sovereign individual and believes privacy and free speech are not only human rights but necessary for civilizational advancement. Private, permissionless and uncensored, you can try it for free without an account. AI Daily Brief listeners receive a 20% discount on Venice Pro. Visit Venice AI NLW and enter the discount code NLWDailyBrief. That's NLW Daily Brief all, all, one word. Today's episode is brought to you by Superintelligent. As you guys know, superintelligent is a platform we are building to help everyone, individuals and teams maximize their use of AI. We help you figure out how to use AI tools as well as what to use AI for. And this is really important. The whole goal of Superintelligent is not just to give you tutorials and lessons, but to show you how other people like you are actually getting value from AI right now. For those of you who are still out there working, learning and grinding deep in the summer, I'm excited to share our best offer ever. If you sign up with code year50 right now, you will get 50% off the already reduced annual price. Due to popular demand, we have extended this offer for just a couple days. We were going to close it at the end of July. Now we are running it through Friday August 2nd. But if you want this 50% off discount, head on over to be super AI and take advantage right now. Like I said, this will close on Friday August 2nd. The code one last time is year 50 for 50% off the already discounted annual rate for superintelligent. See you there. Well, the Economist did offer a counterpoint to this, and it's certainly not by a Luddite Creative Commons founder Lawrence Lessig, also published in the Economist. In a piece titled Not All AI Models should be Freely Available, Lessig writes, free and open source software has driven technological innovation and the spread of technical skills. The freedom to share and build upon software developed by others has given countless young coders the chance to learn and is at the core of the business that powers the digital economy. It therefore feels obvious to many that the principles of free and open source software should be extended to the development of AI models. In principle, they should. But there are important differences between ordinary software and AI technology that counsel against the simple extension of a simple principle to the full range of AI models. AI is more a category than a technology. Like the category weapon, it ranges from the relatively harmless to the potentially catastrophic. No one would believe that the access we allow to pea shooters should be the same for Stinger missiles. Neither should we believe that the software norms developed for operating systems or media players must apply in the same way to highly capable AI systems with the potential to cause immense harm. Nor is it even obvious how the norms of free and open source software should apply. Open source software is software whose source code is released under licenses that allow others to copy and modify the code. It is the access to that code that spreads knowledge. But AI models consist of at least four types of digital components, only three of which are actually software. The fourth model weights is both the most potent and the most obscure. Model weights are the variable or numerical values used to translate inputs into outputs. They encapsulate all that the model learned during its training. Thus, if the training cost $1 billion, the model weights reflect that value. If the training cost $1,000, they are obviously less powerful and less valuable. So which among these four components must be shared to be consistent with open source values. Source code is certainly one for it teaches the world how the model was built. But model weights are just a string of numbers. On their own, they don't teach anything. With the other software components in the data used to train the model, they certainly could teach how the model understands. But distinct from what they teach, they are simply the power of the model. On the analogy to weapons, model weights are not the design or plans for a weapon. They are the weapons. In my view, all four components should be freely available for models of limited capability. Hugging Face, an AI community platform, offers over 350,000 AI and machine learning models, 75,000 datasets, and 150,000 demonstration applications, all open source and publicly available. These models are likely not powerful enough to do significant harm. Making them available supports an ecology of free knowledge that is critical to improving the understanding of AI. Yet the same logic does not apply to highly capable AI models, especially when it comes to releasing model weights. Whatever model weights can teach, that benefit must be weighed against the enormous risk of misuse that highly capable models present. At some point, that risk is clearly too great. Mark Zuckerberg, founder of Meta, the creator of Llama, the most powerful open weight release to date, assures us that open releases should be significantly safer since the systems are more transparent and can be widely scrutinized. They can be widely scrutinized, but when if the danger is discovered after the code is in the wild, then the assurance that all can see the problem equally is not much consolation. Mr. Zuckerberg promises that the foundation models behind freely released model weights have guardrails to protect against harmful or dangerous misuse, and that using llama, with its safety systems like Llama Guard will be likely safer and more secure than closed models. However, researchers are now demonstrating just how easily these guardrails can be removed. Llama 2 had guardrails to block users from developing it for improper or unsafe purposes. But in 2023 and for less than $200, a team from Palisade Research was able to disable these and produce an unconstrained version of llama 2. Just how dangerous could these Frankenstein open weight models become as the foundation models behind them become more powerful and the techniques for removing guardrails become more sophisticated? The point is not that only open weight releases can be hijacked, but they do create a unique risk because once released, they cannot be recalled. By contrast, models that give access through web portals or regulated APIs could in principle identify when users are attempting a hijack in principle, then, they could more easily shut down malicious use than could models that have been freely distributed. Together, these threats suggest we need a more sophisticated framework for understanding what parts of AI should be freely available and what parts should not. That understanding must track model capability and, as I will explain, regulatory capacity. For low capability models, we should encourage the hugging face ethic. The risks are low and the contribution to understanding is vast. For high capability models, we need regulation that ensures both closed and open models are safe before they are released, and that they are not released in ways that could create catastrophic risk. No simple line will divide low capability from high. But if we're to secure the potential for open source development, we must develop the regulatory capacity to draw this line and enforce it. Importantly, these risks are contingent on the regulatory capacity of the infrastructure within which AI operates. AI researchers are now exploring ways to use the chips that run AI models to regulate the models themselves, building governance, as it were, onto the chips. If this capability were required, generally, it could operate as a kind of circuit breaker, disabling runaway systems. Such mandates create their own risks. But a technical governance infrastructure would allow us to focus on model risks, not whether a model was open or not. And as it could help lessen the risks of AI development in general, it could weaken arguments against open source release. In particular, private companies alone in fierce competition with each other do not have sufficient incentives to avoid catastrophic risk. Neither would simply banning open source AI avoid the risk of great harm. Instead, we need to develop the regulatory capacity to ensure an environment within which safe AI can be developed, and the regulatory judgment to determine when the public risk from any AI deployment is too great. Today, these risks are imposed upon us by private actors with little public oversight. The formula has not worked with dangerous technologies in the past. It will not work with AI systems of the future. All right, so that is another side of this conversation. Now, I present these both to you so you can have some time to digest and compare them and figure out what you find more compelling. You can also go back and look at the piece we read from Meta's Mark Zuckerberg last week. As yet another argument in this conversation, I will say that I think it's interesting that Lessig is coming at this from a very different angle than some of the folks that have been critical of open source in the past. He's basically trying to shift the conversation away from open or closed into just model capabilities in general. Effectively, he's arguing that if we have a good system for handling when a model is too powerful to be released in general, it's going to, as a part of solving that problem, also likely solve this problem as well as. Anyways, let me know where you think what you feel about this argument. Use the comments here on YouTube or on Spotify, which of course now has comments available for everyone. For now though, that is going to do it for today's AI Daily Brief. Until next time, peace.