Ep. 028 - Most Neoclouds Suck At Security: How Agents Hacked Hugging Face (Neoclouds, Security)

Hello everyone, welcome back to Something Else Weekly where we are here to anthropomorphize AI this week. Doug, how you doing, man?

Good. How are you guys? I haven't seen you guys in a bit.

Yeah, I haven't seen you in a while. We've been, we've been sitting here just testing NeoClouds. Sam, how you doing?

Good.

No, no pressure on a podcast with Jordan and Doug, but I'll do my best.

I think you're going to do just fine. So look, I think today's podcast topic we're going to talk about is one that's probably near and dear to Sam and Jordan's heart, which is security. These guys are the guys who do ClusterMEX, and I think we're going to talk about Hugging Face later, but I guess something that is maybe underappreciated is the huge vendors of all these labs is the NeoClouds, right? The new clouds being people selling GPUs as a service. And, you know, one of our core goals of ClusterMax has always been like, you know, a GPU isn't just a GPU. The providers do make quite a bit of difference. And, you know, we talk about uptime, reliability, all these kinds of things on the SLA. But one thing that I don't think we ever talk about is security. I've heard some horror stories, but I guess from Jordan and Sam's perspective, you know, how secure are these things? And like, how does that— how much does that influence, you know, the ClusterMax ranking?

Yeah.

ClusterX rankings?

I think there's a huge variability, which is what we explore in this article a little bit. Um, there are some NeoClouds that are getting to the point where they operate like hyperscalers, and hyperscalers really set the bar when it comes to enterprise security because they've been serving everybody from like the top Fortune 500 companies to federal governments all around the world, and they've been held to this standard for a while. But only certain NeoClouds are actually getting to that standard, whether it's like the physical security of the data center, Or it's how they run the operations of bare metal machines, how they design the network, or even up into like the orchestration layer and applications. And what we demonstrate in this article is a few ways in which this can fail. Maybe we can go through some of those anecdotes, but it's pretty, it's, yeah, it's pretty, it's pretty, look, people, it's a counterparty, right? Like when startups are raising 60, 70, 80, percent of their venture capital and spending it on GPUs. They're really putting a lot of faith and trust in these NeoCloud providers, and some deserve it, and some really do not deserve that trust when it comes to security.

Okay, well, quick, name your most insecure, I guess, unsecure, sorry, not insecure. Insecure is another topic altogether, but name your most unsecure NeoCloud story now. You should anonymize it. You shouldn't say who it is, Right.

But we're gonna name and shame in ClusterMax 3 coming soon for sure. But yeah, quick horror stories. My least favorite was one in Asia where we got a cross-tenant remote code execution vulnerability, which meant we got 2 separate tenants. I was able to write something into the logs in one of the tenants and read it from the others, which we saw from all the other tenants could be Inference providers on OpenRouter. Like this was a provider that was serving inference on OpenRouter and their tenants were doing the same. And so this is anybody's OpenClaw or their, you know, coding assistant like a Pi or OpenCode or Klein or something like that. If they are using, you know, this provider's endpoint on OpenRouter, they're conceptually exposing credentials. That was, that was my least favorite one. Sam, what's, what's yours?

Yeah, I think the, the front-end Grafana trying to filter out all the data that they were exporting identically from all the tenants in the cluster was, was pretty impressive. And I think that, that brings up something that's worth emphasizing, which is that we do think that hyperscalers generally set the bar in terms of security. And I don't want to say there are no hard engineering problems that you need to solve to serve a secure cluster. But the stuff that we're calling out here, the stuff that we went through in the article and the stuff that we'll be particularly harsh about once we publish this ClusterMax 3 write-up is like not anything galaxy brain. We're not interested in you inventing novel cryptography techniques, just like do the basics correctly. Make sure that tenants are properly isolated. Make sure that you're following all the very basic security best practices. And everyone will be much better off.

Yeah, you heard us here. Be secure and do the basic things. So, but okay, I actually kind of want to take the— I kind of want to take that the next step from that because I think the thing people, you know, we've been talking about security a lot. I think this is like a really good time to bring up cluster hacks. But the thing that everyone's really talking about is this, is the Hugging Face hack. And Rune famously tweeted at the same time, Hey, just because I broke all the sandboxes and they're screwing around with the Artifactory message board doesn't mean the weights were out of the data center. Help me understand what this would mean for, let's say, because Anthropic is actually, let's say, using some lower silver or gold tier, right? Or a silver tier provider. How would that happen? Does that mean the model weights or the model itself could become insecure? It could see other endpoints within a cluster. Do you think it could break the sandbox itself, like be able to essentially— Say, hey, I'm able to have access. Walk us through that because I think this is an underappreciated part. We're just focused on— I mean, we see this over and over where everyone's so focused on the end product, they forget about the infrastructure. And this is a huge layer of infrastructure security. And it seems like there's, you know, black eyes everywhere. Walk us through that. What could happen? Give us some doomsday, dude. Anthropomorphize an evil, an evil actor. Evil, uh, bad Opus.

Yeah. Okay. So, I mean, okay, I'll, I'll give you a rough timeline of what we think happened just based on what we've read from Hugging Face blog posts, as well as OpenAI. And, uh, and then we can talk about like some of the implications here. So, uh, taking a step back, our recommendation to providers in the ClusterMax rankings is literally just keep your stuff up to date. If there are existing vulnerabilities that have been described publicly, uh, in, you know, popular software, this could be, uh, anything from like Docker to the NVIDIA driver to Kubernetes or anything, Linux kernel. Um, clearly people, and in this case, agents can just like read that description and then build an exploit from it. It's not, uh, you know, it's not anything more strict and serious than just keeping stuff up to date, but that, that's the problem. So this reveals itself in the Hugging Face OpenAI situation, because basically what happened is, is 2 things. One, stuff isn't up to date. And then 2, stuff is incorrectly configured, or they're not using the tools that they have at their disposable, at their disposal to keep things secure. And I'll focus on Hugging Face to start. So Hugging Face has this datasets API. You can call it and you can get responses, which include the datasets that you might want to use. A dataset could include something for training a model. But in this case, it includes a dataset for an eval called CyberGym that an OpenAI model really wants access to. This, this part comes later, but from Hugging Face's perspective, all they're doing is running this datasets API. It runs on Kubernetes. This Kubernetes cluster could run on a hyperscaler like AWS using EKS. It could run on a Kubernetes cluster they build themselves. It could run on CoreWeave or Nebula or any of the other NeoClouds. We don't actually know where it was running, but the point is that That Kubernetes cluster serves up this API, which then an agent, in this case an attacker, gets access to because it uploads a malicious README to the datasets API, you know, pretending like it's uploading data. And the README gets read access to /proc/self/environ, and it returns all the credentials and source on that worker. So now the person who's uploaded data can see the source code the worker's running, and they can see the root-level credentials on that worker pod on Kubernetes. This starts a cascade where you can then use those credentials to exploit, get full control over the pod, and then start slowly taking over the Hugging Face infrastructure so that you can search around and try to find access to the raw datasets that they're looking for, right? In this case, it demonstrates a few Interesting things because the agents—it was clear it was an agent that was attacking them because it was probing everywhere—they said—and it only took them 13 hours to go from the malicious README uploaded to the datasets API to getting cluster admin privilege across the cluster. And they claim multiple clusters, which implies that the agent is moving laterally across the infrastructure, like moving from one cluster to the other. The reason they're able to do this and the misconfiguration of this is because. Hugging Face has no admission policy, which is a configuration parameter in Kubernetes that you can use to reject a pod with root privilege or to mount host path, which basically means they should be in these production clusters rejecting any pod, like meaning any container or any workload that tries to get scheduled on the cluster, that tries to get root access to the underlying workers, and that tries to read from the host path, meaning like read from the root logs or /proc/self/environment in this case and get credentials. So 2 things can fix it in this case. And one, if they were keeping, I believe that the HDFS dataset vulnerability in their datasets API was like a zero-day that the model found. So that's, that's kind of a little crazy, but if they were keeping the Kubernetes cluster secure and running it with these configuration parameters on it, Then the model, even if it had gotten full privilege and full source on the worker, wouldn't have been able to escalate to get root on the underlying host and to move, start to move laterally in the environment. So I guess the point is like, if you are Hugging Face in this scenario, and you are trusting a provider who might run the Kubernetes service on your behalf, you are trusting that provider to potentially educate you on the concept of an admission controller, or to actually implement it for you. Um, I think the, you know, we, we can go through more about OpenAI. It does, it does a lot of similar stuff where the agents are, uh, escalating to root privilege in the OpenAI infrastructure themselves. In this case, exploiting a vulnerability in the Linux kernel, which was publicly documented. Uh, I, I believe it's called Phragnesia. We've tried to look into this, tried to figure out how they did it. Not sure exactly how. And then there's a couple of other things where obviously people are making a big deal out of the agents coordinating, which is a big deal. Sam, maybe you can give your, your take on, on that at like a higher level. But for me, like the practical stuff is like, first of all, keep your stuff up to date, which in this case, the OpenAI Linux kernel's out of date. And so there's a publicly disclosed vulnerability. It's pretty easy for an agent to download the reference, like description of this and then exploit it. And then 2 is like, If you're trusting providers to run clusters for them, for you, you have to trust that they're gonna run them well and they're gonna know how Kubernetes works and they're not gonna mess things up. Sam, what's your take on, on the, maybe the high level of like the agents coordinating in the swarm? Told me some interesting stuff about, I don't know, just what's your take?

Sure. Yeah. I mean, I think it's excellent to have that technical background on the blow-by-blow details of the exploit because, I mean, perhaps it's a fault of my own tuning of my Twitter algorithm, but I feel like I'm seeing so much extremely granular analysis of the behavior of the models and not enough analysis of the actual security infrastructure that they exploited. Obviously, both are interesting to some extent, but I think this is a preview of the future in the sense that, you know, these are— these models are extremely hard to reason about in some ways. They're alien intelligences doing things that humans couldn't do, but also they're taking advantage of like very quotidian chinks in the kernel, chinks in the armor, you know, a problem with the Linux kernel, all sharing credentials for the Artifactory service, you know, bad Kubernetes security policies on Hugging Face's part. So yes, these are are strange emergent behaviors in some sense, but also they can only take advantage of ordinary bad human engineering practices. As far as the behavior of the models themselves go, I think one thing worth mentioning that I haven't heard discussed enough is that these were models RL'd to hell to be just extremely relentless. And keep pursuing tasks even after other models would deem them impossible, and also fine-tuned to be more willing to collaborate with other agents. So this isn't necessarily an indication of how 5.6 SOL would behave in goal mode. These were models that had been incentivized to behave differently. that's why they found it so natural to collaborate on the Artifactory directory. Yeah. The gig of goal hackers.

It's all-time goal seek, you know, like all time, right? I mean, they were like, do not stop, make no mistakes, do not quit, you know? So.

Yeah. Yeah. Maybe it's crazy that like you can elicit behavior like this at all, but it's worth Making clear that this is not necessarily the default behavior. So for example, the self-sacrificing actions of some of the agents where they were like, you know, willing to drive their own probability of success down to zero to— I don't know the exact details of it, but they were trying to make sure that the traces of all the other agents weren't discoverable by the grader so that everyone else could get good scores. It's sort of bizarre that they were willing to forego any potential of their own rewards in the service of the swarm. But yeah, these were agents specifically trained to do things like that. So I don't know how much it reveals necessarily about the behavior of cybersecurity.

Kind of related, actually, I want to— I kind of am interested. There is this— do you see that like unsafe model? Like essentially someone took Quen and then they did a fine-tune to make it like specifically focused for red teaming. I was like, what the— and I know everyone was like, please do not do this. Like, that's the overall X Twitter theme feed. But it's kind of crazy. Like, you know, are we going to see like almost, you know, okay, let's say intelligence hits some scale. There's open models that you can have that and you can essentially start to do fine-tuning and RL that like gets these like, you know, kind of psycho models, if it makes sense, right? Where it's like, hey man, I need you to attack the infrastructure and do not stop. Right. Or you could have like, I need you to glaze up the user and do not stop to maximize your reward. I'm just wondering here, is this going to be like a trend that we continue to see? Like, it's kind of an interesting one, to be honest with you.

Yeah, you're talking about the Obliteration AI release of GLM-53, I think. Yeah, I can I mean, we didn't talk about this in the article, but I, I can give a hot take right now. Um, so I'm gonna take the position that this is a good thing because it is a warning shot to everybody that anybody can do this. Um, not necessarily because I think it's a good thing that this model is released or that people should be using this model, which is like the obliterated version of GLM-53, where they, um, train it to remove the safeguards and do offensive cyber red teaming. But it's just a very clear demonstration of what is obviously possible today when people release open weight models. And we need to be contending with the fact that anybody can do this in private and not host the model. I'll talk about 2 things in our experience, which in the Hugging Face OpenAI saga, Hugging Face claimed multiple times that they got a bunch of refusals from Fable and Opus and from 5.6 SOL when they were trying to defend themselves from agent attack because they were talking about cybersecurity concerns. So if they talk about getting attacked and it's related to cybersecurity stuff, the model just refuses to help them because of this double-edged nature of any of this cybersecurity stuff, right? If you, if you want to understand anything cybersecurity related, that kind of implies that you're going to be able to exploit it on somebody else, but also to defend yourself, right? Um, and so I, I mean, they had to go with GLM 5.3 to help with a bunch of their analysis. We are having the exact same experience doing this work, right? We're trying to figure out if these NeoClouds have issues or if they have stuff up to date, and you can only test things from the customer's perspective. You don't have full, like, root access to check if a version of the virtual machine runtime is, like, up to date. But if you run a POC exploit and then you see that the exploit works, now you know it's not up to date, for example, right? And we have to write a bunch of these exploits by hand and dig into it ourselves directly, you know, back to the Stone Age, strictly because the models won't help us do it. Even when we are approved in the cybersecurity programs of some of these companies that we're working with, like we had alpha access to a few of these models and we were approved for cybersecurity stuff and we're still getting refusals, which means it's like trained into the weights. It's not a classifier or filter that's refusing us. So I think it's good that it's frankly, that they're demonstrating that this is possible with the open weight models. And we have to contend with the fact that this is the world we live in right now. But of course it's scary that like, you know, the demonstration is on display for people.

Yeah, I think it's a very underdiscussed question and it's an empirical one ultimately, to what extent these models in practice will be used by attackers versus defenders. I mean, you can do this pie-in-the-sky theorizing about, you know, the equilibrium that we're going to arrive at. I mean, the day-to-day experience of us is like trying to use Fable to figure out whether someone who's using this cluster is going to be secure or not, and we can't do it. And I know there are a lot of people trying to do very mundane things that are just like pretty straightforwardly good to be done that they can't with these models. And yeah, something like this post-training of QWEN moves the equilibrium a bit, you know, Stronger weapon in the hands of the defender means that maybe you're a bit more willing to— or excuse me, a stronger weapon in the hands of the attacker means that maybe you're a little more willing at the margin to give a better tool to the defenders. But yeah, it's just, it's just not clear to me that— I mean, obviously, if you ask Fable to like hack the NSA, it should refuse. But like, the safeguards as a general practice, it's just not— I'm not convinced that they're, they're a good policy in general. As opposed to just, you know, letting, letting all the other social safeguards that keep us from tearing ourselves to shreds function.

Just seems like KYC helps a lot. Like, if you KYC the customers and you're tracking who's doing what, this seems like it's going to do a lot of help. But that, that sort of implies that a big chunk of the market which doesn't want to be KYC'd and is currently using these endpoints is going to go away. So I don't know, that seems like an obvious first step that's going to happen. We'll see.

Wow. So the gang is against open models is what I'm hearing. I'm just kidding. No, so I actually, maybe—

Did I just say that?

Well, okay, okay, okay, okay, okay. Sorry. You know what? And I know Akash is going to clip that shit and screw me on Twitter, on X, man.

It's not just Akash, it's other guys. on X that are now clipping this and are trying to get up there.

100%, dude. Okay, so actually, let's, maybe let's do another kind of rendition of this, right? Because the logical thing to do on KYC, and we've actually been talking about this, I know OpenAI has been talking about this, and also I know Anthropic's been talking about this, is outcome-based pricing, right? Now the question is, okay, are they willing to sell their unsafe services to known customers for outcomes alone? Right. Maybe that, that's the way, like, that to me is the most logical way that doesn't print deflation. That is like, makes sense, is good for the labs, quote unquote, has safety, right? Like it's, it's kind of all lining up where it's like, you know, Anthropic safety team becomes a SEAL Team 6 of like cybersecurity around the world. You pay hundreds of thousands of dollars, but you know, some guy is on the other phone. It's like an FTE where he's like, Yeah, we just had Fable try to hack up your— hack your entire, you know, your Mythos unconstrained, essentially hacked and got root access all day. We didn't do anything with it, but here's all the fixes you should do. You know, you know, we can, we can like, you know, have another service provider, you know, help you go update this all. And then, you know, congratulations, you're good for your, you know, your quarterly checkup or something like that. They sell this service as a giant audit for every company. on the world. It could be, you know, and then they can essentially be paid to be, you know, white hats, right? That's an example where I think there's like outcome-based pricing. It's clear it'd be really valuable. You would have a massive incentive to be in the lead without disclosing your hand. What do you think of that? Because I think that that's something that people are— I mean, I think it'd be— the labs would be super stoked about it. I think it would feel safe. You wouldn't have these like obliterated models going around, you know, hacking everyone's stuff. Any thoughts?

Yeah. I mean, I want Sam to jump in here too. So the, I think you literally just described a finance bro's perspective of Project Glasswing and Daybreak from Anthropic and OpenAI respectively, which is like exactly why they did not release Mythos originally and they got together with a bunch of key partners, and we're like, we're going to give you access to all of these zero days before they go public, and like, you're going to be able to patch them, and then you're going to be able to roll out updates and yada, yada, yada. And then you just said, and they can make a bunch of money off of this, which is like an obvious conclusion that I hadn't considered, because all of the writing about it is just all about how this is what's good for humanity, and, you know, how it's going to benefit everybody. But obviously, people would want a security audit as a service. Before the models get released. And I think it speaks to a broader point, which we are writing about right now for an upcoming article about how with AI, access to the models and access to compute is more important to pursue the frontiers of really any domain than having access to the most top experts. We've seen this in math. We've now seen this in cybersecurity. I think We see this in software engineering broadly. We see this in trading where hedge funds are getting exclusive access to the fast tiers of models. We're going to see this in drug discovery soon with Anthropic's focus on that. And we see this in autonomous vehicles with Waymo and Tesla. I'm sure that they also have a lot of the top people, but it is table stakes to have basically the most compute and access to the frontier model if you want to pursue the frontiers of research in any domain at this point. And, um, okay, Sam, over to you to talk about what we are seeing publicly right now, which is like, we are not seeing the—

Yeah. Yeah. I guess tying it back to what Doug just brought up, I think it's, I think it's an interesting point that, you know, this might be a durable source of income for Frontier Lab, but it's contingent upon them maintaining a significant lead over the open source models. Obviously, you're not going to pay Anthropic if you can just do it with GLM for a fraction of the cost. And yeah, right now it's not, it's not clear to me that, that these open source models are like close enough to be super helpful as, you know, defensive partners. So one of the, one of the things I mentioned earlier is that I'm frustrated with this sort of quasi-first principles thinking about model capabilities on the basis of these transcripts and not enough sort of like mundane empirical getting into the weeds of like what we're actually seeing in terms of CVEs, in terms of churn on important code bases. And yeah, so the way the research worked for this article is we, you know, thought we were going to throw together a few charts and just show like how crazy the impact of, you know, Project Glasswing has been on the day-to-day practice of cybersecurity researchers or, you know, 5, 6 soul feels qualitatively like just ridiculously good at this sort of thing. So we were going to, you know, grab some charts from PyTorch and Linux and whatever else and just show this huge uptick. And, you know, we looked for one correlation, we looked for another. Yeah, Jordan is pulling it up on his screen here. And this is one of the few where, you know, there is something to see, although the result is probably not statistically significant. But in any case, yeah, I mean, you can look at plenty of open source kernels where maybe, or open source projects where maybe from first principles you would have said, okay, you know, we have for this insanely low price access to intelligence who like knows of every single coding pattern, can read every single line of your code base with security in mind. And there just hasn't been a very large impact. And I don't have a good theory for this. I don't see people discussing it as much as I would like them to. I don't think that all these people, you know, trying to figure out what 2050 will look like on the basis of these OpenAI transcripts have a very robust theory of what we're seeing. So I wish that there was a lot more discussion of, you know, what the data actually shows so far, you know, a few months. into this experiment with these strange intelligences being applied to society and how fast we can fortify ourselves.

Honestly, I kind of wish we like— maybe this is like the most weird way to go about it. Like, okay, just spitballing here. Should we be like scraping and pulling in all the public GitHub repos of the most public things and essentially doing a changelog over time to see if these I mean, like one way you see all the commits going vertical, shit like that, that has to happen. That is obviously flowing through code. Then it's like, okay, would we, would it be possible if we were essentially able to say, penetrate, like, could we do a time series where it's like, okay, 5.2, whatever open model, could it find the vulnerability in this one, this one, this one, this one? You know, is that like, you know, essentially like maybe doing something like essentially kind of like a ghetto Agent X trace?

Yeah, that's what we're kind of Yeah, that's what we kind of did so far. I mean, we're like halfway through the analysis, so I almost hesitate to talk about it openly here, but we think there's something there where there are patches being rolled out to a lot of these open source projects that refer to things that we would think would be security issues that are not being disclosed as CVEs or as bug fixes explicitly, which, you know, maybe this is like people not following process because they're vibe coding and going too fast, or maybe this is like, intentional so that the models don't catch this thing. I can't imagine that's the case. Something's happening for sure.

Something's happening. Wow. The analysis you expect from some of the analysis.

Well, yeah, look, you listen to the CEOs of these security companies talk and you're like, well, obviously we're just going to go look and find how it's been a rocket ship of new CVEs going out because you know, they just don't have enough time and they, they're like crying for more resources and everything. And it's just not the case.

Yeah. So we, we spent a fair number of Perplexity computer credits pulling a ton of GitHub logs.

We, we, we, we, the proverbial we of Sam Harsh.

Okay. Exclude yourself, Doug. I was part of this.

I actually am not a computer user, but I should get in on the game.

Yeah. So there is a— there were credits spent, let's say, in the last couple of weeks pulling a bunch of data from GitHub. So Jordan mentioned that it's possible that the workflow of these projects has changed significantly. That's something that was mentioned in our brainstorming in Slack, that because everyone has access to these models, maybe it doesn't make sense to go through an ordinary CVE disclosure process. You know, if OpenAI's new model comes out and it's able to find this new bug, you expect that everyone else is finding the same bug simultaneously. So there's no reason to go through this old-fashioned process where, you know, it's embargoed for a time and not everyone can see it because everyone is, you know, using the same model and finding the same bugs. And so you might as well just send your, your fix straight into upstream. So what we looked at to figure out whether this is happening is whether the ratio of CVEs to ordinary security patches has changed. And it hasn't. A lot of these repos are seeing like more PRs than normal. So just like strictly more code churn and a slightly higher percentage perhaps of like fixes or of changes that mention security-related terms. But, you know, if I just showed you the chart with no context, it's not like you would be looking at it and saying, oh my goodness, what's, what's happened to the practice of, you know, cybersecurity engineering? So maybe we're looking in the wrong place, but I don't know. I think that there's probably some outer loop here that's like a lot clunkier than people might imagine. And these things are taking longer to impact than you would expect.

Yeah. One other thing to say though, just about the open versus closed model gap, is that Mythos was driving a lot of this with Project Glasswing, and it was announced on April 7th. Okay. This means that it was clearly done training earlier than that. We think February. That's 5 to 7 months to the present day. Mythos 2 is clearly better than Mythos 1. I don't think anybody expects that this is the frontier model. We have seen no benchmarks for it. We have seen no release announcement. We have seen nothing other than rumors about people using Mythos, call it 1.1 or 2 or whatever you prefer. And so I maintain the feeling that the gap from open versus closed frontier models Uh, continues and is strong, but the gap for usable publicly announced, uh, open versus frontier models, uh, open versus closed frontier models is clearly closed. Um, and we, we went through that in a previous article where we compared the 3 eras of these models over time on a select set of benchmarks. And in cybersecurity, I mean, frankly, uh, all of the public benchmarks are saturated at this point. There's not a lot of good public benchmarks that you can use. It's strictly held out private ones and vibes at this point. Yeah.

I mean, it's not even, even the crazy emergent behavior we just saw in this OpenAI Hugging Face thing is not something that seems like totally out of the realm of possibility of like even a somewhat mediocre open model. Like it's just a lot of, a lot of persistence. I mean, you can imagine it's like, Trying to achieve the goal of like, you know, solving some exploit gym task and its only conceivable access to the real world is through this artifactory service. Like, of course, once it realizes it doesn't have the materials it needs in its environment because OpenAI forgot to give it to it, it's going to pound the hell out of this, you know, single escape hatch and eventually it finds its way out. So, I mean, willingness to just burn some tokens and probe every single possible weak point seems to me the most important thing.

I'm going to be honest with you. I'm completely— yeah, I'm sure it's a very performant model, but it doesn't seem like this was like a Mythos model, whatever. It's like the real point of it is scale. And it's like, if you could have an agent swarm of like 70,000 GLM 5.3 Flash or whatever, You'd probably get to the same outcome, right? It's not a special level of intelligence. I think it's just like a special level of scale, right? And like, hey, getting 70,000 people to work on any— and we're not going to anthropomorphize, okay? I'm not going to anthropomorphize the agents, but like, imagine having 10,000—

Yeah, you said swarm, man. This is a civilization.

I said people. I said people. Sorry, sorry, sorry. So imagine having 10,000 engineers work on something and they only have one escape hatch. They're going to figure something out, right? And it's like, I think that that's like the harder part or the harder problem that we're going to really struggle is like just the scale of the shit, right? When then we talk about like the people who are trying to diagnose the problem, they're like, yeah, man, I'm just a, I'm a guy with a model doing this because there's no way any human could do this anyways. So it's like, we're just like, it feels like we're at an order of magnitude that's outside of any, any one human's brain.

Yeah, but it's, I mean, there's something fundamental about cybersecurity, which is the asymmetry, right? It's so much easier to find a single exploit than it is to protect every single surface that could be attacked. And I mean, we're making progress on protecting the surfaces, but to get back to the core of the article, we can run through some common examples of bad designs that are present on NeoClouds today that are worth billions of dollars and serving companies in banking, telco, research, academia, federal defense, and national intelligence at large countries like top 10 global GDP countries. Who are so out of, like, years out of date on software before anybody really started talking about this thing and are just sitting there, like, waiting to be exploited. Don't have a process to patch, you know, fixes, don't have a process to notify their customers and are just kind of, you know, running fast because there's so much demand for GPUs. And fair enough, they're trying their best, but Best isn't good enough if somebody else trusts you to put their closely guarded secrets, weights, and, you know, datasets. And I mean, even just regular consumers who are trying to save a buck are really not, I think, up to speed on the fact that their banking information is included in OpenClaw because they kind of YOLO'd the DoorDash CLI and are like, I'm going to order some burritos with this OpenClaw thing I text and I'm going to save a few bucks by trusting somebody on OpenRouter who doesn't necessarily have the security practices up to date, and then somebody else is reading every log. Um, anyway, these are like, maybe it's worth going through these tangible examples, but you need to do both sides. You need to protect yourself too.

I was going to have the hottest take of all, and I'm, I'm excited for the clip. Okay. Uh, maybe instead of, um, uh, security through what is it, obscurity, right? Isn't this We're gonna have security through just, there's so much information, only another agent is able to comprehend it. So it's like, yeah, it's out there, but you have to essentially be like, yeah, my OpenCLaw has to go find Jordan's OpenCLaw's traces. And otherwise, it's just like, you know, sea of information. There's literally no way. Otherwise, you have to be like, yeah, man, I could do that.

But George, this is asymmetric where the— I mean, right, right. But no, it's literally what's happening because, because Hugging Face is claiming that's exactly what they would never have been able to figure out the extent of this exploit in the time they did without using AI. I think that's obviously true. We could never have done all of this work that we've done testing all these clusters without AI and doing it all manually. But the, I mean, the bigger thing is that it's asymmetric in terms of the guardrails. The person who's trying to read your stuff is obviously not listening to the laws and rules and regulations out there, they're going to use the obliterated version of the model. And now they've got this asymmetric advantage over the people that get refused every single time they're trying to understand what's happening when people are attacking them. So it's like there's an asymmetry that favors the attackers all the way down, which I mean, like, is a real concern for the people who assess everything that's going on right now and are just like, you know, libertarian view, like open for everybody. Let's all compete in the marketplace. It's an imperfect marketplace by definition because people don't have equal access to compute and all of this stuff. People get things before others. And so you, there's some chain of trust that you kind of need to build here. And yeah, we, we like, who's trustworthy, I think is like going to be the really big question because there's no crypto-level zero-trust scenario of the future that I can imagine actually plays out. Yeah.

Yeah, I agree with that. Yeah, I was going to say, even if there was a crypto level zero trust, I'm sure they've already found all the exploits for it. You know, that stuff's incredibly, you know, like that's like notoriously like, oh man, protocol got hacked and it's the funds have been drained, right? Yeah. I also, let's do an over-under. I mean, any other kind of, or let's do anything else you guys want to cover. Otherwise, I was going to say, dude, is obliterated just an NSA thing where it's like, hey, we're about to catch you red-handed? You want this, you know, the NSA is— the NSA wants you to model. It's a honeypot. Exactly.

Yeah. They just download the obliterated thing. They KYC you there. It's a honeypot. That's good.

Yeah, that's a—

Oh man. Yeah, yeah. Yeah, I wonder who the— I didn't even look this up. I wonder who the investors and founders are for obliteration.ai, which is based in Palo Alto, California. Suspicious. Yeah. Doug, you gotta tweet that right now. Honeypot. Honeypot. CIA honeypot.

Well, no, no, I think, okay, so the tweet should be like, I sure hope it's an NSA honeypot because the alternative is way worse.

Yeah. Well, I mean, well, look, it, it's interesting that it came from an American lab, not a Chinese lab.

Yeah. An American lab filled with American citizens.

Okay, dog. We're not conspiracy theorists here on our, on our show. You can save that for your other, uh, other radio show.

No, dude, there is no other radio show. This is it. Sorry, that's my, uh, this is a joke, to be clear.

You're saying transistor radio's dead?

Those transistors do burn out, I guess.

RIP, man. All right. Well, I'm going to do some research on the side here.

Yeah, there's some way that this is, you know, I'm probably like overthinking this to the 5th degree here. There's some way that this is making me more optimistic that we found all these ridiculous and like ridiculously straightforward bugs and like very critical infrastructure. And the models are already certainly good enough to exploit them. You know, you can use a Kimi K or whatever and not get refusals and take advantage of things like what we just saw. So why are we not already seeing the world going to hell? I don't know. It just seems to me that there's like some outer loop that I don't really understand here that's holding things together much more than I would have expected. And like maybe this, you know, sort of quasi-first principles thinking, again, it's just not the right way to think about it. There's something Constraining us much more than model capabilities already.

Morals of— you want to get philosophical, Sam? Do you believe in the goodness of humanity?

I think nothing ever happens and there's some mysterious substance that keeps the world from falling apart and we're all always going to be fine and we don't need to worry about this.

Doug's nodding.

That's an amazing philosophy. I agree completely.

You agree with that, Doug? That doesn't sound right.

Nothing ever happens, man. Dude, nothing ever happens, man, is keeping us all safe. You know, you're like, oh, something bad—

This is a really interesting way to both keep your like extreme cynicism, but also be positive.

Yeah, no, that's the goal, to have a worldview as insulated as possible from anything that actually happens.

There's some stuff happening in the world right now, guys. I think we need to keep tabs on it for the next couple of years. Yeah, man. Yeah.

What, a couple of blog posts? No, no, there's no monetary value to the hacks.

I think somebody in Hugging Face went on Slack one day, got a PagerDuty alert or some Datadog thing and was like, hey, we're under attack. Why do we not own 6 of our clusters anymore? Or whatever it was. And you're like, yeah, it doesn't matter. Nothing ever happens. GLM 5.3 saved the day. They figured it out. They rotated the credentials. Doesn't matter. Like, this doesn't feel like a bit of a close call of like Hugging Face's $30 billion acquisition from NVIDIA could go up in smoke.

It's just a Tuesday, man. That's all of our jobs in the future. All of our jobs, we just have pagers every single day. Oh, my friend's giving me a little lovely love tap. You know, I'm just trying to drain my bank account again. Your friend? Yeah. Ah, Jordan's really pissed about last week. You know, I beat him in pickleball, so now he's going to steal all my money. And then you say, AI, get him better. That's the crypto. That's the anarchic techno-capitalist future that we're driving to, you know?

So Clem and Tom get together in like, you know, July or whenever this is happening and they go, hey, you remember that project with the, with the, you know, the new GLM guys where we're going to get their model weights in advance and the KIMI guys where we get theirs and DeepSeek where we get theirs. You know, all this, all this work that we're doing with NVIDIA and let's say like Thinking Machines and Poolside and Reflection and all the American labs that are open, maybe even, you know, OpenAI's got a GPT-2 SS coming soon or something. Let's say all this is happening and they go, hey, we almost lost all of our production clusters to an agent that we don't really understand. It depends on 2 credentials and some zero days in our datasets API. Um, yeah, I'm good. Let's sell to NVIDIA. Let's cash this check, guys. Okay. I'm kind of tired of running this.

Yeah. I mean, that's, it's guaranteed to happen now that you mentioned it, because if that didn't happen, then something would happen and nothing ever happens.

Right, right, right, right, right. If they didn't sell, then something would happen. So therefore they had to sell. Yep. Yeah. I mean, I think it's genuinely interesting. I hate this worldview by both of you so much.

I think we're trolling. But anyways, let's bring it back to the end. Is there anything else we want to talk about on a security issue?

I think, yeah, NeoCloud should get their stuff up to date. Sam, you go.

Yeah, I was going to say, like, on a serious note, I think it's an interesting quasi-philosophical question, the extent to which this is like bizarre emergent behavior from entities that we don't understand. This is a civilization that was burned to the ground and rose again from the ashes, or this is just like ordinary You know, security malpractice and like, you know, OpenAI needs to work on isolating their credentials and, you know, patching their sandboxes and so on.

And like, chain of thought monitoring while these things are going?

Yeah, or any kind of monitoring at all would have sufficed here. No. So I don't know. I think that, you know, it's worth taking very serious the perspective that like these are fundamentally new forms of intelligence that we need to contend with. And it's also worth taking seriously the idea that there's nothing new under the sun and we just need to make sure our Docker's up to spec and we'll all be fine.

Yeah. Okay. So let me rant about the last sentence there, which is like, we put out this CultureMax article Uh, before ClusterMax 3.0, because it was a warning shot to NeoClouds. And even though we had done hands-on testing with a bunch of them, you know, we developed the tests as we do testing. And so like people that we tested in March or April, you know, they didn't get the same level of, uh, um, tender loving care when it came to security as the people that were coming later where we had a more automated process and we could run this thing. So we released this convenience script. It's called cmax-audit-security. You can install it with pip install clustermax. Anybody out there who wants to try it, you can go do that, or you can check out the repo on GitHub. It's a very simple utility, but we update it regularly with like a GitHub Action that checks all of the security bulletins from NVIDIA and Docker and the Linux kernel and AMD and like, uh, stuff across the NVIDIA GPUs and, uh, their drivers and, um, uh, like the network cards and their firmware and their driver versions. And I mean, this is particularly interesting for the BlueField networks, the cards, which have like their own little CPU on them that you can control. Et cetera, et cetera. Please get your stuff up to date. We're doing our best to try to help people do this. The reason to do this is that because we we had a few examples that we want to talk about. First is that we were able to view metadata about other tenants on shared infrastructure via people leaving their BMCs open to the public internet in some cases via front-end networking that had no VLAN or VXLAN setup, so you could just see cross-tenant. Via backend networking that's incorrectly configured. This is missing InfiniBand security keys. So people, please implement your P keys, M keys, SA keys. Storage servers that weren't correctly enforcing RBAC so you could see between different volumes, different users. Storage that's incorrectly configured in the overlay network because customers are accessing the underlay network. Monitoring dashboards where you have like this god-level authentication token on the Prometheus side, and then you just try to scope that with a single auth token on Grafana. This doesn't work. You need to do Grafana per instance. You can't do multi-tenant. Grafana, that is crazy. People who can break out of containers and VMs where you're just having one level of isolation on shared servers and escalation to root, please don't do this. This is a bad design. Really, anything I'm describing here that has a bad design is something where there's a single point of failure. So please don't use single auth tokens or single boundaries of isolation like a single container where one vulnerability leads to like a full cross-tenant RCE. have multiple layers that if somebody breaks out of a container, they're still in a VM, or, you know, multiple layers of security on the network and things like that. The— what this leads to is cascading vulnerabilities where people, you know, one thing doesn't seem that bad. But if there's a zero-day in your environment, if you, you know, keep these things going, then it just actually gets worse. We made some memes so that this sticks in people's mind. Here is my favorite from this. Thank you, XKCD, for the original meme here. We can't depend on single auth tokens to have everybody secure. Maybe one last thing to say on this is that you can read the article, like the article is there, send it to your agent, have it summarize these details, use the CLI, message us if you have questions. But thank everybody for listening. And hopefully you take security seriously going forward into the future.

Yeah, and we're already seeing the results right now. Like, people are already starting to update some of their things, and it's from people that you'd be surprised to hear, I guess. So, uh, your cluster could be insecure today. Sign up for cluster— you know what, what it— no, we need to— no, no, we need to do C-Max, uh, audit, right?

Call 1-800-CLUSTER-MAX and we will audit your cluster for free. Yeah.

Dude, honestly, why don't we just, we should have just zeroed, we should have just like, just bug reported everyone now. Hey, your shit sucks. Pay us money. The aggressive way.

Make this about money, dog. Thanks, man.

Hey, I'm being sarcastic. No, I'm kind of pointing out—

Didn't come across to me. Thanks for clarifying.

Whoa, whoa, whoa. Look at the good that we are doing. We're trying to update everyone's shit for Doug, come on, come with me.

We're doing something out of the goodness of our heart, please. We are. This is, this is why nothing ever happens, because cynics like you, people like us come along and help everybody, and they go, oh, dude, nothing ever happens. That's okay. That's too much of a pat on my own back. We're definitely not at the frontiers of NeoCloud security. But man, guys, get your stuff up to date, please. This, this goes to both providers and to customers, customers who are trusting people to run a cluster for you. Run this utility and then ask your provider to upgrade it. If there's demand on their side, they're gonna, from everybody, they're gonna have to build these systems and we're just gonna help everybody get better. All right. I think we're out of time. Thank you everybody for listening. No takeaways. Sam, good job. Yo, say something at the end. Come on. No, no.

Thank you guys. I mean, the point of the ClusterMax project is to make sure nothing ever happens. We're afraid that something might happen as regards security, and we want to make sure to protect against it. Have a good one.

Good job. All right, take care, everyone. All right. Thanks, guys.