Dylan explains to @lexfriedman how large-scale operations cause electricity grid to become unstable

When you talk about the scale of data center power, you can blow stuff up so easily. But I think one story that maybe exemplifies how insane this stuff is, is when you're training, you're always doing— you're running through the model a bunch, in the most simplistic terms, running through the model a bunch, and then you're going to exchange everything and synchronize the weights. So you'll do a step. This is like a step in model training. At every step, your loss goes down, hopefully, and it doesn't always. But in the simplest terms, You'll be computing a lot and then you'll exchange, right? The interesting thing is GPU power is most of it. Networking power is some, but it's a lot less. But so while you're computing, your power for your GPUs is here. But then when you're exchanging weights, if you're not able to overlap communications and compute perfectly, there may be a time period where your GPUs are just idle and you're exchanging weights and you're like, hey, the model's updating. So you're exchanging the gradients, you do the model update, and then you start training again. The power goes, right? And it's super spiky. Yeah. And so funnily enough, when you talk about the scale of data center power, right, you can blow stuff up so easily. Um, and so Meta actually has accidentally opened upstream something to code in PyTorch where they added an operator. And I kid you not, whoever made this, like, I want to hug the guy because it says, says PyTorch, uh, it's like PyTorch.powerplant_no_blowup equals 0 or equal 1. And what it does, what it does is amazing, right? Either, you know, when you're, when you're exchanging the weights, The GPU will just compute fake numbers so the power doesn't spike too much. And so then the power plants don't blow up because the transient spikes like screw stuff up.

That just tells you how much power you're working with. I mean, it's insane.