Kaval Shah of Pebble explains why GPU clusters waste energy by ignoring the nonlinear power-performance curve. Most inference workloads hit a saturation point where extra power doesn't produce more tokens. Pebble's system dynamically caps per-GPU power based on real-time metrics and workload patterns, running on Kubernetes or Slurm.
Chapters
Hi everyone, welcome back to GTC 2026. I'm Jordan Nanos on behalf of SemiAnalysis and Sail Media. We're here with Kaval Shah of Pebble. We're gonna talk all about GPU optimization for data centers, energy efficiency, things like that. Welcome.
Thanks. Thanks for having me.
All right, let's start with a little overview. Can you Tell everybody a little bit about Pebble and maybe what's interesting about GTC.
Yeah, so at Pebble, we work on data center energy optimizations. So what that means is we help data centers produce, AI factories produce more tokens per watt. And there's a few different ways we do it. And we've been able to like demonstrate our optimizations on inference clusters and training clusters. So yeah.
Awesome. So why don't we walk through, I guess, the different ways and inference versus training? Maybe you can pick where to start, but at a high level, my understanding is that lots of people are worried about provisioning enough GPUs to maximize all of their power because there's a very big difference between the power capacity of a data center versus the IT load versus the actual Yeah, that's right.
And that is what we discovered when we look at all the observability and telemetry of these GPU clusters is one of the key insights from our work is most of these workloads running on GPU clusters, the relationship between power and performance is it's not linear. So oftentimes you would crank up, you would think you would crank up more power in a GPU, and produce like more tokens, but there is a point on sort of like the power performance curve where it actually starts to like diminish or saturate, where you're essentially consuming, your GPU clusters are consuming more power, but not producing as many tokens. And that's the inefficiency that we exploit through our work.
Can you talk about why that might be the case? Are these memory bandwidth constrained workloads? Are these long context, pre-fill compute-constrained workloads?
Yeah, so with inference, there's a few reasons I would say. The first one being just how power, voltage, and clock frequencies work in a GPU cluster. The relationship between voltage, power, and clock frequencies is nonlinear, so that's one. It's more physics-defined. And then There is the workload itself, right? Like inference workloads are memory bound. So oftentimes like the power that is being used is to read model weights into HBM and not for like, it may not be like using, the SMs may not be utilized by prefill as much. That is also like one of the reasons why we see these inefficiencies where the decode stage is consuming a lot of the power and the SMs and the prefill is just kind of waiting for the decode to be ready.
Yeah. Makes sense. So maybe can you walk me through how Pelple helps solve that problem exactly?
Yeah, so one of the ways we do it is for every workload, we essentially, let's say running on like Kubernetes or Slurm cluster, we we look at the characteristics of the workload. So what kind of workload it is, what is the load on that workload? So some of this is also load dependent 'cause inference workloads can be sort of like bursty, spiky, random load sometimes. So some of the load is one of the factors that we look at. And then we look at, we essentially try and like plot the, understand the relationship between the workload, the power, and performance in terms of tokens or latency, and try to find a point on that curve where we could adjust the— our levers are power and clock frequency where we can dynamically cap the power and the clock frequency of these workloads.
Okay, so the scenario I guess is that a given cluster is over-provisioned, it has more GPUs than is necessary to serve the workload at the steady state that you guys are provisioning for, and then you're able to tune individual GPU nodes or maybe the PDU and data center facilities? How do you actually go about implementing the power capping that would help for this?
Yeah, so we were able to do it at the GPU level. So we essentially adjust the power caps and the clock frequencies per GPU of that workload. So if it's running on like a single GPU, we would just apply our power caps and clock frequency to that single GPU, and the same with like a multi-GPU workload as well. So in terms of like most of the observability that we get, the telemetry data that is either coming from like, say, VLLM metrics or SG Lang, whatever sort of the inference server is, And then, uh, uh, the power consumption data is coming from like NVIDIA GPUs and there, uh, some of the exporters as well. So yeah.
Okay. It makes sense. Yeah. Can you talk about actually how this gets implemented? So would somebody who's running VLLM or SG Lang need to be running a sidecar container? Would they be using the exporter metrics and nothing else? Is there like some node-level controller that you guys deploy?
Yeah, so it is essentially like a Helm install on your Kubernetes cluster. So it spins up a bunch of like DaemonSets and Kubernetes objects on your cluster so that it's able to like read some of the telemetry and essentially calculate what the optimal clock frequency and power cap is for that workload.
Cool, awesome. I was at a talk, I didn't read the actual paper, but at NeurIPS this year where they were walking through how certain kernels would perform at different levels depending on the power cap. I'm wondering if you guys have explored how this might impact the actual performance of a given workload and if there's a way to tune the workload itself, the kernel based on the power cap.
Yeah, so that's something that we certainly, We haven't gone down to the actual code or the kernels specifically. I think that is where we wanna head eventually, where we optimize some of these kernels for energy and watt. But it is something, we're more at the systems level at this point, but certainly wanna dive deeper more into the kernels as well.
Cool. And is it dynamic in that, the knobs or the levers that you guys are able to control would be able to be adjusted based on an inference workload changing, you know, some diurnal pattern through the day, or you've got, you know, a bunch of batch jobs that are scheduled at night, whereas the real-time jobs are people hitting the endpoint during the day or something like that?
Yeah, so we're constantly sort of optimizing the power caps and the clock frequencies. So it is something that our system is sort of like learning throughout the day. So when someone installs us in their cluster, in their Kubernetes cluster, we take maybe a few days to essentially learn the characteristics of that workload before we apply any power caps, because we want to be sure, like we don't want to mess up their SLAs or violate some of their compliance, SLO compliances, right?
That makes sense.
Yeah.
Is there one thing we've been talking about is, well, all that we've been talking about really is inference. And we've been talking about the cluster itself. So is there more to talk about maybe on the training side or maybe on the data center infrastructure side?
Yeah, so on the data center infrastructure side, there is like something that we've been sort of, we've also been exploring the idea of like flexible power in the data centers where If right now, like most of the data center designs designed are sort of static consumers of power from the grid, one of the ideas that we're exploring is like, what if these data centers are sort of grid responsive and the whole idea of like flexibility as well. There is 100 gigawatts of flexible power out there in the US. In the US. In the US. And if the data centers can demonstrate that they can be flexible, and that is they can curtail power for certain peak periods in the entire year, they can get access to this 100 gigawatts of flexible power for certain trade-offs. And one of the things where one of the products that we're working on is like, how do we make these AI clusters or these AI workloads sort of like responsive to the grid where we can essentially lower their power consumption without impacting SLOs or with minimal impact on performance.
Yeah, that seems like a huge thing. I'm excited for you guys to get working on that and yeah, see some of the results. So yeah, I appreciate you coming in today. We gotta wrap here, but again, this is Jordan Nanos for Sale Media and SemiAnalysis at GTC 2026, and thanks a bunch for coming in and having a chat.
