This content was created by Dylan Patel. This content was uploaded by Dylan Patel. This is a YouTube researcher conversation titled "Bryan Shan x Cameron Quilici | Researcher Conversations at GTC" (recorded April 7, 2026) featuring Bryan Shan and Cameron Quilici. The discussion focuses on InferenceX/InferenceMax benchmarking: open-source benchmark methodology and goals (Pareto frontier, reproducibility), NVIDIA vs AMD inference performance, LPU and disaggregated inference, KV‑cache offloading and long‑context serving, MoE and evolving model architectures, plans for InferenceX v3 (agentic multi‑turn benchmarks and prefix caching), and collaboration with chip vendors. The hosts also point listeners to inferencex.com for results.
Chapters
What's up, Brian? How's it going?
Hi.
Yeah, Brian and I both work on Inference Max, so it's called Inference X. All right, because we got sued by HBO Max. Just kidding, that's a joke. But yeah, I guess like, let's just kind of talk through like what it is and like what we've been working on and kind of its influence. I guess today, you know, you kind of saw, you know, Jensen go and talk about it and declare himself the inference king. I guess, what does it mean to you to kind of be the inference king and have the highest throughput chip? What do you think is important about that in today's pattern?
I mean, yeah, Jensen's probably pretty proud. Across all of our benchmarks, NVIDIA chips get the highest scores, the lowest cost for each per token, et cetera. So it's really The king of inference, I guess.
I guess, like, why do you think that? I think a lot of people, right, like on X and stuff say, oh, well, like Dylan and SemiAnalysis are just like, you know, kind of like praising NVIDIA, but like explain to people how this is truly like an independent third-party benchmark and it's not like other suites where like we have like rules and stuff and people can hill climb, like kind of explain your thoughts on that.
Yeah. So I guess the biggest part is that we are open source. I mean, if there are people who think that AMD can do better, they can feel free to just submit runs or submit configs, raw configs, and we can run it for them. But so far all the data has been showing that NVIDIA does in fact surpass AMD in almost all aspects. Yeah. Yeah.
I think that's a good point. The idea of InferenceX is sort of to provide a benchmark such that when people are hill climbing it, it's making the entire ML community better and more efficient and stuff. So yeah, I'd say to add to your point, right, it's completely open source and anyone is welcome to kind of contribute to it and just make the entire ecosystem better. So yeah, we always want to be really transparent about that. But yeah, I think that makes it even more prominent when a chip provider can be like, yeah, we have the fastest chip.
Yeah. Actually, I want to know more about what's it like working with AMD and NVIDIA because you've been here longer than me. So how's the experience actually?
Yeah, actually it's been really good. And Inferentia X would not be possible without their engineers. They do a lot of the tuning of the configs and stuff. And yeah, I think it's been really a joy to work with a lot of them. You know, and yeah, it's been really great. I guess we can talk about, so like, what do you think about the LPU stuff, right? Because that's kind of like a hot topic right now. And like it directly kind of ties into like the Pareto frontier that we show on InferentX. Like what are your thoughts on that just in general?
Oh yeah, Jensen did just announce that earlier, I guess. I mean, NVIDIA does know the most about its customers. Of course, the customers would directly ask NVIDIA for stuff that will improve their inference performance. So I guess in a way, disaggregated inference is the future. And to really optimize each specific part of disaggregated inference is that this LPU does in fact surpass GPU in some aspects. And that's why NVIDIA developed it, already acquired Groq and Uh, used it instead.
Yeah.
Yeah.
I think it's really interesting that instead of just like using the, the Groq rack to do inference, like across the whole curve, they're really just leaning into the, the strengths of Groq and just saying like, okay, well it can do really fast at a high interactivity and we're just going to do that like co-packaged with, with the Ruben system. And like, I wonder if like, do you think we'll see that from other alternative chips like Maddox and And others, or do you think, will this be a continuing trend? I mean, maybe not with NVIDIA specifically, but.
I mean, it'd be interesting to see. There are theories that like Amazon is using NVL72 along with their own chips for disaggregated inference. Like there's really a lot of space to optimize each part, especially since it's disaggregated.
Yeah. Yeah. I agree. And especially with like new architectures, like obviously MOTE, mixture of experts, isn't new, but as we see new architectures, right? Like I think inference, Really, like Jensen was talking about today, there's a lot of opportunity for optimizations and like you can really disaggregate a lot of the inference parts. So I think I'm really excited to see like what comes up and what we can actually showcase on InferenceX, right? Really, I hope we can get like some alternative chip providers on for us.
Of course. Yeah. And the trend is towards, I guess, towards like everyone's trying to develop their own chips. OpenAI has their Titan in development and some other trends also like regarding the model specs. We are seeing MoE models being more sparser with smaller experts. And of course this plays into YEP. So it's quite interesting to see that the researchers who are planning model architectures actually care more about inference optimizations. I used to think this was only a DeepSeek thing, like only DeepSeek really cared about these inference optimizations. But yeah, it seems like inference performance is getting more and more important and it's being considered along with the model's actual performance.
Right.
Okay, so tell me what plans you have for InferenceMax.
Yeah, yeah, that's a good question. So yeah, right now with InferenceMax, we're using random data for all of our benchmarks, which is actually, Quite good for a baseline benchmark, right? So when you think about it, the way I like to explain it is, in a way, right now, the benchmarks are kind of showing like the worst the chips will actually perform, right? Because it's just a baseline. There's no prefix caching because we don't do multi-turn datasets. It's all random. And then speculative decoding acceptance rates are quite low because it's the random data. But for InferenceX V3, you know, we want to add— agentic benchmarks, like really, really good, like representative multi-turn QA chat benchmarks where there are a ton of client sessions, each with multiple turns and will enable prefix caching. And like to your point, right? Like NVIDIA is beating AMD across the whole curve, like you said, or most parts, but like AMD MI355X versus B200, MI355 has 1.5 times the HBM bandwidth capacity. So. with prefix caching, right? Like we'd expect it to do better possibly. So I think it will help showcase the chips from a more— Yeah, nuanced, I was going to say like perspective, right? And like capturing all the parts of each chip. But yeah, I think actually making a good agentic benchmark is something that we're realizing is quite difficult. And something that hasn't really been done publicly yet. So that's a lot of work, but yeah.
Oh yeah. And I've heard like doing GTC after parties or like events, I've been speaking to people and it seems like KVCache offloading and KVCache discount strategies are getting more and more popular.
Yeah, yeah. KVCache offloading is something we've been investigating a lot, like VLLM just Basically in VLM 0.16, I think they made native KV cache offloading. You have things like LM cache, distributed offloading, like with LLMD and stuff. And I think, you know, we were talking to a lot of people at the hackathon the other day, like doing this as like their project and stuff. And yeah, I think, I mean, at the end of the day, when you want long context inference, you need a big KV cache and you also need that to be extremely fast. So there's not really enough room just in HBM. You need to offload it somewhere. So yeah, I mean, I don't know, like, what do you think about this topic? Like, do you think like basically the future is just going to be like how fast we can offload KV cache and how much we can offload? Or do you think it's bigger HVM or what do you think?
I mean, I'm very interested. To be honest, I'm very interested to see what the Frontier Labs are actually doing. But I mean, no one knows and open source is probably behind those labs by like half a year, maybe at best. Yeah.
Yeah, I mean, that's a good point because like Opus 4.6, you know, 1 million context is the default now, right? So like, how are they serving like hundreds of thousands of users with 1 million context? Like it's quite interesting. So yeah, I mean, I don't know, there are a lot of interesting topics to cover and like the goal of InferentX, I think both of us have this like policy where we just want to make the most real world benchmark possible. And we don't want to force people to use certain optimizations and we just want to showcase like the optimal Pareto frontier. So whether that's using, you know, AFD or, you know, like KVCache offloading, like we just, we just want to show it the most honest way possible. But yeah, I think, I think that about wraps up unless you have anything else to say.
Yeah.
So I think anyone listening to this podcast should definitely Uh, check out inferencex.com to see all these results for free. Beautiful visualization. We got a great frontend developer, Alec. Shout out Alec. And yeah, that's it. Thanks. Thanks for talking, Brian.
