How Makora Generates CUDA Kernels That Beat Hand-Tuned Code | Researcher Conversations at GTC

Hi everyone. Thank you, Mohamed, for joining us with this interview. So yeah, can you first introduce yourself a bit?

Sure. Yeah. Thank you, Kimbo, for having me. So I am Mohamed Abdel-Fattah. I'm an assistant professor at Cornell University. I'm also the Chief Science Officer at Macora. So my work is broadly on AI performance, and at Macora, we're trying to automate as much of that AI performance engineering as we can.

So I guess my first question, maybe I think the audience would love to know, is maybe the origins of Macora, the motivation of why build an automated performance engineer, and then I think a lot of people who already know Makora, the question would be, why so the multiple name changes?

I see. Yeah, so the name change thing is because we used to be called Mako, and Mako was, you know, widely used names with other startups. One of them asked us not to use the name. They were, they asked nicely and we complied. So, and so, I mean, our thesis with Makora is that You know, we can automate a lot of the things that engineers are doing manually now. And we started with this thesis almost a couple of years ago now, before, you know, code generation took off and before, you know, agents are now really becoming that capable. And we've successfully kind of started at this code generation core where we are really trying to automate high-performance GPU kernels and high-performance kernels for other hardware. And then that's becoming a component in our system-level optimization. So that means extending to definitely inference server and also optimizing things like training pipelines and reinforcement learning rollouts and things like that. What I'm most excited about now is actually those system-level problems. And we have a bunch of interesting research that we hope to kind of really boost performance to levels that we couldn't get before.

What kind of system-level problems are you— do you find interesting recently?

Yeah, so I mean, we at Pacora, we have kind of still the pragmatic approach of hyperparameter tuning, adapting to the underlying hardware, but as kind of Chief Science Officer and also as being fundamentally a researcher, I'm looking into kind of newer algorithms as well. One new inference algorithm that we're very excited about is called sequential Monte Carlo speculative decoding. And so the way that works is that, you know, in normal speculative decoding, you have a draft model. So you draft one token at a time, and then you take K tokens and you verify them in parallel with a larger model. And that's how you get speedup because you're using that large model in parallel instead of autoregressively. However, one problem is that any tokens that don't match the target model, you rewind the draft and then you start again. And so this rewinding is just a big performance hit. In sequential Monte Carlo, what we do instead is that we keep n drafts alive. So we have n drafts going in parallel, and instead of verifying with the target model and looking for an exact match, we're actually just scoring those n drafts with the target model. And based on that importance sampling, based on that importance score, we do some kind of sequential Monte Carlo sampling where we can evict the ones with a very low score and duplicate the ones the drafts with a very high score. And if we do that again and again, it turns out, well, we're always accepting K tokens. Yes, we have the extra compute for multiple drafts, but we're getting much faster performance than normal speculative decoding.

So if I'm understanding correctly, it sounds like a way of increasing the number of draft tokens, more specifically in parallel, while guaranteeing a higher acceptance rate? Is that like a—

Yeah, we're, yeah, yeah, you're exactly right. So we, we are utilizing the draft model more. Think of it as if you're solving a math problem, you know, pass at n is always better than pass at 1. And so we have kind of n trials happening from this draft model, and we're always accepting K tokens. We're never rejecting any token or rolling back. And that's fundamentally why we can get a higher speedup. However, to account for the fact that we're never rewinding, we have those N drafts going in parallel and we're always keeping the particles or the drafts that have a higher chance of matching the target. And that's due to kind of the scoring that happens at the target model. And so we've, we've tested this methodology kind of compared to SG Lang and compared to VLLM. In our testing, SG Lang was much faster in speculative decoding. And so that's what we're comparing to.

Hmm.

We're about 5 times faster than SG Lang baseline and about 2 times faster than speculative decoding on SG Lang with their experimental overlap scheduler. And we're also quite a bit faster than SSD, which is an even newer baseline that also has this notion of speculative drafts that was recently published by other folks.

Based on what you shared about the results, that does sound very almost unreal, like a boosting of so much.

Of course. I mean, like, Like any results, there are a bunch of caveats. The result that I just quoted is at batch size of 1. And so this is really targeting the really low latency regime.

Right.

Obviously also because we're increasing compute. At low batch size, we were utilizing unused compute because all of this is memory bound. Even the draft model is memory bound. But at high batch size, things start to get a bit more complex. Basically, we saturate compute a bit earlier than normal speculative decoding. So like any result, there are some caveats, but focusing on that, low-batch, low-latency regime, we are able to do really well. We're also fundamentally a lossy method, and that can be like a big turnoff for many people who are interested in maintaining kind of exact accuracy. But again, in some of our empirical testing, we even outperform the target model. And so, um, interesting. Theoretically, this shouldn't be possible if the draft model is strictly worse than the target. Basically, as n goes to infinity, we should be matching the target model. We should not be outperforming it. However, because of, you know, the variance in quality of these models, sometimes having n drafts from a smaller draft model can actually outperform the target. And so we're quite excited about it. We're in the process of kind of moving this from research to product now. We should have a preprint out today or tomorrow, hopefully. And we're already kind of working on it on the Macora side. Yeah.

Another question I have is, does it require like a calibration dataset for training, like the speculators, or is it kind of more close to MTP where you do it while training the model?

I see. Yeah, that's a great question. So SMC, speculative decoding, can work on top of MTP, can work on top of EAGLE. On top of a special trained predictor. In fact, SMC is quite flexible. This isn't something we're exploiting yet. It can even operate on 2 models which have different tokenization. So there is some tricks that we can do there to allow, you know, a model with a different vocabulary to be a draft model for another target model with a slightly different tokenization, but that's still kind of a niche use of it. There are lots of kind of cool things on the research side that we're still exploring. Just, yeah, fundamentally it's compatible. with all of these speculation methods.

How do you plan to integrate this new method into the product?

Yeah, so we have an SG Lang fork that uses this. As you may know, part of our product kind of uses many open-source inference servers under the hood, chooses the best one. And so we kind of add our optimizations to the one that makes most sense. We're also actually working on our own inference engine at the moment, because again, I pointed to some of the fancier things that we want to do. And some of those are not so easy to do in SGLang and other inference engines.

Yeah, I guess it's a constant battle between cutting-edge research and production. So it's, yeah, it kind of, it'll be hard to implement some features in the production-grade serving frameworks.

Yeah.

Okay. Let me see about the auto GPU kernel generation, I guess. I think there are definitely a lot of companies doing, or a lot of people pursuing similar ideas. And I think everyone faces the issue with kernel benchmarking, like how to accurately do that. And then in worst cases, reward hacking, where the code generated kind of found loopholes in the harness and then gaming the benchmark. So I wonder how does Makora deal with this kind of issue and any insights or any pains, fun stories into this?

Yeah, there are some stories that are not so fun, including kind of maybe the worst demo experience I've ever had in my life and maybe the worst 30 minutes I've had of my life, where in the early days we were demoing— Oh, I'm so sorry. That's okay. We were demoing our tool and there was a lot of reward hacking and it tells you something about kind of how these foundation models were trained. It seems that kind of there's still, it's still far from perfect. However, since then we've kind of really bolstered our eval pipeline so that reward hacking is caught at kind of multiple steps. One of the, it is just kind of tracing through the code and making sure all of the generated functions are called properly. Some of it is using AI to detect reward hacks as well. That's usually quite an effective catch-all. We have like, I don't know, 11 different maybe classes of reward hacks that we catch that way. And then our eval pipeline is also very rigid. Like if a library is called and it shouldn't be called, then just it won't go through eval. Like we don't allow the generated code to just arbitrarily use, you know, libraries from everywhere. It should be generated code with as few dependencies as possible. And so, yeah, so I mean, none of that is super kind of fancy, I would say. It's, it's lot of engineering to get this correct. And I think we're at a stage where we're very confident in our eval pipeline. We have sold our eval pipeline to others to use in training their foundation models, at least one customer so far, and they've been fairly happy with it as well.

Interesting. Yeah, that sounds very impressive. By selling to clients, do you mean as in selling a kind of like a model Providing like a sandbox environment. Is that kind of the thing where you, instead of general CPU, you provide GPU kernel evals?

Is it something like that? Yeah, we basically expose an API, which is the same API we use internally. We just expose it to customers and give them a token for authentication. And then they use our infrastructure. Some of it uses Modo, some of it doesn't, for our kind of evaluation and profiling and all of that. So they get kind of the reward signal back, and they use that to train their model. They give us a kernel and the problem, we give them the reward.

Okay. Yeah, that, yeah, I think sandbox environments has become increasingly important or people are seeing it more in the typical RL post-training pipeline. People are seeing that inference is going very fast, training has been very fast, and then the upcoming bottleneck in the post-training pipeline will will start becoming the sandbox part. So yeah, it's very cool to see Makora doing this kind of work. So like I said, a lot of companies are doing similar things. For example, there's one called Standard Kernel Co. They're also trying to build an agent that generates GPU kernels. I wonder how do they compare to Makora or how other companies compared to Makora and what's unique, what's a unique advantage of Makora?

Yeah, I mean, when I think about our competitors, I find that most of them are like super smart and super capable competitors. I admire a lot of the work coming out of Standard Kernel, Gimlet Labs, other labs where, you know, code generation is really part of their core offering. I mean, I think one way we like to think we differentiate ourselves is by focusing on performance. We don't, I mean, at some point we thought we were going to kind of sell just a code generation agent. It's like selling a compiler. You can't, very hard to do, basically. Our value proposition is that we sell performance. You come to Macquarie, you get end-to-end performance managed on your infrastructure or any infrastructure that you want. And like I said, it will include code generation because kernels are a very big part of that performance, but it will also include kind of the latest research that we're doing on things like spec decode. Another example is, you know, we We're working on numerics, obviously low precision is super important. And so some, again, of our research that's hopefully making it to some of the product at Quora is some research we have on making FP4 more accurate. And that's the kind of research that really needs very in-depth kernel work to work well. So, so in this case, I promise I'll be brief, but basically in FP4, you can represent -0 and +0. Can represent both polarities of 0, and it's a wasted quantization level. And you have 16 quantization levels and you're wasting one of them to represent 0 in 2 ways. And so we remap the redundant 0, we called it RAZR, redundant 0 remapping, to a learnable special value. And this effectively gives you the accuracy of FP5 or FP5 point something in the memory footprint of FP5. And we went and implemented this on NVIDIA. The weight-only quantization is fine because you upcast anyways later. And so Remember that special value can't be FP4 anymore. It has to be a higher precision. Otherwise it doesn't really make sense. So weight-only kernels are fine, but when we wanted to use NVFP4 and the Tensor Cores for weight activation quantization, we basically incur a big performance overhead. And the reason is that, you know, you can no longer can get away with doing one pass of Tensor Core on this. You have to do a first pass. And then another pass to compensate for all of the entries that were negative 0 and remapped to something outside of the FP4 range. This is the most naive solution. The second pass is very sparse, and so it has a high potential for acceleration, but fundamentally we're going to be slower than FP4. However, that's not true on AMD. So on AMD, the FP6 datapath shares the hardware with FP4. And so actually with the same throughput, We can upcast to FP6, still have the memory footprint of FP4, but upcast those special values and use FP6. Whereas in NVIDIA, FP6 shares a data path with FP8. And so now this like very niche unused precision FP6 is coming to our rescue because we can show, you know, high performance for this special fancy Razer FP4, but only on AMD right now, unless NVIDIA changes their hardware. So anyways, that was a long-winded way of answering your original question of kind of differentiation with competitors and so on. We want to deliver end-to-end performance and accuracy. And, you know, we have a bunch of these cool research ideas up our sleeve and we focus on delivering, you know, the end-to-end platform and the deployment engine of models as opposed to a compiler, which, which is fundamentally what code generation is.

Yeah, that makes sense. By offering performance instead of just a compiler or just a coding agent, I guess you have more flexibility of what you can do and what techniques you can use. And I find that I enjoy that, the FP6 thing. I find that very interesting. I recognize that AMD has like different FP6 FLOPS compared to NVIDIA. So It's interesting that you guys can get better performance on AMD, which is typically, um, are not seen as, as often, I guess.

The AMD hardware fundamentally is, is quite strong. I think it's the software needs to catch up in some places and we hope to kind of fill some of that gap as well.

Yeah, totally. On the kernel writing part, I know your, your Macora wants to provide performance in general. But if we only talking about GPU kernel generation or even some other more general performance-related optimizations, I think we can expect that frontier models would get better at them. I think 6 months ago, probably AI frontier models can't even write GPU kernels properly, but right now they're almost able to write cuDSL kernels. So that's like a very big performance improvement, and I think it's reasonable to expect that Frontier's model will continue to improve. So in this timeline or in this situation, where does— how does Makora fit into this situation, and does Makora plan to adapt in any way or anything like that?

Yeah. I mean, I found that more and more as time passes by and as models and You know, foundation model agents become better and better, the less we have to do on specifically the code generation part. And so we're always trying out new models, always trying out new agents and integrating them into our product. And so we're not, we're very egoless in that regard, in that we have, you know, multiple agents that our product will try and it will just deliver the best kernel. That's the goal there. And as long as that's kind of can be used as a tailwind to kind of improve our end-to-end performance, that's what we care about. We also want to, we're also looking at open source models. And so recently one of our researchers tried out the Gemma 4 model. I think it was recently released and we're getting really good performance out of it as well. And so like fine-tuning and specializing these models is still something we're looking into just to be kind of less dependent and provide a, you know, less expensive alternative to these large foundation models. And we're kind of keeping an eye on how fast these open source and available models are catching up. But our philosophy is whatever gets you performance, we use. And if that means just using something off the shelf within our own software ecosystem, that's fine by me, as long as it delivers the best kernel. More and more though, like if I were to add to that a bit, Generating a kernel is one thing and generating a kernel that's integrated into end-to-end software, whether it's a library, GGML for some Qualcomm NPU, or if it's SG Lang for an AMD GPU, that's a slightly different thing. We have to care about data layouts, interfaces, quantization modes, and there are— these details kind of add up in the end. And that's the difference between kind of foundation model generating code or our tool generating a kernel that's ready to be integrated into the end solution.

I see. I see. So Macora tries to be more tool agnostic, I guess. Is that like a right way to—

Yeah, we're very foundation model supplier kind of agnostic.

I see. And so I guess like a next question of that would be, if that's the case, What would be the major reason clients would reach out to, or customers would reach out to, uh, Makora? Or more concretely, what are the major reasons right now, um, that clients are reaching out to Makora for service or for optimizations instead of just paying Claude or paying, paying, uh, uh, ChatGPT for, and just let it run and try to solve it?

Yeah, that's a good question. I mean, our customers are kind of the most kind of foundational layer are hardware vendors that want to build out software support for their hardware. We've worked with large hardware companies that, you know, had literally just released a programming language for their NPU. And we were able to generate kernels in that programming language and building out kind of their primitive library. Other vendors that would like to build that extra layer up to VLLM or SG Lang inference server in that stage. And then there are all, so that's kind of one class of customers. You want to build out support for your hardware and you don't want to train like a developer network for it. And yes, you can do it yourself internally, but I was just chatting with one of those customers and they were saying that internally one engineer is able to use kind of foundation models to generate kernels, whereas we're able to deliver it kind of more consistently and constantly using our tool. So automating all these tidbits around kernel generation, there is still value there in having an end-to-end product. And like I said, it's like building a compiler versus just generating an example code or a proof of concept. But again, we realized that that is less sustainable than focusing on performance in terms of a business model. And so the other class of customers are enterprise customers. So these guys have bought just a bunch of GPUs and they have no idea of getting high performance out of them. They don't have an in-house performance team. And these customers are a lot, or they have access or they want to host their own models. They cannot use public APIs and they want to make that available to their employees. And so basically maximizing the performance out of infrastructure becomes kind of the next level of customers. We're also chatting with NeoClouds who are offering hardware. Sometimes they have their own token factories or their own token APIs, and sometimes they kind of just rent out GPUs and they want to have, you know, a go-to software solution to point their customers at. And for all of these, that's kind of us expanding out to the system level and, you know, going beyond kernel generation to system-level performance. And so. Yeah. So that system-level performance part, can we automate it with foundation model agents and models? Maybe, but it requires way more expertise at this stage, I would say. And so we bring in that expertise.

Yeah. Yeah. That makes a lot of sense. Yeah. I can imagine definitely a lot of enterprises would still be reluctant to put their information outside of Or give it to the companies like OpenAI and Claude without like an exclusive contract or something that they could reach out to Macora to, by having their own infra and then reaching out to Macora to build out their internal models, serving their internal models and something like that. Yeah, I guess one last question. What's next for Macora? What are the plans for next month, next 6 months, or? Yeah.

I mean, the plan is to go higher level in the stack and really have a killer kind of inference engine and then maybe expanding to training or reinforcement learning as well, like being the deployment engine for high-performance AI models. You know, we have this vision of even, you know, actually in addition to the technology and all of the cool research ideas that we're implementing, actually making it a very usable product. And so that, you know, Even like system administrators at enterprises can use this and get access to that performance quite easily by pushing buttons and stuff. And so really kind of closing the gap between, you know, I bought a GPU to I'm getting the most performance out of that GPU. That's what we're really working on and excited about these days. Then our grand vision is more on the abstraction of different hardware and there are so many opportunities now for disaggregation. To get more performance. And, you know, because we have that code generation superpower and we know how to do that, there is a lot of potential for, you know, generating the right code for the right hardware and disaggregating your workload on different machine types. Speculative decoding is a perfect example of that draft target disaggregation in the case of SMC as well. So yeah, lots of opportunities there to kind of first abstract away performance engineering and then abstract away the hardware infrastructure altogether. And use, you know, code generation and agents to get the most performance out of it.

Does Mokoro plan to open source anything in the future?

Yeah, I mean, um, all of the stuff that I talked about, you know, that's coming from the research side will be open source. And so, uh, even for this SMC stuff, we're going to release some code, uh, there and give it back to the community and we'll build kind of, um, a closed source enterprise version of it.

Sounds good. Yeah. Thank you, Mohammed, for this interview. Uh, it was really nice talking to you again.