Dylan Patel
Hi, everyone. Thank you, Mohamed, for joining us for this interview. Can you first introduce yourself a bit?
Mohamed Abdelfattah
Sure. Thank you, Dylan, for having me. I’m Mohamed Abdelfattah. I’m an assistant professor at Cornell University, and I’m also the chief science officer at Makora. My work is broadly focused on AI performance, and at Makora, we’re trying to automate as much of that AI performance engineering as we can.
Dylan Patel
My first question, which I think the audience would love to know, is about the origins of Makora and the motivation for building an automated performance engineer. For people who already know Makora, the question is probably: Why the multiple name changes?
Mohamed Abdelfattah
The name change happened because we used to be called Maeko, and Mako was a widely used name by other startups. One of them asked us not to use the name. They asked nicely, and we complied.
1. Makora Automates Performance
Our thesis with Makora is that we can automate a lot of the things engineers are doing manually now. We started with this thesis almost 2 years ago, before code generation took off and before agents became as capable as they are now. We started with a code-generation core, where we were trying to automate high-performance GPU kernels and high-performance kernels for other hardware, and that’s becoming a component in our system-level optimization.
That means extending to inference servers, as well as optimizing things like training pipelines, reinforcement-learning rollouts, and things like that. What I’m most excited about now is actually those system-level problems. We have a bunch of interesting research that we hope will boost performance to levels we couldn’t reach before.
Dylan Patel
What kinds of system-level problems have you found interesting recently?
Mohamed Abdelfattah
At Makora, we still have a pragmatic approach involving hyperparameter tuning and adapting to the underlying hardware. But as chief science officer, and as someone who is fundamentally a researcher, I’m also looking into newer algorithms.
2. Sequential Monte Carlo Decoding
One new inference algorithm that we’re very excited about is called sequential Monte Carlo speculative decoding. The way that works is that, in normal speculative decoding, you have a draft model. You draft 1 token at a time, then you take K tokens and verify them in parallel with a larger model. That’s how you get a speedup, because you’re using that large model in parallel instead of autoregressively.
However, one problem is that any tokens that don’t match the target model cause you to rewind the draft and start again. This rewinding is a major performance hit. In sequential Monte Carlo, what we do instead is keep N drafts alive. We have N drafts going in parallel, and instead of verifying them with the target model and looking for an exact match, we score those N drafts with the target model.
Based on that importance score, we do a form of sequential Monte Carlo sampling, where we can evict the drafts with a very low score and duplicate the drafts with a very high score. If we do that again and again, it turns out we’re always accepting K tokens. We have the extra compute for multiple drafts, but we’re getting much faster performance than with normal speculative decoding.
Dylan Patel
If I’m understanding correctly, it sounds like a way of increasing the number of draft tokens, more specifically in parallel, while guaranteeing a higher acceptance rate. Is that right?
Mohamed Abdelfattah
Yeah, you’re exactly right. We’re utilizing the draft model more. Think of it as solving a math problem: pass@N is always better than pass@1. We have N trials happening from the draft model, and we’re always accepting K tokens. We’re never rejecting a token or rolling back, and that’s fundamentally why we can get a higher speedup.
To account for the fact that we’re never rewinding, we have those N drafts going in parallel, and we’re always keeping the particles, or the drafts, that have a higher chance of matching the target. That’s due to the scoring that happens at the target model.
We’ve tested this methodology against SGLang and vLLM. In our testing, SGLang was much faster than speculative decoding, so that’s what we’re comparing against. We’re about 5 times faster than the SGLang baseline and about 2 times faster than speculative decoding on SGLang with its experimental overlap scheduler. We’re also quite a bit faster than SSD, which is an even newer baseline that also has this notion of speculative drafts and was recently published by other researchers.
Dylan Patel
Based on what you’ve shared about the results, that sounds almost unreal, like a major boost in performance.
Mohamed Abdelfattah
As with any results, there are a bunch of caveats. The result I just quoted is at a batch size of 1, so this is really targeting the low-latency regime. Obviously, because we’re increasing compute, at a low batch size we’re utilizing unused compute, because all of this is memory-bound. Even the draft model is memory-bound.
At a high batch size, things start to get more complex. Basically, we saturate compute a bit earlier than normal speculative decoding. Like any result, there are some caveats, but focusing on that low-batch, low-latency regime, we’re able to do really well.
We’re also fundamentally a lossy method, and that can be a major turnoff for many people who are interested in maintaining exact accuracy. But in some of our empirical testing, we even outperform the target model. Theoretically, this shouldn’t be possible if the draft model is strictly worse than the target.
Basically, as N goes to infinity, we should be matching the target model, not outperforming it. However, because of the variance in the quality of these models, sometimes having N drafts from a smaller draft model can actually outperform the target. We’re quite excited about it.
We’re in the process of moving this from research to product now. We should have a preprint out today or tomorrow, hopefully, and we’re already working on it on the Makora side.
Dylan Patel
Does it require a calibration dataset for training the speculators, or is it closer to MTP, where you do it while training the model?
Mohamed Abdelfattah
SMC speculative decoding can work on top of MTP, on top of EAGLE, or on top of a specially trained predictor. In fact, SMC is quite flexible. This isn’t something we’re exploiting yet, but it can even operate on 2 models that have different tokenizations.
There are some tricks we can use to allow a model with a different vocabulary to be a draft model for another target model with a slightly different organization. That’s still a niche use of it. There are lots of interesting things on the research side that we’re still exploring, but fundamentally, it’s compatible with all of these speculative decoding methods.
Dylan Patel
How do you plan to integrate this new method into the product?
Mohamed Abdelfattah
We have an SGLang fork that uses this. As you may know, part of our product uses many open-source inference servers under the hood, chooses the best one, and adds our optimizations to the one that makes the most sense.
We’re also working on our own inference engine at the moment because, again, I pointed to some of the fancier things we want to do, and some of those aren’t easy to implement in SGLang and other inference engines.
Dylan Patel
Yeah, I guess it’s a constant battle between cutting-edge research and production. It’ll be hard to implement some features in the production-grade serving frameworks.
About the automatic GPU kernel generation, I think there are definitely a lot of companies doing this, or a lot of people pursuing similar ideas. Everyone faces the issue of kernel benchmarking, specifically how to do that accurately. In the worst cases, there’s reward hacking, where the generated code finds loopholes in the harness and then games the benchmark.
How does Makora deal with this kind of issue? Do you have any insights, pains, or fun stories about it?
3. Stopping Kernel Reward Hacking
Mohamed Abdelfattah
There are some stories that aren’t so fun, including maybe the worst demo experience I’ve ever had in my life—perhaps the worst 30 minutes of my life. In the early days, we were demoing our tool, and there was a lot of reward hacking. That tells you something about how these foundation models were trained. It seems that they’re still far from perfect.
Since then, we’ve really bolstered our evaluation pipeline so that reward hacking is caught at multiple steps. One of those steps is tracing through the code and making sure all of the generated functions are called properly. Some of it involves using AI to detect reward hacks as well. That’s usually quite an effective catch-all.
We have maybe 11 different classes of reward hacks that we catch that way. Our evaluation pipeline is also very rigid. If a library is called and it shouldn’t be called, then the code simply won’t go through evaluation. We don’t allow the generated code to arbitrarily use libraries from everywhere. Generated code should have as few dependencies as possible.
None of that is particularly fancy, I would say. It’s a lot of engineering to get this correct, and I think we’re at a stage where we’re very confident in our evaluation pipeline.
We have sold our eval pipeline to others to use in training their foundation models. At least 1 customer so far, and they've been fairly happy with it as well.
Dylan Patel
Interesting. Yeah, that sounds very impressive. By selling to clients, do you mean selling a model provider, like a sandbox environment? Is that the thing where, instead of general CPU, you provide GPU kernel evals?
Mohamed Abdelfattah
We basically expose an API, which is the same API we use internally. We expose it to customers and give them a token for authentication. Then they use our infrastructure. Some of it uses models, and some of it doesn’t, for our evaluation and profiling and all of that. They get the reward signal back and use that to train their model. They give us a kernel and a problem, and we give them the reward.
Dylan Patel
Okay. Yeah, I think sandbox environments have become increasingly important, or people are seeing them more in the typical RL post-training pipeline. People are seeing that inference is going very fast. Training has been very fast, and the upcoming bottleneck in the post-training pipeline will start becoming the sandbox part. So, it's very cool to see Makora doing this kind of work.
A lot of companies are doing similar things. For example, there's one called Standard Kernel Co. They're also trying to build an agent that generates GPU kernels. I wonder how they compare to Makora, or how other companies compare to Makora, and what a unique advantage of Makora is.
4. Differentiating Through Performance
Mohammad
When I think about our competitors, I find that most of them are super-smart and super-capable competitors. I admire a lot of the work coming out of Standard Kernel, Kimera Labs, and other labs where code generation is really part of their core offering.
One way we like to think we differentiate ourselves is by focusing on performance. At some point, we thought we were going to sell just a code-generation agent. It's like selling a compiler; it's very hard to do, basically. Our value proposition is that we sell performance. You come to Makora, and you get end-to-end performance managed on your infrastructure or any infrastructure that you want.
As I said, it will include code generation because kernels are a very big part of that performance, but it will also include the latest research that we're doing on things like SpecT code. Another example is that we were working on numerics. Obviously, low precision is super important, and some of our research that's hopefully making it into the Makora product is research on making FP4 more accurate.
That kind of research really needs very in-depth kernel work to work well. In FP4, you can represent negative zero and positive zero. You can represent both polarities of zero, and it's a wasted quantization level when you have 16 quantization levels and you're wasting 1 of them to represent zero in 2 ways.
We remap the redundant zero—we call it TRACER, redundant zero remapping—to a learnable special value. This effectively gives you the accuracy of FP5 or FP5-something in the memory footprint of FP4. We implemented this on NVIDIA. Weight-only quantization is fine because you upcast later anyway.
Remember that the special value can't be FP4 anymore. It has to be a higher precision; otherwise, it doesn't really make sense. Weight-only kernels are fine, but when we wanted to use NVFP4 and the tensor cores for weight-activation quantization, we incurred a big performance overhead.
The reason is that you can no longer get away with doing 1 pass of the tensor cores on this. You have to do a first pass and then another pass to compensate for all of the entries that were negative zero and were remapped to something outside of the FP4 range. This is the most naive solution. The second pass is very sparse, so it has high potential for acceleration, but fundamentally we're going to be slower than FP4.
However, that's not true on AMD. On AMD, the FP6 data path shares the hardware with FP4. With the same throughput, we can upcast to FP6, still have the memory footprint of FP4, and upcast those special values and use FP6.
Whereas in NVIDIA, FP6 shares the data path with FP8. This very niche, unused precision—FP6—is coming to our rescue because we can show high performance for this special, fancy TRACER FP4, but only on AMD right now, unless NVIDIA changes its hardware.
That was a long-winded way of answering the original question about differentiation from competitors and so on. We want to deliver end-to-end performance and accuracy. We have a bunch of these cool research ideas up our sleeve, and we focus on delivering the end-to-end platform and the deployment engine for models, as opposed to a compiler, which is fundamentally what code generation is.
Dylan Patel
Yeah, that makes sense. By offering performance instead of just a compiler or just a coding agent, I guess you have more flexibility in what you can do and what techniques you can use. I find the FP6 thing very interesting. I recognize that AMD has different FP6 FLOPS compared with NVIDIA, so it's interesting that you guys can get better performance on AMD, which typically isn't seen as often, I guess.
Mohammad
The AMD hardware is fundamentally quite strong. I think its software needs to catch up in some places, and we hope to fill some of that gap as well.
Dylan Patel
Yeah, totally. In the kernel-writing part, I know Makora wants to provide performance in general, but are we only talking about GPU kernel generation, or are there other, more general performance-related optimizations? I think we can expect our frontier models to get better at them.
6 months ago, AI frontier models probably couldn't even write GPU kernels properly. Right now, they're almost able to write QDSL kernels. That's a very big performance improvement, and I think it's reasonable to expect that frontier models will continue to improve.
In this situation, where does Makora fit, and does Makora plan to adapt in any way?
5. Makora Stays Model Agnostic
Mohammad
More and more as time passes, and as models and foundation-model agents become better and better, the less we have to do specifically on the code-generation part. We're always trying out new models, always trying out new agents, and integrating them into our product.
We're very ego-less in that regard. We have multiple agents that our product will try, and it will just deliver the best kernel. That's the goal. As long as that can be used as a tailwind to improve our end-to-end performance, that's what we care about.
We're also looking at open-source models. Recently, 1 of our researchers tried out the Gemma 4 model. It was recently released, and we're getting really good performance out of it as well. Fine-tuning and specializing these models is still something we're looking into, just to be less dependent and provide a less expensive alternative to these large foundation models.
We're keeping an eye on how fast these open-source and available models are catching up. Our philosophy is: whatever gets you performance, we use. Whether that means using something off the shelf or something within our own software ecosystem, that's fine by me as long as it delivers the best kernel.
More and more, though, generating a kernel is 1 thing, and generating a kernel that's integrated into end-to-end software—whether it's a library, GGML for some Qualcomm NPU, or SGLang for an AMD GPU—is a slightly different thing. We have to care about data layouts, interfaces, and quantization modes. These details add up in the end, and that's the difference between a foundation model generating code and our tool generating a kernel that's ready to be integrated into the end solution.
Dylan Patel
I see. So, Makora tries to be more tool-agnostic, I guess. Is that the right way to think about it?
Mohammad
Yeah, we're very foundation-model-supplier-agnostic.
Dylan Patel
I see. If that's the case, what would be the major reason clients would reach out to Makora? More concretely, what are the major reasons right now that clients are reaching out to Makora for service or optimizations instead of just paying Claude or paying GPT and letting it run and try to solve it?
6. Why Customers Choose Makora
Mohammad
That's a good question. At the most foundational layer, our customers are hardware vendors that want to build out software support for their hardware. We've worked with large hardware companies that had literally just released a programming language for their NPU, and we were able to generate kernels in that programming language and build out a primitive library.
There are other vendors that would like to build that extra layer up to a vLLM or SGLang inference server. That's 1 class of customer: they want to build out support for their hardware, and they don't want to train a developer network for it.
Yes, you can do it yourself internally, but I was just chatting with 1 of those customers, and they were saying that internally, 1 engineer is able to use foundation models to generate kernels, whereas we're able to deliver it more consistently and constantly using our tool.
So, automating all these tidbits around kernel generation, there is still value in having an end-to-end product. And like I said, it’s like building a compiler versus just generating example code or a proof of concept. But again, we realized that that is less sustainable than focusing on performance in terms of a business model.
And so, the other class of customers are enterprise customers. These guys have bought a bunch of GPUs, and they have no idea how to get high performance out of them. They don’t have an in-house performance team. These customers either have access to their own models or want to host their own models. They cannot use public APIs, and they want to make those models available to their employees.
And so, basically, maximizing the performance out of infrastructure becomes the next level of customers. We’re also chatting with neoclouds that are offering hardware. Sometimes they have their own token factories or their own token APIs, and sometimes they just rent out the GPUs and want to have a go-to software solution to point their customers at.
And for all of these, that’s us expanding out to the system level and going beyond kernel generation to system-level performance. So, can we automate that system-level performance with foundation model agents and models? Maybe, but it requires way more expertise at this stage, I would say. And so, we bring in that expertise.
Dylan Patel
Yeah, yeah, that makes a lot of sense. I can imagine a lot of enterprises would still be reluctant to put their information outside of, or give it to, companies like OpenAI and Claude without an exclusive contract or something. They could reach out to Makora, have their own infrastructure, and have Makora build out and serve their internal models and something like that.
Yeah, I guess one last question: What’s next for Makora? What are the plans for next month, the next 6 months, or even next year?
7. Makora Moves Up The Stack
Mohammed Abdelfattah
The plan is to go higher level in the stack and really have a killer inference engine, and then maybe expand to training or reinforcement learning as well—being the deployment engine for high-performance AI models.
We have this vision of actually making it a very usable product, in addition to the technology and all of the cool research ideas that we’re implementing. So even system administrators at enterprises can use this and get access to that performance quite easily by pushing buttons and stuff. We’re really working on closing the gap between “I bought a GPU” and “I’m getting the most performance out of that GPU.” That’s what we’re working on and excited about these days.
Then our grand vision is more about abstracting different hardware. There are so many opportunities now for disaggregation to get more performance, and because we have that code-generation superpower and we know how to do that, there is a lot of potential for generating the right code for the right hardware and disaggregating your workload across different machine types.
Speculative decoding is a perfect example of that: draft-target disaggregation in the case of SMC as well. So, there are lots of opportunities to first abstract away performance engineering, then abstract away the hardware infrastructure altogether, and use code generation and agents to get the most performance out of it.
Dylan Patel
Does Makora plan to open-source anything in the future?
Mohammed Abdelfattah
Yeah, all of the stuff that I talked about that’s coming from the research side will be open-source. So even for this SMC stuff, we’re going to release some code there, give it back to the community, and build a closed-source enterprise version of it.
Dylan Patel
Sounds good. Yeah, thank you, Mohammad, for this interview. It was really nice talking to you again.