Training a 400B Model on 2,048 Blackwell GPUs for $20M | Researcher Conversations at GTC

Hi, Lukas. Thank you for joining us for this interview. It's, personally, I'm a big fan of RCAI's work. I'm very honored to meet you. So yeah, can you first introduce yourself, please?

Yeah, my name is Lukas Atkins. I'm the CTO and head of research over at RCAI. You know, I have worked in the open source AI space for a long time. Previously on the post-training side almost entirely. And then we as RC made the move into pre-training our own models. Started work on it early 2025, but it didn't materialize till July 2025. And then since then we've been pre-training a lot and moving further down the stack.

First of all, some question about RC. So why a pre-training lab? or like an AGI model lab right now? And despite all the challenges, like competing in talent and everything?

I've thought on that AGI lab question. It's like, would I consider ourselves an AGI lab? Certainly our ambitions aren't currently AGI in that I don't think that as I go into the next release or planning out the next models, we're Putting it against a roadmap of how is this going to help us reach AGI. Now, certainly, you know, never say never, but it's really how could we be like the most economically viable company for developers, startups mainly, and then enterprises. The why is a little more nuanced. I think that, that there's a few answers to it. And Sometimes the answer changes depending on who the audience is, but I'll try to give you the short version of each. For us, when we were largely doing just post-training on top of other open models, so Ollama, Mistral back in the day, and then later on Quen was a very popular one because we were trying to stay sub-20 billion parameters. We reached a point where the ceiling of how well we could do on a particular job or a particular contract was largely dictated by how well the lab had pre-trained the base model. And that was fine when you had kind of really competitive options coming out of the West. You know, when Meta was kind of leading that frontier, that became a little more limiting when they stopped. And, and you have most of, if not all of the models sub-20B out of China. And we had a lot of customers who, uh, stopped wanting us to work off of a Chinese pre-trained base model, which whether or not I agreed that that was like a viable take based on the current risks didn't matter. You know, their legal teams and compliance teams reigned supreme. So that was one reason that we felt that if we were going to be able to offer the best services to customers, build the best products, um, that owning the whole stack was gonna be needed. There was also the aspect of the West lacks a company whose sole existence relies on how good their base models are, or just their pre— their, their pre-trained models. So you'll have like OpenAI did GPT-OS, which was amazing. And then NVIDIA has really spurred along with Nimetrunk.

Mm-hmm.

But those are all subprojects of a much larger company whose revenue is not dependent on that. And as long as that's the case, I do think that we're gonna struggle to have like a reliable Western open weight player and that it's continually kind of pushing, you know, what the frontier is in the West and, or open models in general. And then lastly, it enables us to do a lot more in that customization world for downstream customers when we control the whole stack. When we know what data goes into it in pre-training, that can very strongly inform what data you put into post-training, especially in the mid-training phase. I'd say that that last 10% of pre-training is some of the more consequential data for how well you do in post-training. Um, so those are the main factors that we did it as to why we did it and decided to jump in there with all of the risks and difficulties was, uh, we thought we could. I mean, we've always, we've made it to where we are as a company by picking like super ambitious things that we have no business trying to do and then like not stopping until we do it. And we felt that pre-training was a similar, uh, vertical. So yeah, those are, those are the main reasons. And then You know, I could wax poetic all day about how I think open weight models are extremely important and that, you know, having alternatives outside of the frontier that you can own and have complete control of the data that goes in and out, the way that it speaks and the way that it interacts, the way that you can, you know, interpretability research is super important there. That unless we have competitive base models, Or just like competitive open models outside of Anthropic and OpenAI, you're never like, unless you work at Anthropic or OpenAI, you're never going to get a chance to do interpretability research at that level.

So yeah, actually that's like my next question, like why open weights? Among all the reasons that you just briefly mentioned, could you pick maybe like one or two and explain why is it so important for us to have open weights?

I mean, if I'd pick one, It would be, you know, sovereignty. I think the ability for someone to completely own the pipeline is extremely important. I think that enterprise aside, which like is definitely the most strict on compliance and like data sovereignty, the ability to be in complete control of your product and development process is extremely important. So. If you are building, let's say an invoice processing app, right? You're gonna take in parsed receipts and then you want the model to then take all of those charges and the prices associated with them and put 'em in an Excel sheet. And then you want it to calculate kind of like monthly spend or something, or set it aside for your accounting team to do taxes on later. If you reach a point where margins are too low, meaning like you're actually losing money, uh, or you have no margins because you're, you know, the, the models coming out of those labs are so expensive.

Yeah.

Yet improvement on your task is not increasing over time. So the models are getting more expensive. They're taking longer. They're doing more tokens for the same task, but they're scaling like frontier mathematics, but you need them to scale like tool use. speed, cost, the ability for you to then take control and say, no, actually I'm going to take this 4 to 8 billion parameter model. I'm going to collect a bunch of my data. I'm going to format it in a way that is conducive to fine-tuning, and then I'm going to fine-tune it. And then when I run into an issue in the future, say I want to build a new feature that requires a new skillset, I'm going to go and make sure that my model's good at that too. I think it gives you a lot more control. than you have when you're kind of reliant on the whims of, you know, these kind of frontier labs, which ultimately are in a, you know, arms race for profitability. And so they're going to make sacrifices or choices that lean towards that, which aren't always in the best interest of consumers, developers, startups, and then ultimately enterprises. So I think control is the number one. I think the second is from a research perspective. Anything with kind of an unlimited cap on the potential for abundance and good, which I think AI has, necessarily kind of has the opposite potential as well, right? Like it, you know, it could be, there could be bad side effects. I think an example of that is like the mobile boom and phones and social media where like it really had this upside of like connecting the world in a way that's unprecedented. But at the same time, we're seeing the tail effects of that, which is like reduced attention span, um, you know, depression and anxiety at like the highest levels. And I think it's a similar thing that can happen with AI. There's obviously gonna be negative side effects. And if the people who are able to monitor and see at a fundamental level and what those risks are and to analyze and try to mitigate them are only 200 safety researchers at OpenAI and Anthropic. I think that from a transparency perspective, a trust perspective, and from an effectiveness at combating those risks perspective, that open models are unbelievably important. I would rather the risks be very transparent. I would rather the mitigations be extremely easy and well, you know, Well, understood. Understood and properly, what do you, properly—

Maybe like diffused?

Yeah, like the ability is easier for everyone to implement. And then I would like the smartest minds in the world able to analyze those. And I think that having that out in the open is extremely important.

Yeah, totally. I think, open source community is great evidence of how we can do greater good by collaborating, by being transparent with each other. And I think the early progress of AI definitely also benefited from open source. If AlexNet, they don't open source anything, or if PyTorch isn't open source, I don't know where we will be.

That's a scary idea.

Yeah.

We're where we are because of open weights. And open research. I mean, open weights aside, like open models aside, open research in academia. And then, you know, having that then applied at a lab level. And then, I mean, all of it, we are only where we are because stuff was done in the open and, and, uh, done for other people to judge and, and build on top of. Um, and I think that. by commercializing everything to the extent that we are, um, and having everything be this arms race of capital and talent, uh, especially, but compute as well, uh, you run the risk of, of not having those same compounding breakthroughs, right?

Mm-hmm.

You know, if, if OpenAI is, you know, able to solve continuous, you know, they have some path to continuous learning and Anthropic has their own and Google has their own. Um, it's going to, you know, saturate to the wider world much slower than if it were to happen, you know, in, in the open with artifacts for others to build on top of. Now, there's no going back. I mean, there's no world in which, um, no matter how much, you know, whining and, and advocating is done, there's no world in which these players are going to— open up their research to the degree that they once did, but not allowing us to ignore it entirely in the open world and for other players to be pushing for it, I think is going to be extremely important.

Definitely not going back. They definitely have some, from a company's perspective, they need to guard their secrets and everything. Yeah. Yeah. Another question is, So how do you plan to compete with the West AI labs in resources, i.e., compute, and also more importantly, actually talent?

Talent's actually easier than compute because compute comes down to funding, which comes down to money, which comes down to revenue, which is If, if it's not an easy problem to solve, but talent is easier in that, not that it's easy to find the talent, but retaining it and, and, and, you know, convincing someone to join a lab of, of our size has been surprisingly not as hard as I thought it was going to be. Well, and I think there's a few reasons for that. The first is that People want to work on open weights. Like, people want to work on models that are going to be out in the public where people can appreciate the research that went into it, the engineering, the infrastructure, which, um, at, you know, labs where they're serving it via APIs or downstream products, we only see the symptoms of their work instead of the actual, like, work itself, right? Like we see how quickly OpenAI and Anthropic are improving their models month to month. And you can infer that, wow, they have some amazing infrastructure engineers and wow, their researchers have really built this amazing feedback loop because we can infer that that's what it takes to have that kind of progress at that kind of speed.

Right.

But we don't actually get to see it. We don't actually get to appreciate the elegance that goes into it. We can only Um, again, kind of intuit based on the, what needed to go into it in order to have that outcome. Um, and then also when we're small, uh, and I think regardless of if we went out and raised or somehow got access to $2 trillion, I think I would still like to keep our research team under 30 people. I think that the way that you make breakthroughs and momentum be a side effect of the way you work is by having really opinionated, talented and opinionated people in a room debating in good faith. I think that is the way that you make these things become— I think you make breakthroughs in the ability to solve these hard problems that you have no business trying to solve possible and consistent. And by that I mean is if I have 30 researchers who are all involved in every step of the training process, Instead of just one specific niche within a much larger training loop, like you would get at the labs, right? Like, you know, there might be one or two or three people who are aware of and involved in every step from pre, you know, data collection and synthesis to pre-training to architecture to mid-training to SFT to, you know, but not everyone isn't. And therefore. It can be hard to, with that lack of context, to make the best decision for your niche. Whereas at RC and other labs of our size, everyone's working on every single step of the process, which makes us very malleable. I can have 5 people working on pre-training that then I need 3 of them to move over to mid-training because that team is struggling with a certain tricky challenge. I can move them around and they're able to jump in without having to like go through a big onboarding process. And people like that. I think that, you know, for people who are intellectually curious and unbelievably like love solving really hard problems, being able to understand the whole process is extremely enticing. So the talent part is one that I'm less concerned about. Now, when it comes to compute, and if you want to use compute as an aggregate for intelligence, right? Like the smarter a model is, likely the more compute went into it. I think that framing open weight models or just open source in general as needing to match or exceed the capability of these frontier labs is the wrong viewpoint. You hear often like, oh, Chinese labs are 6 months behind or they're 4 months or 3 months. And that kind of implies that they're catching up or that they're trying to exceed. And maybe in some cases they are. But I think that just over time, the distance between what frontier labs are focusing on and what open model labs are focusing on is going to be very different. Again, for instance, I'm not currently developing a team and a model to do like theorem proving or frontier math.

Right.

But OpenAI is, they're hill climbing that very heavily. So while their next model might go up 30% on frontier math, the ability for it to generate front end and therefore like slide generation for apps that are building like PowerPoint, that percentage might only go up 5.5%.

Right.

And so I think that our goal at RC is to be as reliable as possible, as fast, as inexpensive as possible, and as performant and economically viable The 80% of economically viable tasks that don't require that level of insane kind of frontier intelligence. So therefore our goal isn't to catch up. Our goal is to undercut, I guess, if that's a better way to put it. So therefore it's just a completely different compute spend and allocation than what they're doing. So I think that, Raising the funds and being able to execute down for the next couple of years is entirely feasible and well within motion. What becomes tricky is how do we navigate the kind of ebbs and flows of this industry and ensure that as compute demand goes up and down and the compute required to do any kind of new training paradigm shifts, right? RL is significantly more compute and infrastructure intensive. than what post-training used to be. You can expect that those kinds of paradigm shifts are going to continue to happen over the next months and years, that we have a loose enough and modular enough kind of research structure that we can kind of move with those and ensure that we're not spending 10 times more than what we make.

I think RC has the Trinity series models. So first of all, please let the people who still don't know what Trinity stands for appreciate to know why it's called Trinity. And then maybe talk about what is it like to work with the other companies and how is it like to train the first model with a big cluster of B300s?

Yeah, Trinity was named first and foremost because I thought it was a cool name. It ended up being There are nice reasons for it in that we work with a company called Datology for helping us curate our pre-training data. We work with Prime Intellect on infrastructure scale-up and kind of our GPU management. And having 3 companies involved in making one model felt apt for that name. And we were also making 3 models in the first generation. So there's several different reasons for the name.

Oh.

But yeah, Trinity is is mainly just like, well, you know, 3, you know, uh, 3 companies all focusing on different areas of the stack, working in tandem to build something special that we think is better than one company of our size trying to own it all. The B300s were, uh, interesting. It was, it was a, a decision made out of A, practicality and that they were available and we wanted to train as fast as possible. We wanted this pre-training to take A month, not 3. And so B300s became the next, you know, the obvious choice. It was definitely intimidating because there wasn't a lot of, outside of what NVIDIA had shared themselves or PyTorch benchmarking, there wasn't a lot of like at-scale B300 benchmarks, but also like— tools to take best advantage of them. Kernels weren't readily available, especially like super sparse MOE kernels, like what we have. You know, up until that point, we had largely been utilizing and building off of the work that had come out of other labs that had done models and shared their artifacts and how they'd done that. So like a lot of it was, you know, this, the ecosystem that built around the DeepSeek, like super sparse MOEs.

Mm-hmm.

helped us on Hopper because a lot of those models were trained on Hopper. So we could use that as like a golden example of what throughput looks like for a model of this size and this scale. And we could use that as a benchmark for how well we were doing in our own stack, based on how well others had trained models.

Thank you for letting us interview you and looking forward to the new, maybe post-trained Trinity Models. And, um, yeah, thank you again.

I appreciate it.

Thank you.