Training a 400B Model on 2,048 Blackwell GPUs for $20M | Researcher Conversations at GTC

Dylan Patel

Hi, Lucas. Thank you for joining us for this interview. Personally, I'm a big fan of Arcee's work, and I'm very honored to meet you. Can you first introduce yourself, please?

Yeah, my name is Lucas Atkins. I'm the CTO and head of research over at Arcee AI. I've worked in the open-source AI space for a long time, previously on the post-training side almost entirely. Then we at Arcee made the move into pre-training our own models. We started work on it in early 2025, but it didn't materialize until July 2025, and since then we've been pre-training a lot and moving further down the stack.

Dylan Patel

First of all, a question about Arcee: Why a pre-training lab, or an AGI model lab, right now, despite all the challenges, like competing for talent and everything?

I've thought about that AGI lab question. Would I consider ourselves an AGI lab? Certainly, our ambitions aren't currently AGI. As I go into the next release, or plan out the next models, I don't think we're putting them against a roadmap of how this is going to help us reach AGI. Certainly, never say never.

It's really about how we could be the most economically viable company for developers, mainly startups, and then enterprises. The why is a little more nuanced. I think there are a few answers to it, and sometimes the answer changes depending on who the audience is, but I'll try to give you the short version of each.

For us, when we were largely doing just post-training on top of other open models—Llama, Mistral back in the day, and then later on Qwen was a very popular one—we were trying to stay below 20 billion parameters. We reached a point where the ceiling of how well we could do on a particular job or contract was largely dictated by how well the lab had pre-trained the base model.

That was fine when you had really competitive options coming out of the West, when Meta was leading that frontier. That became a little more limiting when they stopped, and you had most, if not all, of the models below 20 billion parameters coming out of China. We had a lot of customers who stopped wanting us to work off a Chinese pre-trained base model. Whether or not I agreed that was a viable take based on the current risks didn't matter; their legal and compliance teams reigned supreme.

So that was one reason we felt that, if we were going to offer the best services to customers and build the best products, owning the whole stack was going to be necessary. There is also the aspect that the West lacks a company whose sole existence relies on how good its base models, or its pre-trained models, are. OpenAI did GPT-OSS, which was amazing, and NVIDIA has really spurred things along with Nemotron.

Dylan Patel

Mhm.

But those are all subprojects of much larger companies whose revenue is not dependent on that. As long as that's the case, I do think we're going to struggle to have a reliable Western open-weight player that's continually pushing what's at the frontier in the West and in open models in general.

Lastly, it enables us to do a lot more in the customization world for downstream customers when we control the whole stack. When we know what data goes into pre-training, that can strongly inform what data you put into post-training, especially in the mid-training phase. I'd say that the last 10% of pre-training is some of the more consequential data for how well you do in post-training.

Those are the main factors behind why we did it. The reason we decided to jump in there, with all of the risks and difficulties, was that we thought we could do it. We've always made it to where we are as a company by picking superambitious things that we have no business trying to do, and then not stopping until we do them. We felt that pre-training was a similar vertical.

I could wax poetic all day about how I think open-weight models are extremely important. Having alternatives outside of the frontier that you can own, with complete control of the data that goes in and out, the way that the model speaks and interacts, and the way that you can do interpretability research, is extremely important.

Unless we have competitive base models, or just competitive open models, outside of Anthropic and OpenAI, you're never going to get a chance to do interpretability research at that level unless you work at Anthropic or OpenAI.

Dylan Patel

Yeah, actually, that's my next question: Why open weights? Among all the reasons that you just briefly mentioned, could you pick maybe 1 or 2 and explain why it's so important for us to have open weights?

If I had to pick 1, it would be sovereignty. I think the ability for someone to completely own the pipeline is extremely important. Enterprises are definitely the strictest on compliance and data sovereignty, but more broadly, the ability to be in complete control of your product and development process is extremely important.

Let's say you're building an invoice-processing app. You're going to take in parsed receipts, and then you want the model to take all of those charges and the prices associated with them, put them in an Excel sheet, calculate monthly spend, or set it aside for your accounting team to do taxes on later.

If you reach a point where margins are too low—meaning you're actually losing money—or you have no margins because the models coming out of those labs are so expensive, your improvement on the task isn't increasing over time. The models are getting more expensive, taking longer, and using more tokens for the same task, but they're scaling on frontier mathematics, while you need them to scale on tool use, speed, and cost.

The ability for you to take control and say, “No, actually, I'm going to take this 4- to 8-billion-parameter model. I'm going to collect a bunch of my data, format it in a way that's conducive to fine-tuning, and then fine-tune it. When I run into an issue in the future—say I want to build a new feature that requires a new skill set—I'm going to make sure that my model is good at that, too.”

I think it gives you a lot more control than you have when you're reliant on the whims of these frontier labs, which ultimately are in an arms race for profitability. They're going to make sacrifices or choices that lean toward that, which aren't always in the best interests of consumers, developers, startups, and ultimately enterprises. So I think control is the number 1 reason.

I think the second is from a research perspective. Anything with an unlimited cap on its potential for abundance and good—which I think AI has—necessarily has the opposite potential as well. There could be bad side effects. An example of that is the mobile boom, phones, and social media, which had the upside of connecting the world in an unprecedented way.

At the same time, we're seeing the tail effects of that: reduced attention spans, depression, and anxiety at the highest levels. I think a similar thing can happen with AI. There are obviously going to be negative side effects, and if the people who are able to monitor and see at a fundamental level what those risks are, analyze them, and try to mitigate them are only 200 safety researchers at OpenAI and Anthropic, then from a transparency perspective, a trust perspective, and an effectiveness-at-combating-those-risks perspective, open models are unbelievably important.

I would rather the risks be very transparent. I would rather the mitigations be extremely easy and—

Dylan Patel

Well understood.

—understood and properly—

Dylan Patel

Maybe, like, diffused.

Yeah. The ability should be easier for everyone to implement, and then I would like the smartest minds in the world to be able to analyze those things. I think having that out in the open is extremely important.

Dylan Patel

Yeah, totally. I think the open-source community is great evidence of how we can do greater good by collaborating and being transparent with each other. I think the early progress of AI definitely also benefited from open source. If AlexNet didn't open-source anything, or if PyTorch wasn't open source, I don't know where we'd be.

That's a scary idea, yeah—where would we be without open weights and open research? Open models aside, open research in academia, and then having that applied at a lab level—we're all only where we are because things were done in the open, for other people to judge and build on top of.

I think that by commercializing everything to the extent that we are, and having everything be this arms race of capital and talent, especially, but compute as well, you run the risk of not having those same compounding breakthroughs.

Dylan Patel

Mhm.

If OpenAI is able to solve continual learning, and they have some path to continuous learning, while Anthropic has their own and Google has their own, it's going to saturate to the wider world much more slowly than if it were to happen in the open, with artifacts for others to build on top of.

There's no going back. I mean, there's no world in which, no matter how much whining and advocating is done, these players are going to open up their research to the degree that they once did.

Dylan Patel

But not allowing us to ignore it entirely in the open world and for other players to be pushing for it, I think, is going to be extremely important. Definitely not going back. From a company's perspective, they need to guard their secrets and everything.

Yeah.

Dylan Patel

Another question is, how do you plan to compete with the Western AI labs in resources—that is, compute—and, more importantly, talent?

Talent's actually easier than compute because compute comes down to funding, which comes down to money, which comes down to revenue, which is not an easy problem to solve. But talent is easier in that—not that it's easy to find the talent, but retaining it and convincing someone to join a lab of our size has been surprisingly not as hard as I thought it was going to be.

Well, I think there are a few reasons for that. The first is that people want to work on open weights. People want to work on models that are going to be out in the public, where people can appreciate the research that went into it, the engineering, and the infrastructure—which, at labs where they're serving it via APIs or downstream products, we only see the symptoms of their work instead of the actual work itself, right?

We see how quickly OpenAI and Anthropic are improving their models month to month, and you can infer that, wow, they have some amazing infrastructure engineers, and wow, their researchers have really built this amazing feedback loop, because we can infer that that's what it takes to have that kind of progress at that kind of speed. But we don't actually get to see it. We don't actually get to appreciate the elegance that goes into it. We can only intuit it based on what needed to go into it in order to have that outcome.

When we're small, I think that, regardless of whether we went out and raised money or somehow got access to $2 trillion dollars, I would still like to keep our research team under 30 people. I think that the way you make breakthroughs and momentum a side effect of the way you work is by having really opinionated, talented people in a room debating in good faith.

I think that is the way that you make breakthroughs and make the ability to solve these hard problems that you have no business trying to solve possible and consistent. By that I mean, if I have 30 researchers who are all involved in every step of the training process, instead of just one specific niche within a much larger training loop like you would get at the labs, there might be 1, 2, or 3 people who are aware of and involved in every step, from pre-training data collection and synthesis to pre-training, architecture, mid-training, and SFT. But not everyone is, and therefore it can be hard, with that lack of context, to make the best decision for your niche.

Whereas at Arcee AI and other labs of our size, everyone's working on every single step of the process, which makes this very malleable. I can have 5 people working on pre-training, then I need 3 of them to move over to mid-training because that team is struggling with a certain tricky challenge. I can move them around, and they're able to jump in without having to go through a big onboarding process.

People like that. I think that for people who are intellectually curious and love solving really hard problems, being able to understand the whole process is extremely enticing. So the talent part is one that I'm less concerned about.

When it comes to compute, if you want to use compute as an aggregate for intelligence—the smarter a model is, likely the more compute went into it—I think that framing open-weight models, or just open source in general, as needing to match or exceed the capability of these frontier labs is the wrong viewpoint.

You hear often, “Oh, Chinese labs are 6 months behind,” or “They're 4 months or 3 months behind.”

Dylan Patel

Yeah.

That kind of implies that they're catching up or that they're trying to exceed. Maybe in some cases they are, but I think that, over time, the distance between what frontier labs are focusing on and what open-model labs are focusing on is going to be very different.

Again, for instance, I'm not currently developing a team and a model to do theorem proving or FrontierMath, right? But OpenAI is hill-climbing that very heavily. So while their next model might go up 30% on FrontierMath, the ability for it to generate frontend and, therefore, slide generation for apps that are like PowerPoint might only go up 5.5%.

I think that our goal at Arcee AI is to be as reliable as possible, as fast and inexpensive as possible, and as performant as possible on the 80% of economically viable tasks that don't require that level of insane frontier intelligence. Therefore, our goal isn't to catch up. Our goal is to undercut, I guess, if that's a better way to put it. So it's just a completely different compute spend and allocation than what they're doing.

I think that raising the funds and being able to execute for the next couple of years is entirely feasible and well within our means. What becomes tricky is how we navigate the ebbs and flows of this industry and ensure that, as compute demand goes up and down, and the compute required to do any kind of new training paradigm shifts is significantly more compute- and infrastructure-intensive than what post-training used to be, we have a loose enough and modular enough research structure that we can move with those shifts and ensure that we're not spending 10 times more than what we make.

Dylan Patel

I think Reflection AI has the Trinity series of models. So, first of all, please let the people who still don't know what Trinity stands for know why it's called Trinity. Then maybe talk about what it's like to work with the other companies and what it's like to train the first model with a big cluster of B300s.

Yeah, Trinity was named first and foremost because I thought it was a cool name. It ended up having nice reasons for it, in that we work with a company called DatologyAI to help us curate our pre-training data. We work with Prime Intellect on infrastructure scale-up and our GPU management, and having 3 companies involved in making 1 model felt apt for that name. We were also making 3 models in the first generation.

There are several different reasons for the name, but Trinity is mainly just 3 companies, all focusing on different areas of the stack and working in tandem to build something special that we think is better than 1 company of our size trying to own it all.

The B300s were interesting. It was a decision made out of practicality: they were available, and we wanted to train as fast as possible. We wanted this pre-training to take 1 month, not 3, and so B300s became the obvious choice.

It was definitely intimidating because, outside of what NVIDIA had shared themselves or PyTorch benchmarking, there wasn't a lot of at-scale B300 benchmarking, and there also weren't many tools to take best advantage of them. Kernels weren't readily available, especially super-sparse kernels like what we have.

Up until that point, we had largely been utilizing and building off the work that had come out of other labs that had done models and shared their artifacts and how they'd done that. A lot of it was the ecosystem that built around DeepSeek-like super-sparse models that helped us on Hopper, because a lot of those models were trained on Hopper.

We could use that as a golden example of what throughput looks like for a model of this size and scale. We could use that as a benchmark for how well we were doing in our own stack, based on how well others had trained models.

Dylan Patel

Thank you for letting us interview you. I'm looking forward to the new, maybe post-trained Trinity models. And yeah, thank you again.

I appreciate it. Thank you.