Teknium（Nous Research）谈后训练、Hermes与开放模型：Delphi AMA

Who are you, what do you do, and how do we know you’re not AGI?

Hey, everybody. I’m R***, but the internet knows me as Teknium. I’m the Head of Post-Training at Nous Research, and you never know—I may be an AGI.

How would you describe Nous Research to Elon Musk? You have pre-training, models, RL environments, and more.

Nous started as a group of high-signal AI researchers and developers trying to bring out the most from open-source models, and became a company to drive a free and open future toward AGI. We do post-training, including chain-of-thought reasoning training, RL environments, agents, exploring new interfaces for model interactions, and, of course, decentralized training infrastructure using breakthrough research techniques like DisTrO.

Narrowing in on that post-training section of your answer, what does post-training mean? Can you walk us through how you actually do this work? How closely do you interface with pre-training folks?

Before a model gets to me, it starts as a pre-trained model. These models are trained on everything and anything available on the internet—Twitter posts, webpages, textbooks, and so on. They’re decent next-word predictors, but not great at understanding the role of being your assistant, co-programmer, friend, and so on.

That leaves pre-trained models in a state kind of like clay: they can be molded into anything you want. Post-training takes that clay and molds it into a smarter, more steerable, fine-tuned version that can be aligned to the morals or ethics you want, or built specifically for working with your product or other tooling. This is where post-training work happens. Post-training is where the specificity of downstream use cases and product integration begins.

The post-training team works with the pre-training team on a variety of things, including data ratios. For instance, if you want a coding model out of post-training, it’s very good to inform the pre-training team to allocate more resources to coding data. We also help specify architecture needs, like the context length you need, especially if your downstream use case is agentic or RAG-focused; modalities you want to explore—image, text, video, and audio are just a few examples—and the inference speed required for the use cases. That can help determine how big the model should be in parameters and its shape.

Without sharing too much of your secret sauce, how do you actually do this post-training work? How do you take a pre-trained model and turn it into Hermes—turn it into a great coding model? What’s the day-to-day like for you?

Day-to-day looks like a lot of data engineering: finding, curating, cleaning, creating, and sourcing the right kind of data to fill any gaps for the tasks you care about, while keeping it high-quality and diverse.

From there, it’s a lot of infrastructure-related challenges like sourcing compute, getting the training and hyperparameters just right, and pushing through the training.

Finally, there’s a lot of post-training work on testing, evaluations, documentation, and all that, which sometimes takes as long as training. If the evaluations don’t look right, we go back and start from square one to find out what went wrong.

This is really interesting. Maybe an odd question, but everyone is so focused on emergent behavior with models and new use cases, yet it sounds like you need to have your task in mind—coding, etc.—when you’re doing post-training work. Do you need to have a very specific focus on the end game, say Hermes?

If you want a specialist model, yes—or if you have at least some ideas of the end use cases you want to support. The models do generalize, but they generalize best when given a good, diverse dataset to work with. We try to be as generalized as possible to support as many use cases as possible, but when you know a task is desired downstream, it’s always a good idea to give it special attention.

With the agentic future, these often require much more foresight into the specific downstream use cases you want to target, as the model is often paired with an agentic harness. Training with that pairing in mind can unlock a lot more value.

This makes sense. Before switching gears, do you believe we are running out of data, or is that a myth? I ask since you are very focused on the data that goes into Hermes.

I think the data issue that’s popularly discussed is more in the realm of pre-training data, where the entire internet and beyond has been scraped and input into these models. For post-training, especially with RL environments, we are just barely scratching the surface, in my opinion.

Stepan

Teknium, thanks for doing this—super interesting! My question is about the future of models and agents. Broadly, I see two schools:

One thinks that, over time, models and apps will become more specialized and heterogeneous, with some kind of division of labor for AIs.

Others believe that agentic capabilities will become baked into even larger models. It’s cheaper to send gradients within a single network than JSONs via HTTP.

Which one—or both or neither—do you believe is more likely, and why?

When thinking through Nous Research, what is the fundamental reason an open-source AI lab has the ability to be impactful in the face of hyperscalers that have orders of magnitude more compute, data, and capital to spend?

Helpful, Tek. Feel free to answer mine, then Speaker 1’s, if easier.

Austin Muñoz | Amber Group

Can you elaborate on this? What data seems to have been left out of pre-training that you guys are capturing in post-training?

The best shot to win against tech oligopolies is to embrace open source and open science. Open research compounds on itself, and it undermines their industry capture. If you can get the world to work on the same problem together, it can be a massive boon against the centralized players.

I do think that training for specific use cases paired with specific harnesses, products, and so on can unlock a lot of value. You can see it even at big labs: they have specialized harnesses and training for different “products,” like Deep Research or the web-browsing agent. I think over time these capabilities will find their way back into their primary models, though. So, I suppose, both.

It’s not really that the data has been left out of pre-training; it’s more about the fact that in RL, the data is generated by the model itself. You can take existing data or completely new, generated data and take it way further, because all you need is a tiny slice of information in some cases to create an insane amount of problems for verification in RL.

On deck for you, Teknium, as people read your answers: Hermes 4 is your latest model release, which I’ve described as an uncensored model, with 60% non-refusals versus GPT-OSS at 5%. This opens up a new design space for anyone using the model who wants to build something where past models said no or dropped disclaimers. Can you walk through your view on the impact Hermes 4 will have?

On its less-censored nature and its use cases, I think there’s a pretty wide array of tasks that just can’t work within the paradigm corporate-sanitized models like ChatGPT exist in. This only gets worse year by year, not just with AI, but with a whole variety of internet products and services. More censorship, corporate “enshittification,” and so on make the world more sterile and close up the breadth of human expression. As AI models do more and more of our work, the paths that have been closed will constrain the forms of expression humanity can take, and I’ve called this phenomenon the mode collapse of the world.

Simple things like automating sales often become extremely tricky to pull off with models like ChatGPT, because the slightest hint that the model is asked to put any pressure whatsoever on a prospective buyer may cause it to break down entirely and refuse to work with the problem. Writing stories of authentic human experiences—even everyday ones, like romance, horror, and comedy—becomes severely suppressed.

White-hat hackers doing things like vulnerability discovery and red-teaming have no viable way to use these models to do their work.

These are just a few examples of the real value of reducing where the model says, “No, you can’t express that.” There are countless more.

Pondering Durian

Everyone seems very excited about RL environments, but they seem quite manual to craft and not very “bitter-lesson”-pilled. Curious if you agree with that and, if so, what other approaches you’re excited about—or is that where distributed versions become more competitive or interesting?

This is great framing.

Do you see any way ChatGPT or a big AI lab could reverse this situation of continuous restriction?

Anil | Delphi

Also, what unique datasets or collection methods do you believe others cannot easily copy? How do you prove provenance and quality for that data?

PD and Anil’s questions are great. No rush; Teknium can add to the docket.

I think xAI is the only lab even considering doing so. I do not see ChatGPT, Google, or Claude doing so.

I think a way to make it bitter-lesson-pilled is if the entire world contributes toward making environments that are meaningful to them. One thing about RL environments is that, despite being deeply integral to machine learning, they rely more on domain expertise than ML engineering skills. This is perfect for an open-source initiative. Even gigantic companies can’t source specialists in every field, but open contributions can.

I’d also note that before RL, big labs were spending hundreds of millions of dollars sourcing complete prompt-response samples from humans—i.e., Scale AI. So it hasn’t really made the bitter-lesson calculation any different here; it’s likely even more efficient.

Harrison | Hack

Sorry if this has already been asked, but in post-training Hermes 4, which is awesome, how much repeat post-training work do you need to do once the weights of the underlying base model change—for example, from Llama 3.2 to 3.3 to 4? Is there a repeatable layer so you’re not spending a bunch of GPU cycles?

Does that process translate across different architectures, such as Llama transformers to Kimi or Qwen, or to more MoE models?

On dataset collection, without sharing any secret sauce, can you share a little more about where you’ve found good sources for the high-quality post-training data Hermes finds useful and that’s orthogonal to what the major labs use?

Love this.

For Nous specifically, we have the creator of Generative Reward Models as our lead on RL, so we have a big leg up when it comes to rubric- and LLM-as-a-judge-based environments, and things like RLAIF, or Reinforcement Learning from AI Feedback. The Generative Reward Models paper describes the best ways to use LLMs for scoring, rather than a hard verifier like you might use in math.

This is similar to the rubric-based rewards OpenAI used for things like Deep Research, where the reward is fuzzier and less verifiable than in a completely objective scenario. We have a lot of expertise on this front.

Hey, Harrison. Unfortunately, if you want to change the base, unless the new base is a continued pre-train of the previous base model, you would have to retrain the model.

In a case like Llama 3.1 to 3.3, if they had released base variants of both, you could do model merging to upgrade for basically free.

Merging across architectures is currently not possible, but friends of ours like Charles Goddard and our Researcher-in-Residence, Bhavesh, have been exploring it deeply for a while now, looking to find a way.

On data curation, we have 3 big sources:

1. Our DataForge pipeline has great mechanisms for creating new problems and diversification through pre-train mix-ins as random salt, multiple stages of validation on instruction and answer generation, and more for synthetic data creation and distillation.

2. Rejection sampling and RL through Atropos, our repository of RL environments, which can generate reasoning data on over 1,200 different tasks and growing.

3. Great filters and curation tools for sourcing data from open-source datasets.

While people read, Tek, I’m curious about the interplay of Nous’s initiatives. How does Psyche Network interface with Atropos on the RL side, with your consumer-facing chat app, and with all the research work you put out?

We are building Psyche out in stages, with traditional pre-training and SFT as the first phase for training capabilities. Next is RL, which is a bit more complicated, requiring coordination of training nodes, inference nodes, and environments—all together. The environments are really just data creation and validation servers. These will be integrated in some form as a next step.

For the consumer app, there’s some research on greatly personalizing models for each user, similar to how Midjourney has personalized models you can make from its v7 model. The work is a bit early, though, so stay tuned!

Maybe a bit of a curveball and a product question: how do we get people to have muscle memory for Nous? ChatGPT has this with AI, Google with search. How do we get Hermes into people’s hands so they have that feeling from a brand or project perspective? Or does Hermes live behind the curtain, powering apps and use cases?

I think it’s a bit of both. We can see Hermes is used in a lot of applications already, especially role-playing and creative use cases. I’d like it to break out into more personalized assistant use cases, especially when we have the agentic prowess that will be coming soon. We see a lot of people using it privately at home for things like legal questions and strategy that they don’t trust in OpenAI’s data center, so there’s a lot of value in a powerful generalist assistant that’s easy to access when you want it and fully private when you need it.

I think our biggest draw is that it’s aligned to you right now, in a way no other company currently offers. But it will become much more powerful as we enter into agents and build products directly integrated with the training to unlock a lot of potential value.

This introduces an interesting question. Hermes is way more open and aligned to you—60% non-refusals versus GPT-OSS at 5%—but ChatGPT has muscle memory and integrations with Docs, Chat, email, Slack, and more. Do you think alignment beats integrations for an assistant?

Christine Yip

Drawing from your experience, what emerging trends in AI research, such as multimodal models, edge computing, and AI memory, could revolutionize AI and agent ecosystems?

First, Christine, then back to Tommy: I think we’ve done a lot of work on memory internally, and there is a beta of our memory system accessible to some users right now. I think it has the potential to be a path to some sort of online learning and in-context personalization for the end user.

In other research-related unlocks, DiSTRO was a huge step in enabling and empowering us to coordinate mass-scale compute for training our models and the community’s next models, opening doors to research on new architectures, and more.

Our Chief Scientist is also researching any-to-any model architectures right now and developing novel ways to produce them, so we may see a Psyche-powered any-to-any model soon. This means any modality—text, audio, PDF files, anything—to anything.

I do think a big open area of research is context length, which Nous pioneered with YaRN around 2 years ago. It was the unlock for OpenAI, DeepSeek, Llama, and others that brought us from 2,000–4,000-token context windows to 128,000. But with agents and reasoning, we need to go a lot further.

I think in the long term, alignment is crucial. We cannot have this mode collapse of humanity and continue on as the same species. More people look to alternatives to reopen the door to their expression and needs every day.

Integrations are quite easy, open source or not, with so many harnesses to enable that, and potentially Nous’s own soon.

Great questions from the crowd, thank you. Keep it going; we have 15 minutes more.

Teknium, Meta has been the main open-source champion in the U.S. If Meta drops a clearly stronger Llama tomorrow, how does Nous stay essential?

Pondering Durian

Kind of a high-level question, but in the same direction: are you more bullish on the current U.S. closed-source approach to model development, and the investment capital it can command, or the Chinese approach with open-source compounding but more limited capital investment? I guess it’s a long-winded way of asking about open versus closed source. I’m curious about your take, and then a steelman of how someone at DeepMind would reply.

Christine Yip

Very cool—and yes, DiSTRO and YaRN were very promising and exciting. Quick follow-up about the first point on memory: can you tell us more about the beta of the memory system Nous has built and how it helps users?

Nav

Teknium, how do you alignment-audit and safety-test your models? Is it fully manual, or have you built any automations?

Nous has been keeping pace and often outperforming Llama instruct models with our own base models since Llama 1, despite their much larger team. They are also the reason Nous exists, because they provided the first block of clay Nous used to sculpt with.

We have done very well against their own instruct models, and we are much more steerable and free to explore anything you want to.

With our latest Hermes 4 model, we outperform Meta’s own instruct variant on MATH by something like 60 points and on MMLU-Pro by 21 points.

I think ultimately, more open-access models are always a good thing. We build off of each other continuously and push the field forward with each step.

Stepan

Question on tool use and orchestration: for more complex tasks, we typically run a planner or orchestrator of multiple LLM instances. Recently, I’ve seen a ton of research around Chain-of-Tools and Pre-Act—ways to create plans based on unlimited and unbounded pools of tools or MCP servers. Do you think specialized models for planning and tool-calling are needed? Do you work on this?

A lot here. I’ll pause my questions given how many we have from the crowd.

I think a mix of both, and I think there is room for both. I’m obviously someone who leans toward the fully open path, but there can be value in closed-source models as well. To some degree, the grab bag of options keeps these models differentiated and unique in their own ways, which is a great thing. The last thing anyone should want is a single model everyone has to use without any choice in the matter.

The memory system right now is primarily designed as a personalization unlock. It will remember you and all the things you want the model to be.

In the future, with agentic capabilities, it will be able to permanently—or for as long as you want it to—learn new tasks and information as it performs its work for you and you provide it feedback.

Another follow-up.

You mentioned, “I think our biggest draw is that it’s aligned to you right now, in a way no other company currently offers, but it will become much more powerful as we enter into agents and build products directly integrated with the training to unlock a lot of potential value.” Are you actively building agents or products directly integrated with the training today? If so, what can you share about the form factor and who they’re for?

Teknium, how effective has Nous been in attracting talent? You’ve raised capital, but it’s obviously well below the billions hyperscalers are spending on talent. Zuck has been firing off $100 million offers, but this seems to be backfiring as some people have left, granted it appears to be less senior folks. Are you competing for mercenaries, or do AI folks come for the mission and ethos?

1. General Tools for Agents

The work we did on Forge was one of the earlier products that used this sort of capability. It could create tools on the fly and used embedded tools to find the right tools for the job as it went about assisting you.

Right now, I think there is still a place for this, but even more exciting to me are much more general tools. Right now, every task—or at least set of related tasks—needs a specialized harness for performing the duties of that role. But ultimately, there are 3 generalized harnesses for all tasks:

1. A CLI terminal on a Linux box. Computer use there unlocks a huge spectrum of tools as a single tool.

2. GUI computer use, which can access the full range of visual-based tools like browsers and interfaces.

3. Long-term, embodied AI inside of robots as the ultimate harness.

Related to your question as well.

Currently, Nous has a very good lock on CLI and terminal-use harnessing in our internal agent. That will be a big focus of our RL efforts, alongside a suite of other very generalist tools.

The AI industry is definitely more competitive for talent now than it has ever been, but we’ve found some real star players and continue to.

Our researchers, like our Chief Scientist Bowen, have unlocked unimaginable things like YaRN and DiSTRO. The post-training team has some superstars like Dakota, our lead on RL infrastructure, who created Generative Reward Models. Overall, the team composition has been strong and continues to grow stronger.

I think we attract talent because of our unique philosophy and position in the space, and that this drives us and our prospective new team members more than anything else.

Doesn’t the challenge—the most important task—then become planning the right order and parameters to call these tools?

Any final questions for Teknium? Please post. I’m sure he has meetings to hop to and models to post-train!

In some ways, yes. If there’s only one tool, like the CLI tool, it’s more about in what ways to use that tool and in what order you use its specific capabilities.

RL is perfect for this, especially outcome-based reward RL. It can learn quickly how to determine the most efficient and effective path to use tools—or not use any—to solve the problem at hand from these environments.

The best part about generalized tools like this is that you can drop these agents into any environment. They don’t have to be made specifically for the tools.

A final one from my side as we gather others’ final questions: looking ahead in AI, what is your strongest contrarian view? It can be anything related to AI.

Many people in the space like that Nous releases are uncensored. Out of curiosity, have you faced any concrete hurdles—hosting refusals, takedowns, legal or IP concerns—and, if so, does or will that affect Nous’s approach?

I probably have many and wouldn’t want to give just one.

First, I disagree that we should focus exclusively on benchmark tasks. Nous has made trade-offs to retain base-model qualities for creativity enhancement and exploration that many others haven’t.

Second, evals aren’t useless or dead. Despite the last point I made, I think benchmarks are an essential tool in inspecting the model. The real problem is that people focus on too few—LMSYS, for example—as the sole arbiter of model performance, when in reality every model should be judged on your downstream tasks exclusively, or on a much wider array of evaluations than most people often do. We need way more evals, not fewer.

And the final one I’d say is that I disagree that synthetic data is bad. There are lazy ways to do it that are undeniably bad, but we have found so many ways to make synthetic data useful to our models that 90%+ of our data is synthetic.

What company or network do you think will be the most valuable in the world in 2030?

Our RefusalBench results have two forms. The first tests the model on a wide array of commonly refused categories, but you set which categories you want refusals for and which you don’t.

So there is an overall score, but also a score for how much you refused categories you chose to disallow. For us, that’s CSAM and suicide. Most everything else is in the realm of free speech and free use, and we attempt to stand by that as best we can.

On takedowns and so on, since most of our post-training data is synthetic, it generally has much less risk of matching copyrighted data as well.

That one is a hard one, and I am not sure. Sorry. Nvidia might be near the top of the list, though.

I’ll answer this one: Nous, among other great projects here.

Teknium, one last one since this is so good. Looking back over the last year, what is your biggest mistake or learning—a constructive growth item you’d share? It could be something you got wrong or were very surprised by.

Just have my mouse hovering over 50× leverage.

Probably underestimating the work that it takes beyond the model itself. Writing papers, documentation, and blogs has always been harder for me than creating state-of-the-art models.

Luckily, we’ve added many team members who are extremely good at this in the last few months, and we will be unifying, expanding, and cleaning up a bunch of our documentation from across all our projects in a much cleaner and central location soon.

Teknium, any closing thoughts based on this conversation and the questions you’ve gotten?

Also, what question should we have asked but didn’t?

My closing thought would be to stay tuned to Psyche’s progress, as we have a lot coming out soon on that. The whole team is now focused on the final infrastructure push, new models, architecture experiments, and more on there!

Thanks for having me, everyone. I have to run momentarily for another meeting, but feel free to ping me any time in here.

Teknium, you knocked it out of the park. Thanks for being our first AMA guest in Ex-Machina.