Watching America Run Away With AI - Alistair Pullen (Cosine AI)

I was horrified, as many were, when Fable suddenly got banned.

We have obtained the mandate to build the UK’s first sort of sovereign LLM.

How can you do in millions what they are doing with billions?

There isn’t a huge amount of room for error or wiggle room. It was not something that was on my bingo card in January. Numbers like 10 trillion are being knocked around. We’re compressing most of the internet at that point.

It’s really funny. After Claude deletes your production database, it’ll say, “Oh, you’re right to point that out.”

Agentic harnesses are getting less important over time. A model can probably do, with Bash only, basically any task these days. Also, thank you, Donald Trump.

For the first time, I feel like a second-class citizen because they are going faster than I am, and I really hate that. That boils my blood more than anything else. We are going to do everything we can to pull this off. We have no choice but to make it happen.

Alistair, it’s great to meet you, mate. We are here in London, where it’s customary to say, “Hello, geezer.”

Hello, geezer.

So, where in London are we?

We are in Hoxton right now. So, we’re in Shoreditch. We’re about half a mile away from where Cosine started in my apartment, which was in Hoxton Square, just over there. We haven’t come very far, but we have expanded a fair bit since then.

And what is Cosine?

Cosine is a frontier lab based here in the UK. Prior to about 3 months ago, we built best-in-class coding agents specifically for highly regulated and high-side environments. Think things like financial services, insurance, defense, and so on.

More recently, we have obtained the mandate to build the UK’s first sort of sovereign LLM, which is a much more ambitious vision and something on a scale much larger than we’ve done before. It’s very exciting to be working on it.

So, tell me about that sovereign AI piece. I should say, by the way, I read the article about you in The Telegraph.

I was horrified, as many were, when Fable suddenly got banned.

Yes.

Because of this export control, and now everyone suddenly is thinking about sovereign AI. So, tell me the story.

It ties into a bunch of different things. It ties into the backstory of Cosine, and one of the reasons we’re fortunately placed to be able to do sovereign AI is that we have a lot of expertise around model training, model building, all of the infrastructure and algorithms, and the data people that you need to do that kind of thing. We’ve been doing that for some time, so we already had all of those things in the organization.

Then, probably 9 or 10 weeks ago, we were inducted into the government’s Sovereign AI Unit—or backed by them, I should say. That is something they have put out to increase the number of sovereign AI initiative companies being built in the UK.

What that looks like in practice for us is an allocation of compute on the Isambard-AI supercomputer cluster out in Bristol. Honestly, that’s one of the things that’s unlocked our ability to even have the ambition to do something like this. Fundamentally, one of the biggest blockers for a startup of our size or smaller, to be honest, in being able to approach work like this is compute.

If you raised $50 million to $100 million, a good chunk of that would go on compute for a project like this. To have an allocation come from the Sovereign AI Unit is huge because it genuinely does enable it. We still use some private compute on the side, but fundamentally, all of it will be done on Isambard, which is super cool.

To be honest with you, it was not something that was on my bingo card in January, at the beginning of the year when we started out. We still do our conventional business of coding agents and the models that we’ve already built, but we’ve been able to take that vision and really take it to the extreme in a way that we wouldn’t have been able to otherwise.

So, I’m not being funny, but the million-dollar question is—well, actually, more than that. Folks over there in the US have probably got on the order of hundreds of billions. You’ve got Mistral, which is on the order of, let’s say, 14—

Single- to double-digit billions. Yeah.

Something like that. How can you do in millions what they are doing with billions?

Yeah, no, it’s a very fair question and one that I probably get more than anything else. At Cosine, we are not an inference company. That ties into the kinds of deployments and the way that we sell our product.

For your viewers, I should probably give a bit of background. Given the fact that we predominantly deploy into highly secure, high-side environments, most of the time now—nearly all of the time—we are not hosting the model ourselves. A customer isn’t hitting Cosine/API/v1 and then hitting a chat completions endpoint or something like that from us.

They are either taking the model weights that we give to them and deploying them on their own GPUs. We have a lot of that; that’s the most air-gapped, most secure deployment we do. Or they are renting GPUs in some hyperscaler cloud that they’re already a part of, whether it be Azure, AWS, or whatever, and then they’ll run the model there.

What that means in practice for Cosine is that we license the technology that we build. We don’t actually make a margin on tokens or anything like that. All of this ties into your question, meaning we don’t have to spend a lot of the money that the Americans are having to spend on data centers for inference purposes.

That’s not to say that you don’t also need a huge amount of compute for training. Obviously, you do, and a huge amount of the infrastructure they have in the US will also be used for training. But I think that one of the biggest reasons you’ve seen people like Anthropic struggle recently, and the reason they’ve signed the deals they have with the Colossus cluster and so on, is inference and not training.

You do need significantly less resource if you’re not going to do the inference bit. We’re fortunate in that the way that we sell the product means we don’t really have to. On the other side of that, we are taking some interesting research approaches in terms of how you pull something like this off.

We can talk more about that in a minute, I’m sure, in terms of how we are architecting the model, how we’re training it, and some algorithmic stuff. All of that is to say that we do have a credible shot at pulling off the full run, including the continued pre-training, the mid-training, the post-training, and all of those bits.

But, to be completely transparent with you, there isn’t a huge amount of room for error or wiggle room. There are obvious places where we have had to make trade-off decisions. That includes—and extends to—the scope of RL.

I’d always like to do larger RL runs, more generations, and more inference-time compute during the RL process to get more variety. We can’t do as much of that as we would like to if we had 10 times more compute, for instance. There are trade-offs, but fundamentally, given the way that we’ve scoped the project, I think it is viable in that very narrow scope.

We have some of the largest companies in the UK all feeding use cases and their desires for what they want the model to be able to do directly into us, so that we can train a model that’s really for them. I think that is potentially a feedback loop that hasn’t really been explored as much in the space.

Obviously, I don’t want to say bad things about some of these companies, but the models from Mistral, from Cohere, and so on, they’re not competitive. Even the Chinese models, arguably, have only really started getting competitive the day before yesterday. So, you know—

Months or so. Yeah.

GLM-5.2. So, the vibes are good. Although on the ARC challenge, it didn’t do very well, but maybe that was just a red herring. I can talk about that in a minute, but yes. Very cool.

But you were almost implying that it’s because they weren’t trying to make it better. How can we make models that are as good as those frontier models?

Crudely, I think there are a couple of key things—maybe 3 key things. I think one is architecture and raw model size. I think the second is active parameter count. The third is data.

I believe—and correct me if I’m wrong—the largest model that Mistral has made to date is the 675B Mistral Large 3. It’s sparse, very, very similar to DeepSeek’s architecture, if not the same. Fundamentally, that model exists in the way that it does because it fits a use case that they have seen.

It probably fits a GPU deployment profile that they have seen in the enterprises they’re trying to sell to in France or Europe. As a result, pragmatically, they’re like, “Right, this is probably the biggest that we can get away with given what certain companies have access to.”

As a result, you’re obviously going to cap out how far you can go in terms of model performance. There was a very interesting analysis done in a blog post, which I can’t remember the name of, but I can send it to you post-talk because I just found it very interesting. It was a breakdown of the probable sizes of models like Sonnet and Opus.

Yeah, I saw that.

Did you see that? It was so cool, the way that it was done, right? Through Vertex, and figuring out, given what we know about open-weight models, latency times, and stuff like that—

The one I saw was where they came up with a bunch of questions, and they could infer based on the general knowledge that one has as well. Yeah, but that was a little bit sketchy.

I’ve seen a more empirical one. I’ll send it to you post-talk because it just found it very interesting. That blog post alone played a large part in the architectural decisions we took for the sovereign model.

One of the things that was very clear from that is that the likelihood is—and, again, no one really knows outside of Anthropic—that something like Sonnet is, I believe, in the 1.3 to 1.5 trillion total-parameter range and probably has 100-plus billion active parameters. An Opus model is probably in the 1.5 to 1.8 trillion range and probably has 150 to 180 billion active parameters, depending on the data type we’re talking about, whether it’s FP8 or FP4.

What about Fable?

It wasn't up for long enough for me to try. What I wanted to do was give Fable that blog post, point it at an endpoint, and say, “Right, do the analysis on yourself and tell me how big you are.” I never got that far, though, because I don't think it was up for long enough.

It was a ridiculous uplift, though, isn't it? I saw—I'm sure you saw on X—numbers like 10 trillion being knocked around. I don't know if that's true or not.

I have no idea. I think it's obviously bigger than the other ones and obviously has way more active parameters, but I couldn't speculate. I genuinely don't know.

I think the net of that article was that Opus was in the region of 1.5 to 1.8 trillion, with around 150 billion active. Obviously, there's a lot of algorithmic and data work that goes into it, but I think if you don't at least match that architecture, then you're already going to struggle to reach that ceiling.

An example of this—and it's way more nuanced than this fairly basic argument I'm going to make—is if you look at something like DeepSeek V4 Pro: 1.6 trillion total. I can't remember the exact number, but it's going to be in the region of 30 to 50 billion active.

Right. My view is that the reason for these architectural decisions is largely inference of the model. Sure, it's great that you have those headline 1.6 trillion parameters, but if no one can run it because they need 2 nodes of B300s just to fit it into memory and actually run it at decent tokens per second, then how many people can actually take advantage of that?

I think that's one of the reasons that Chinese, European, or American open source has not reached closed-source performance. There's that pragmatic question of, “Okay, the labs have a huge number of GPUs, and they have enough inbound demand to make sure those GPUs are utilized to a level where they're not that worried about having them up.” The open-source community doesn't really have the same argument.

If you're running it yourself, if you're going to run a model of that scale on your own hardware and it's not really being utilized that much by your organization—

You're going to worry about how much money you're spending on those GPUs just sitting idle.

I think that, on that first point, architecturally, overall parameter count is obviously important. Active parameter count is also incredibly important. The last bit is data.

I think the labs have some element of an edge in data, both because they're able to procure so much from the brokers who sell it and because they have internal data functions that are very mature at this point. I'm not saying the other labs don't have that, but they're definitely not at the same scale, both in terms of spend and maturity.

To an extent—and this is definitely not true, but I think to an extent—the pre-training corpora: is there going to be that much difference once you're in the 30 trillion-token range? You kind of all have roughly the same stuff. We're compressing most of the internet at that point.

Mid-training is a similar story. I think post-training has been and remains one of the most interesting areas for these labs. Having worked with some of the labs on post-training data—because obviously we're very good at coding—it's been interesting to see how they've been procuring it, even the formats they've been using for the raw data and the different use cases they're interested in when they're putting requests out for, “We want this, we want that, we want the other thing.”

I think that's one of the key areas where they're differentiating: having really good post-training data and also just being able to run RL at ridiculous scale.

Quick pause. Agents are getting smarter every day, but even the smartest agents get stuck without the right context and the right tools. That is where Notion comes in. With the recent launch of custom agents, Notion became the collaborative AI workspace where teams and agents work side by side. And now their new development platform is turning that workspace into infrastructure developers can build on. Now, this is exactly how I run MLST. The whole show lives in Notion. My guests, the publishing calendar, the commercial side, everything is in there. But what's changed is that it's now agentic. I just talk to my agent. It can be Claude or any agentic harness. And then it talks to Notion via the MCP or the CLI and it's just done. And then I can access it on my phone. It's an absolute game changer.

We'll get to the RL bit, because I know you've got an opinion that RL training is really important. There are a few things about what you just said. First of all, is the MoE thing just a trade-off for inference speed, or do you think there's actually some beneficial advantage in terms of factorization from having an MoE?

I think, at least in my opinion, if we could have trillion-parameter dense models, we would.

It would just be strictly better.

I think it would just be better. But again, that's taking it to the logical extreme of: what would you need to run inference on that? What would you need to actually run a model of that size?

An interesting example of this—and it's not really a fair fight—is if you take, at small scale, something like a GPT-OSS-12B. That is an MoE; I believe it's 5 billion active, although I can't remember exactly. Then you take something like Devstral 2 123B, which I believe is a similar architecture to Llama 70B, so it's fully dense. I don't know if you've used them back to back before, but Devstral feels so much better than the GPT-OSS model.

They're architecturally quite different, particularly in the attention mechanism. Fundamentally, though, I think a huge part of that comes from the fact that one has 120 billion active parameters per token and the other has 5. We deployed both to customers last year, and the difference was night and day in terms of how they felt and what customers got out of the coding agent when they were running it.

Oh, interesting. On the other stuff, you were talking about the data and the pre-training and the algorithmic work. Maybe the algorithmic stuff has converged only because we now have this basin of attraction where there are kernel optimizations and entire ecosystems around them, and maybe it's kind of converged.

The data thing is interesting, right? With LLMs, it's a little bit like, “What's the magic word?” If you frame the question in the right way, it has that representational friction and does interesting things. I'm guessing Anthropic does a whole bunch of data curation and pruning.

Another thing Anthropic has is Claude Code. They have the ecosystem, and—

They have trajectories coming in all day, every day.

Exactly. Those trajectories—because you've got this big thing that it's not about where you end up; it's about how you got there. Software engineering isn't about writing code. It's actually about creating a process.

Creating mental abstractions, doing experiments, refining those abstractions, and sharing them with the team. This process—iteratively, I'm running code, testing things, and refining my abstractions—and Anthropic has access to so much of that data. How much of an advantage is that?

Obviously, it's a huge advantage. I don't know what their terms of service say, and I'm not saying that they train on all that data. I don't know whether they do or not.

One of the interesting things about having trajectories—and something that we obviously do internally—is that we collect our own trajectories from our own use of Cosine. Obviously, we don't have anyone else's because of the way we deploy it. For Cosine employees using Cosine, at least, we still get a fair number of trajectories—nothing like the order of magnitude that Anthropic gets.

The most useful thing about trajectories for us is seeing how users prompt models. There's one thing when you're putting together RL datasets and you can come up with these beautifully formed problem statements that are fed into the model. They're very well structured, very well formed, and very clear about what the expected outcome is.

The reality is that users don't prompt models in that way at all. They're like, “Fuck you. It doesn't work. Why the hell—are you dumb? Why have you done this, that, and the other thing?” If that's the distribution that the model is going to be exposed to in the real world, then you need training data that looks like that.

I would speculate that a lot of the alpha Anthropic gets out of the trajectories it receives is obviously helpful. I don't think they canonically know; it's not like there's a grader attributed to that trajectory. It's not like an RL process where you know that the trajectory was good.

Absolutely, I'm sure you can do some judging and see how the users responded, so you can say, “Okay, at this point the user was probably satisfied that the job was done.”

But it’s also fundamentally about seeing what the user is saying during the conversation and how they’re saying it. That is incredibly useful, and even at our scale, there’s an argument to be made about how representative it is of the entire development ecosystem. That’s a different question, but we do have an idea, generally speaking, across months and months of usage, of how engineers interact with these products and how they respond and all these things. That’s very useful for making things more realistic at training time.

Yeah, 100%. You’ve spoken about slop online, and it’s one of my pet topics as well. Vibe coding works in the sense that quite often it does the thing that you tell it to do and the test will pass, but actually you’re building a spaghetti monster. You’re throwing more bad spaghetti after good spaghetti, and instead of doing a one-line fix that it’s supposed to do, it’ll give you 200 extra lines. There’s the whole understanding-debt thing; we can talk about that.

So what we actually want to happen—and neural networks do this to a certain extent—is that they learn statistical invariances that represent some kind of abstract structure, and that helps them generalize. Obviously, what we want them to do is learn problems in the abstract so that they can generalize to new, novel problems that they’ve never seen before. The idea is that we capture the thought process and then capture that generalization if we do it well enough. That’s the rough idea. How do you think about slop?

Slop is also one of my pet hates. Obviously, any of your viewers who use Claude Code or OpenCode, or whatever agentic coding harness they like, see this ad nauseam. It’s a combination of model, vibe, and slop problems, and you hit the nail on the head in your question: the tests pass. Is it technically functionally correct? Sure. But at what cost is the big question.

Fundamentally, when you think about how these models are trained to do software engineering—and this is the realization we had when we built Outpost, and I’m sure you’ve seen the blogs about how we did it—normally, and again, we don’t have that much insight into how this works in the big labs, but normally, when you’re training a model to be better at software engineering, you have some kind of software engineering problem. You have a problem statement and some kind of test, often a unit test, but not always, depending on the task type, that is failing in the prior state and passing in the after state.

Then you give the agent the problem. It goes and does its trajectory, its rollout, and then, at the end, you run the unit test. If it passes, then, okay, you got it right. Great—you get a reward, and then the weights update.

Obviously, the problem with that is a fewfold, but fundamentally, it could have come up with the most insane way of doing something. That could be in terms of commands it ran that were unsafe, or code that’s absolute crap compared to what it should have actually done. All of those things get reinforced, whether you like it or not, when you give that reward based purely on correctness.

There are a number of things we have done and are continuing to do in the RL process to try to ameliorate this. In terms of slop specifically, there are a couple of key things. One is that correctness gates everything else. If you get the problem wrong, regardless of whether you did it in an elegant way, you don’t get rewarded. But beyond that, we do have other rewards that target the exact things we’ve been talking about.

Also, in some cases, but not all, we do have reference implementations for these things. If you are using a pull request from a permissively licensed open-source repo as some seed data, you do have the original patch the human made. You can actually do some level of, okay, let’s compare what the agent wrote, let’s compare what the human wrote, and does this seem reasonable? If it’s 500 lines longer, probably not. Maybe we shouldn’t give the full reward for this. So there’s stuff you can do on the pure reward level, but we’re also doing stuff on an algorithmic level, which is being looked at more and more in the space.

That has to do with credit assignment in trajectories. One of the big problems with RL as it stands—I think Andrej Karpathy said this over a year ago—is fundamentally this notion that you have a rollout of maybe 256,000 tokens in some extreme cases, and that culminates in a 1 or a 0, depending on what the model did. What we’re saying at the moment in many cases is, okay, all of those tokens are equally weighted in getting us to that answer. Yeah.

Which, when you think about it, is insane, because that’s clearly not true. In so many cases, there will be small—or important—decisions in a trajectory that were forks in the road, and that could have resulted in a bad outcome, but the model decided to go down the right path and ended there. What we and others in the space are trying to do right now is, if you can find those ranges of high-entropy tokens, or places where a decision was made, finding that is half the problem. Then, once you know that this is an important thing that happened, determining whether it was good or bad relative to the final outcome is a different story. Yeah.

But if you can do that, your RL gets significantly more efficient because you’re not relying on the entire trajectory. The analogy I always come up with is, say you were doing your English A-level and you’d written a 2,500-word essay for your teacher. You’d written it as practice, and the teacher just gives you, “Right, okay, that’s a B. Thank you so much.” And you’re like, “I don’t know what made it a B.” The teacher says, “I’m not going to tell you what made it a B. It was a B.”

Then what you’re going to have to do is write hundreds of essays. You’ll get an A on some, a B on others, and a C on others. Eventually, you’re going to be like, “Okay, when I do this, I tend to get an A more, so I think this is probably a good thing to reinforce.”

But it would be far easier if the teacher just sent you a circle around the sentence and said, “This is rubbish. Don’t say this.” That is fundamentally the principle we’re trying to bring into RL across the board, because you get so much more performance and get more out of the FLOPs that you have. You’re also teaching the model to learn the things that are actually important, and not just the filler, right?

I know. I mean, the great thing about machine learning is that it just generalizes low down the abstraction mountain. So, from very superficial statistical generalizations, the bad thing about machine learning is that it generalizes. So, yeah, I completely agree with you.

We have a huge problem with benchmarks and machine learning. We’re obsessed with pass@1 and pass@5 accuracy, and we don’t seem to care about reliability, consistency, security, or abstraction-forming. That’s clearly the most important thing. François Chollet did the ARC challenge, and unfortunately, those were brute-forceable. Now he’s got this new version, which is so difficult to brute-force that you have to form abstractions to get any kind of good performance on it.

So you’re saying there’s a new form of RL, perhaps different from the DeepSeek type of RL, where rather than just being rewarded for getting the right answer, you’re forcing it to form reusable abstractions and go higher up the mountain?

Yes, in short, you’ve explained that far better than I did, but yes, that is essentially what we’re trying to get to. One of the nice things is that I think, essentially, what we’re talking about here is credit assignment within a trajectory. That ports quite nicely to a bunch of different RL algorithms that are in vogue at the moment. You can use it with GRPO, you can use it with GSPO, and all the different flavors of that algorithm.

Fundamentally, having a rigorous and, importantly, unopinionated way of pointing at ranges of work that an agent has done and saying, “This is good; this is bad,” and so on—and having the advantage calculated across those different trajectories attributed to those ranges disproportionately to the rest of the tokens in the trajectory—means that your weight updates will be more targeted at making those characteristics either appear more frequently or less frequently.

Do we still have an epistemic problem? Because the one problem with machine learning is that it doesn’t really have the notion of true and false. We can do feedback from code execution, and we can do a whole bunch of abstract lenses on actual processes that engineers are doing, but don’t we still have this gap that we don’t really know whether it was correct or not?

Yes, it’s one of the hardest things.

Particularly, one of the obvious reasons coding has taken off as a use case is because you have verifiable rewards in some guise. I think one of the things that I’ve just said is that we’re trying to bring some of the fluffier, taste-related things and make them verifiable, but on a more floating scale. Obviously, for things like law and other nonverifiable domains, that is way, way harder.

I think that is one of the key reasons that we haven’t seen the same revolution in other industries as we’ve seen in coding, maths, or physics, because you can’t just statically compile some law and see whether you get a 1 or a 0. I don’t know what the answer to that is. I really don’t. Maybe there are new ways of codifying those domains, or maybe you just bring everything in-distribution, which I think is what’s happening these days. I think if you just make everything in-distribution and target every use case, and you have some level of either a human judge or a good LLM judge that has been trained or well-prompted by a human, you get close. But I still think that, essentially, in many cases, you need to enumerate a lot of these problem sets at train time and make sure the model’s good at them, because otherwise the dream of generalization across the board, I think, has already been shown not to really exist that much.

I know, but we’re in such an interesting time because every new model comes out and Fable is so much better. We want to have systems that do more with less, which is what you’re saying. We want them to acquire these abstractions. It’s a really weird situation, right? Do you think we’ll ever get to a point where we can remove the human from the loop?

Because right now, I think the basis of AI psychosis is that you have very, very talented humans, and they know how to ask the question, and they have taste, and they go in the right direction. There’s this virtuous co-creation cycle. It’s very, very good. So we’re now in the realm of—not vibe coding, it’s agentic engineering. It’s very exciting. But do you think it could ever be done without humans?

Yes, I think it can be done without humans. I don’t think we’re anywhere near there yet.

What would that look like? Do you mean in well-specified problems? I’m not talking about style transfer. Anthropic built a C compiler, and that’s a well-specified problem. Say I give you a novel application and I can only vaguely specify it.

Guest

We’re going to need a human for a long time, aren’t we, if you want it to be good and maintainable and actually in the style of something that a senior engineer would write. For now, yes, you definitely need a human there. Probably post hoc, to be like, “Okay, here’s the mountain of stuff that you need to change,” and all the design decisions that, in your chain of thought, you thought were good but actually weren’t because of real reasons.

But I do think that we will get there. I think it’s going to be through a combination of model improvements, creating RL environments and problems that really look like the kinds of things you’re talking about, and harness engineering. I think it will be a combination of all those 3 things.

Yes. And we’ll get to harness engineering, but in the meantime, we’ve got the spaghetti monster mitigation strategy, and I think one of the big problems is code review, right?

Oh yes, totally.

Because the AI psychosis has manifested in my understanding that, increasingly, I become less aware of what’s going on, and that’s actually really bad for maintaining my competence and for being able to evolve the software going forward.

Ask the right questions. Yeah.

Absolutely. Exactly. So how can we do this? Because now we’re generating ridiculous amounts of code, right? Is this a case of let’s use more AI to do the review, or do we still need humans in the review?

I think we need more runtime validation of what AI is producing. What that looks like is, I think, code review will evolve somewhat. I think it will be more like proof that the thing it says it’s doing is actually doing that thing. Obviously, AI reading Git diffs is not useful. I think it can catch things, and I’ve seen it catch things in the past.

One analogy that I can tell you that’s really good when it comes to, say, our cybersecurity scanning product is that it’s analogous to code review because it’s basically reading a whole codebase using a swarm. One of the key things that we saw with that was that it would go through that process and pick up so many things across large codebases: “This could be a problem. This could be a problem. This could be a problem.”

I think one of the things that we see in that and in AI code review is that, sure, if you just look at that code in isolation and that function definition, it can look quite dodgy. But in reality, the code path is never hit, or there’s another function that’s called first that mutates this variable, which then means that it doesn’t do what you think it does, or there’s an environment variable that’s set at runtime that means this doesn’t happen. All of these things—you know, it’s just static analysis. You can’t do it.

One of the best ways that we mitigated that problem in that product was that we had what we called exploit validation. The swarm comes up with its list of things, and before any of them makes it to you, we spin up the application in a virtual machine, in a way as close to production as possible, and we tell the agent, “Well, you’ve seen the source code, so if it’s vulnerable, you should be able to figure out how to get through it. You should be able to craft your horrible ZIP file to exploit this thing that you think exists.” If you can’t, then you just take it out of the list because it’s clearly a false positive.

What we’ve been working on is applying the same logic to PRs. We’re not necessarily looking for cyber vulnerabilities, but we’re looking for this: You have allegedly built out this feature. You have this new screen that has a table in it or a form in it that does this. When you click on this button, it should result in a new entry in the DB, and it should show up in all the stuff you’d expect.

Instead of just looking at the diff—which, obviously, you can do, and for the most part you can get a lot of mileage out of that—also just show me it’s doing that. Prove that it’s done that in some reasonable way before it even makes it to me, because otherwise you end up in this situation.

I was just talking to a customer earlier today, and they were saying that when they first started adopting agentic coding tools, they were in this spot where either they would end up with this enormous backlog of code review, or you’d get people saying, “After a minute, looks good to me. Merge. Thank you so much.” You just get this YOLO merging into your main branch, and that’s bad as well.

I think there is going to be way less cognitive burden if, however you’re doing a review, you can see the code, but you can also see canonical proof that, at least on the happy path, the thing it’s saying it’s doing is actually happening. The other half of this, at least in the present day, is also really comprehensive end-to-end testing of everything you build.

That is something that really sucks to have to build out, but once you have it, it saves you from so many problems. I’m sure, as you know, you’ve seen it—I’ve seen it in personal projects and so on. You vibe-code for an afternoon, you build 10 new features, then all of a sudden the other 5 you had before stopped working. Why has this happened? It’s like, “Oh well, okay, the abstraction you had—I’ve just messed with it, and now it doesn’t work for that thing.”

So, yeah, it’s a combination of defensive stuff, like the end-to-end testing, and also proactively lifting mental burden from people by being like, “Look, here’s either a screen recording or some screenshots or whatever of me showing you that this is what I think it is.”

I know. It’s really funny. After Claude deletes your production database, it’ll say, “Oh, you’re right to point that out. I’m so sorry for doing that. There’s nothing you can do about it, by the way.”

I mean, this is another alignment problem, though, right? If we are accumulating understanding debt, the functional descriptions themselves are going to suffer from that because we don’t understand the functional description anymore. And it’s not just functional descriptions. There are intents, there is behavior, there are all of these different levels of describing a system, user stories, and stuff like that.

Unfortunately, these are different views of the blind elephant, right? They don’t necessarily have friction with reality. You see the problem here: we’re just losing touch with what it’s supposed to be doing. In many cases, actually, Claude and the models are writing the functional tests, and then they’re kind of hacking their own…

Oh, it didn't pass. Okay, I'll just change the test. Now it passes. Great. Here we go. You see it all the time.

I know. It's a very difficult problem, but one that we need to fix. For me, I think a lot of it has to do with scoping and constraints, to at least cut down the size problem. But we should move on. What are your thoughts on agentic engineering?

You guys have an agentic harness. If I understand correctly, a couple of years ago, you actually forced everyone to start using that because you really wanted to optimize the hell out of it. You were talking about the RL piece, so maybe there's some co-evolution with the agentic harness and the RL. What's important in this agentic harness?

I think that, broadly, agentic harnesses are getting less important over time.

Oh, interesting. Why?

The models are just getting so good.

Oh.

I think you can get the proof point from the fact that a model can probably do, with Bash only, basically any task these days—more slowly and with more tokens, but it can probably still do it. That's not to say that agentic harnesses aren't important, but I think, over time, where's the value coming from? It's coming from the model and not from the harness.

I think the way that we built ours—and we have been building agentic harnesses for a very long time—is important here. The first agentic model that we had was a fine-tuned GPT-4 Turbo model that we trained in January 2024. The coding-agent harnesses didn't exist at that point. Claude Code didn't exist. None of this stuff existed, so we had to build one out.

We did that symbiotically with the design of the model, which is something we still do today, because you get way more performance out of tightly coupling the two. Fundamentally, harness engineering is still something we care a lot about. These days, we actually care more about efficiency than anything else.

In a world where token costs and tokenomics—which is a word I heard for the first time today; awful word—are becoming increasingly important, particularly to enterprises that we sell to, we want our harness to use as few tokens as possible. Full stop. That's what we're trying to do.

Yeah, we'll get to that in a second, because if the models can do epistemic quantification, you could, in principle, allocate a budget of tokens to get certain things done. But even before we get there, I want to push back on the harness engineering, because one thing I have found is that subagents are a game changer.

Yes, absolutely.

A lot of problems are too complicated for an LLM to do in a single pass. I'm sure you've had a similar experience. As a problem becomes more specified, as you reduce the ambiguity, the entropy goes down, the models get better, and the models are better when they have less rot in their context.

So what happens is, after doing a bunch of engineering, what engineers find is that they decompose problems into agentic subtasks, and then they have a fresh agent. The agent has a clear specification—

Only do this one thing. Yeah.

Exactly. But what you're doing logically as an engineer is factorizing a problem into smaller subproblems. You're getting agents to orchestrate, and you're not rotting the context in the main one. How do you see that evolving over time? Because now it's quite a manual process, but you do have this Swarm thing, maybe.

Oh, thank you for bringing that up, because that was exactly what I was going to answer. Swarm, to me, is subagent orchestration taken to the logical extreme. We're kind of lucky to be able to do it because, obviously, Claude Code has, I believe, workflows, and Codex has subagents as well.

A swarm is what it sounds like on the tin. A swarm is genuinely a lot of subagents running at the same time in a hierarchical way. Because Cosine isn't trying to serve hundreds of millions of people a day, we are able to serve a swarm-like feature. Whereas I think if Anthropic had a swarm for Opus, even Colossus Swan would run out of tokens.

Swarm does exactly what you've just outlined automatically, basically. There is a video on my Twitter and also my LinkedIn of me taking our Lumen Outpost model, which is post-trained from Kimi K2.6. So definitely not an Opus or a Mythos or anything like that.

I asked it, "Okay, I want you to build me a mechanical-watch compiler." I am Swiss by birth, so I have a reason to do this. I basically asked, "I want you to build me an SDK in Python so that I can specify mechanical watches in code. I don't know horology, but I want to be able to do it anyway."

I want it to be physically congruent. I want you to use some kind of physics engine. I also want a 3D viewer so that I can see the thing running. It all needs to be possible in real life. You can't have things intersecting each other when that wouldn't be possible, and so on.

That is something that, out of the box, Kimi cannot do. There's no way, not even close. It would be terrible. In fact, even Gemini 3.5, Opus, and 5.5 can't really do it.

But as soon as you put them in a swarm—and what Swarm looks like for Cosine is that you have one orchestrator at the very top—it breaks down a problem into subproblems for basically product managers, or whatever you want to call them. We call them subplanners, but they own verticals of this.

Within that task, you would have had a subplanner to do the SDK, a subplanner to do the 3D viewer, a subplanner to write the documentation, and so on. Those subplanners could then delegate to workers, and they have a flat layer of as many workers as they like.

For that problem, we use subagents, which I think is more than you tend to see in a Claude Code session and so on. You'd probably hit your usage limit pretty quickly that way. But when you do that, it is possible, and you can do that entire project in one shot.

I'm contradicting myself quite badly because I've just said harnesses don't matter, but in that respect, they obviously do matter.

Oh, indeed. I'm very excited about that. But it raises the question: first of all, when you start to have loads and loads of agents, you have more understanding debt and less interactivity, because, for me, the lack of interactivity is part and parcel of the understanding debt.

Sometimes you want to interject and say, "Oh, you've gone slightly wrong there. I want to change what this agent's doing." What many folks have found when they build these agent systems is that the agents interfere with each other. They kind of overwrite each other. They go into deadlock. How are you dealing with all that?

So it's a hard problem, and we experience all those things. One of the key things that we did is give you the ability to interject with an agent at the lowest level. Say you had a worker that was two levels down from the top one; you can actually talk to that one, which is important.

With regard to other problems, in terms of treading on each other's toes, you can put write locks on files so that only one agent can edit a file at a time. You can also provide context to agents when they're using files. Say an agent is reading a file that it just read: We have stuff in the harness that says, "Okay, another agent has just edited this file, so don't be surprised if you see it slightly differently from how you saw it last time."

All these things help. They're not a panacea, but they certainly help. They make sure the agent is less surprised when it's like, "Oh, where did that come from? That wasn't in my last edit?"

Fundamentally, it comes back to the point I made earlier: yes, you will run this thing, and it will provide you with a huge amount of value very quickly, but you're still going to have to comb through it afterward and be like, "Actually, the reality is I don't like the abstraction you've done. I don't like the way you've done this." There is going to have to be some sweeping afterward, I think.

Yeah, what are your thoughts on memory?

Very hard to get right.

Okay, tell me more.

It's very hard.

Well, a similar thing actually applies to the RL, because memory is not about the destination; it's about how you got there.

Yes, memory is very hard to get right. We've tried a bunch of different approaches. Fundamentally, I think every approach to memory that exists right now is a bit of a hack, right? It's like a tool.

In many cases, it's a vector database or an embedded version of some tidbit of knowledge, but it's very hard for agents to know when to query. It's also fundamentally quite hard for the agent to know whether something was useful enough to write to memory. It's also difficult to keep these things up to date.

We've had many situations where an agent's been doing a trajectory, and when it's been doing something that was genuinely the right thing to do, it's used its memory, but the memory's been old and out of date. Then the agent's like, "Oh, well, the memory says you should do it this way," and it changes its tack. There, you're like an engineer, "No, please don't do that."

There are things that we’re looking at internally with regard to continual learning and stuff like that, to try to avoid memory being a tool and for it to just be something that’s in the latent space with the model. That is also very hard. But I think it is a more intuitive and elegant solution than it just being a tool. It’s also a tool that’s very hard to get right during RL because it is a huge surface area for foot-gunning, in terms of reward hacking, leakage, and being able to query something from the future that it shouldn’t have access to yet, despite all the guardrails that you can put in place. It’s just hard.

Yeah, exactly. And in a sense, this is another area for AI psychosis, because I’ve written a memory CLI, and I would almost argue that now you don’t even need vector databases and so on. You can just have an inverted index, just SQLite, because the models are so good at asking in different directions. So, yeah, there is a huge problem: it needs to know to retrieve that. That’s a big one, but it actually works.

It only works for me because it creates this fractionated spaghetti mess again. So there’s another spaghetti mess in the memory CLI, but it works really, really well. It doesn’t work very well at the organizational level because my spaghetti monster doesn’t play with John’s spaghetti monster. If we could solve that problem, you’re drawing an interesting picture as well of how we can actually optimize the different layers of the sandwich together.

Yeah. I think that as soon as someone gets it right, you’ll just know when you’re using it immediately. I haven’t seen a single implementation, whether it be Claude Code, to be honest, whether it be ours or ChatGPT’s or any of them, where I’m truly like, “Oh no, this isn’t just a hack. This isn’t just RAG.” This is the one remnant of RAG that still exists, really, in the more traditional sense, and I am certain that there is a better way out there somewhere.

Oh, definitely. But I think another thing you’ve said is specialization, not generalization. For me, a lot of agentic engineering is emergent specialization. So it’s like, let’s take a big intelligence to crystallize a small intelligence to do the particular thing we’re doing.

But as we’re nearly out of time, the final question is synthetic data generation. So what are you guys doing about that?

Tons. [laughter] I don’t know how much of it I can get through in 5 minutes, but I’ll do my best. There are a number of areas where we’ve done synthetic data generation in the past. We’re doing a lot more of it in the more forward-looking sense for the sovereign model, because the sovereign model can’t just be good at software engineering. It has to be useful across the board, and that means that we have to get good at synthetic data generation, particularly in the RL realm, for things that aren’t just coding.

The bread and butter, though, is coding. The way that we’ve done this in the past is with a very cool and sophisticated pipeline that we’ve built out over the course of about a year and a half now. One of the big cold-start problems in coding RL, particularly if you’re using open-source repositories as a sort of seed data—and even if you’re using closed-source, it actually doesn’t matter—is that nearly all of the PRs, commits, or whatever you want to refer to them as, don’t have a built-in grader.

Right.

The ones that do are often bug fixes. The classic example, I guess, if you’re nerdy enough to be in the space, is an SWE-bench-style problem. You have a GitHub issue, you have a PR that fixed it, and then, because it’s open source, you have some sort of regression test that was added. That’s your seed data.

The real world doesn’t look like that, unfortunately, and software engineering in the broad sense doesn’t look like that. Meaning that, to do RL well, you still need to fundamentally be able to do tasks that aren’t bug fixes, which is the vast majority of what engineers do. You still need to be able to tell whether the agent got it right or not, broadly speaking.

At Cosine, we take real work that was done. We’re not magicking up made-up problems for the model to solve, because fundamentally, if the model can come up with a problem, it can probably solve it. We take real problems that were solved, whether it be feature work, refactoring, or whatever it is, and what we’re synthesizing is ground truths, graders, or ways of measuring whether that thing has been done.

It is a bit of a minefield because, particularly with RL, obviously it’s not supervised. Fundamentally, we need to be able to test these things in a way that isn’t too tightly coupled to the original implementation. There are many ways to skin a cat, as we know. What that looks like in practice is that you need an implementation-agnostic enough way of testing it that’s still rigorous enough to check functional correctness. It is a very fine line to tread.

We’ve done a lot of work around it, and we have a long pipeline built on it. We have custom post-trained models that live inside that pipeline that have essentially gotten good because we had to do a lot of manual labeling in places and stuff like that. What it has allowed us to do is have an autonomous pipeline, which is particularly important for enterprises.

We can point at essentially any programming language, any type of task, any stack, anything like that, and say, “Okay, I want RL data for this problem set,” and we can get a good chunk of it. For our Outpost model, when we were coming up with the languages that we wanted to get the model good at—things like Java, Fortran, C++, I could go on—we used that pipeline to gather the RL data for these things.

In many cases, and in some programming languages, there aren’t even test suites. That’s where it gets really hard. That’s where you have to get a bit inventive in terms of measuring whether the agent has gotten something right or not. I think it’s things like Verilog and SystemVerilog, where you have to actually run what’s called a synthesizer in your environment to be able to tell whether the chip actually works or not.

An EDA.

Yeah, exactly. Fortunately, I don’t run this pipeline. A chap called Ben does, and he knows far more about that than I do. But, yeah, that is broadly, within software engineering, how we’ve gotten very good at it, and it is something that we’re generalizing out into other use cases that aren’t just software engineering as well.

Very cool. So, in closing, you might argue that the US government has handed you a commercial advantage here, because now it’s more important than ever to build sovereign AI. You guys have this model coming out towards the end of this year, so do you think you’re going to be able to do it? But also, are you still at risk from a supply-chain point of view? Because so much hardware is controlled by America. How’s this going to pan out for you guys?

Naively, I think that if the hardware is already in the UK, I don’t know how much they can do about that. All of the infrastructure the model is going to be trained on already exists and is up in the UK because we’re doing it very soon, right? In fact, experimentation is already happening upstairs.

In terms of whether what’s happened recently has been a bit of a gift, given our positioning, absolutely, yes, without a doubt. It has been probably the busiest I have ever been since founding the company, in terms of people coming to us saying, “Okay, we now realize what you’re doing is really important. How can we be involved?” The consortium of companies that you read out is growing by the day.

The involvement and the urgency, importantly, from those companies, from government, and from just citizens as well has gone through the roof. I feel very fortunate and, to an extent, lucky that obviously we were well positioned to take advantage of this early. Also, we did put ourselves in that position, but also, thank you, Donald Trump.

How did we not see this coming, though? Because it was such an “Oh, shit” moment for so many people.

I think we did, though. I think a lot of people did. We certainly did, but it was always fobbed off as, “Oh, sure, okay, I suppose that could happen.” But I personally didn’t expect it to happen as soon as it did.

I’m also freshly surprised by the 5.6 news that I’m sure you’ve seen as well, where that’s going to be rolled out. Even for me, there’s a huge part of me being like, “Oh, man, that really sucks. I wanted to try that model, and I don’t know whether I’ll be able to now.” That might just be the existence for us now, unless we and others do work to get that level of performance out in some other way. That’s our job now, I guess.

Yeah. I mean, for the first time, I feel like a second-class citizen because those folks over there in America have better AI than I do. They’re going faster than I am, and I really hate that.

And believe me, that boils my blood more than anyone else. We are going to do everything we can to pull this off.

You mentioned, “How are you going to do this?” It’s like, we’re just going to make it happen. We have no choice but to make it happen.

Please do.

Yes, we are going to do everything we can to make it happen.

On behalf of everyone in the UK, please do.

We’ll do our best. Thank you.

It’s been a pleasure. Thank you so much.

Thank you very much for having me.