R1, OpenAI’s o3, and the ARC-AGI Benchmark: Insights from Mike Knoop on the Gradient Dissent Podcast

You’re listening to Gradient Descent, a show about making machine learning work in the real world, and I’m your host, Lukas Biewald. This is a conversation with Mike Knoop, who is both an AI researcher and an incredibly successful entrepreneur. He started a company called Zapier about 15 years ago with a very small amount of funding and grew it into a very large business. He then got up to speed on the frontier of AI research and recently started an organization called Ndea, which is one of the new research labs working on the forefront of AI. This is a really interesting conversation. We go into business and how AI fits into business, and then also some of the details on how the new R1 and R1-Zero models work, especially on the ARC-AGI Prize, which he funded and made popular. I really hope you enjoy this conversation.

Why don’t we start with R1 and work backward?

Sounds good. Basically, 2 actual models were released: R1-Zero and R1. They’re very similar in nature to what I would call OpenAI’s o1 model. They’re reasoning models, and they were trained in a similar way. We know for sure how R1-Zero and R1 were trained because DeepSeek chose to open-source its training methodology.

For R1-Zero and R1, they really did move the science forward. I saw a public comment from Mark Chen, who leads research at OpenAI, where he shared that he thinks o1 was trained in a similar spirit as well. I think there’s pretty good agreement between the ideas that went into creating both of these systems.

Fundamentally, the o-series from OpenAI and the R-series from DeepSeek represent a paradigm shift from the types of AI systems we’ve seen in the past. For example, OpenAI’s GPT series—from GPT-3 to GPT-3.5, GPT-4, and GPT-4o—has all followed the same broad paradigm of scaling up pretraining. We’re trying to make the models more intelligent by feeding them more data and making the models bigger.

It’s also the case that o1 was probably a little smaller. There was probably some distillation from GPT-4 to make it more efficient, but roughly, we’re still in the broad paradigm of making models smarter by giving them more human data. They’re effectively memorizing answers, which means they really have no ability to adapt to novelty.

1. Building Zapier: The early days and scaling automation

What ARC-AGI does is assess an AI system’s ability to solve problems that it hasn’t seen before. The data is highly resistant to simply memorizing the answers. Even if you’re given the training set for ARC, you can’t just memorize the training set and solve the test set.

This is another underappreciated point: ARC-AGI is legitimately very hard. That’s why version 1 went unbeaten for about 5 years. In December, we had this big news moment with OpenAI’s o3. They had a 75% score on ARC-AGI v1, and an 85% score with a really expensive, high-compute performance version of it.

This showed that this new type of reasoning system has a fundamental capability we haven’t had in computers before: the ability to adapt to novelty. That actually has implications beyond this quirky fact. It will lead to more robust and reliable AI systems, too, which we can touch on. It matters a lot for agents.

Before we go down that path, let me take us immediately off track, because I love it. When you talk about novelty, how do you define that? There’s a lot of debate about whether these systems are just memorizing or not. They’re obviously not simply memorizing text and regurgitating exactly the same text. It’s quite clear that there’s some adaptation.

There’s some generalization that comes from compression. It’s not literally a database, which would be a lookup table.

Exactly. If you had 1 parameter for every fact, you could just have a database.

These GPT-style systems do compression, and that’s where you get some of the interesting generalization. The claim I would make is that the amount of generalization they’re capable of doing is fixed because the architecture is effectively fixed.

You’ve got the Transformer underlying architecture in every language model up until last year. That means your level of intelligence is fixed. You can memorize more, but the amount of generalization you have—the amount of adaptability you can apply from your training data to a new situation—has been fixed. That didn’t change until o1.

o1 was the first prototype we saw last September of a system that legitimately had an increased amount of intelligence in its ability to be given a fixed amount of input information and do more things accurately farther away from its training data.

Would you say, for example, that AlphaGo couldn’t generalize? It seems like, within the domain of Go, it could reason pretty well and adapt to novel situations.

No. The same is true for ARC. ARC-AGI has been around for 5 years, and there are a lot of state-of-the-art systems on ARC. From a pure solver standpoint, disregarding the front-end language models, the best systems were getting around 50% in the 2024 ARC Prize contest.

But those are very domain-specific solvers. All of their generality is being put into the system that solves ARC during the contest. That comes from the researcher’s brain. The researcher thinks about the problem, models it, and says, “Here’s what I’m trying to get this computer to do. I’m going to encode my understanding of the problem into the system and the architecture in order to get this thing to work.”

2. How AI is transforming business automation

It ends up being fairly constrained in domain, and it doesn’t generalize well. The same is true for AlphaGo. We’ve had AI systems for years that can be superhuman at games, but the fact has historically remained that I could sit you down, teach you a new card game or board game in a couple of hours, and get you up to human-level proficiency.

I could take you into a totally new domain and teach you how to drive a car you’ve never driven before. You’d probably be proficient in a couple of days. That ability to adapt on the fly to a situation or type of problem that you’ve never seen before and never trained on has historically been very unique and special to humans relative to the AI systems we’ve had.

It’s probably worth saying that I don’t know how ARC is best described in words. It’s better to look at it. We’ll overlay a puzzle here somewhere.

Totally. It’s definitely better to overlay a puzzle. For someone just listening, what’s astonishing about ARC is how easy the puzzles seem to be, and how these systems fail on these easy-looking puzzles.

Maybe you could describe a little more of what it looks like.

It looks like an IQ test. It’s a grid of colors, a 2D grid. You’re given some examples of inputs and outputs, and your goal is to find the rule or pattern between them—the common, consistent rule between those inputs and outputs—and then apply it to a test case.

It’s easier to describe visually than verbally. ARC challenges you to recompose, on the fly, knowledge that you’ve acquired throughout your life—what we call core-knowledge priors. These include symmetry and rotation, object detection and tracking, and basic understandings of physics.

ARC requires you to abstract and compose those core-knowledge priors on the fly for a task that you’ve never seen before. This is what the historical language-model scaling paradigm was never very good at. GPT-4, for example, scored around 4% on the ARC dataset, in contrast to the impressive performance we started seeing from the o-series systems.

What changed from 4% to these o-series systems that are doing well? What do they do differently to make it work better?

Fundamentally, the strongest thing I can say is that there’s a very strong capability assertion, and then I have some informed speculation about how that capability was achieved.

You can see it in the score. If you look at ARC-AGI v1, over a 5-year period—from when it was introduced in 2019 through last fall—the best GPT-4-class systems got around 4%. Then o1 came out, o1 Pro came out, o3 came out, and the score rapidly went from 4% all the way up to 75% and 85% on the extreme high-end performance version of o3.

It looks like a straight line. It’s pretty remarkable. That’s actually a good thing to see in a benchmark because it means there’s more signal in the benchmark. It’s truly making a capability assertion. It’s harder to understand capabilities by looking at benchmark scores that monotonically increase smoothly over time. When you see a sharp bend, something has distinctly changed.

3. The ARC Prize: Why AI reasoning is the next frontier

I think that’s the case with these reasoning systems. Specifically, they added the ability to recompose knowledge that they had been trained on in their foundation models at test time. OpenAI calls this test-time compute. The broad paradigm is that we want to use more compute at test time to think before jumping to an answer.

GPT-4 starts spitting out tokens in about 500 milliseconds, and that’s its answer. The intuition here is that we want to allow these systems more time to think up front before they render an answer. They think using what’s called a chain of thought.

You might have heard the prompt, “Let’s think step by step.” That came from a paper published about 3 years ago. The paper was very special to me and was one of the things that first got me all-in on AI. This chain-of-thought moment happened in January 2022, and we’re still downstream of it, trying to figure out how to apply the paradigm of having models think out loud.

You say, “What’s the next step to solve this problem?” The model takes that step, then you ask, “What’s the next step?” It does that again, in a big chain. The model can use that entire thought trajectory to ground its final answer. That’s what o1 does, what R1 does, and what R1-Zero does.

Before we get there, maybe let’s take a moment to be astonished that this works. You’re saying these language models are compression algorithms that can’t really reason; they can only do limited generalization. Then they start generating text, and the chain of thought has them generating text in steps. Suddenly you claim they can reason. That doesn’t seem obvious.

I don’t think it is obvious. It’s one of the reasons I went all-in on AI back in 2022. Up until January 2022, I had founded Zapier 15 years earlier, and I was an executive at the time, running about half the company and building our new product initiatives. I had been paying attention to AI, and I thought I had a good perspective on what language models could and couldn’t do.

Then the chain-of-thought paper came out. Just by asking the models to think out loud, you saw performance scores on reasoning benchmarks spike. They grew from around 30% to around 70%. That was my “Oh, shoot” moment: are we on track for AGI with this technology?

I felt it was important to know this from a Zapier standpoint—should we start using this in our products?—but also as a human. This is some of the most important technology in the history of the world, and I wanted to understand the paradigm.

One interesting thing about that old-school version of chain of thought is that at Zapier, we were proudly using chain of thought in the fall of 2022 with a very similar paradigm. You ask the model to think out loud, and it has a very low degree of ability to adapt.

4. The shift from deep learning to reasoning models

We see this even with o1 and R1. The scores on ARC are only around 15%. That’s a big step up from the GPT-4-class systems in the 4% to 5% range, but it’s still a relatively weak amount of adaptation.

What has made these systems work with o1 Pro and o3—and this is informed speculation—is that they’ve added search and sampling on top of the chain-of-thought generation process. They’re not just asking for a single chain of thought and then saying, “Now give me the answer.” They generate multiple chains of thought in parallel, ask which one is best, use that one, and then proceed to the next step.

They’re effectively doing program synthesis or program search at inference time. That insight has enabled a significantly higher degree of adaptation, which is what gets them to 75% or 85%.

This is astonishing. These systems demand serious study, and that’s why I’m excited to see R1 and R1-Zero open-source. It will allow more people to do the science.

Before we get into R1 and R1-Zero, what is human performance on ARC? Where are the best models today? When you say 85%, how does that compare with human-level performance?

Smart humans can get basically 100% on ARC-AGI v1. The data we have is that if you take 2 STEM graduate students and put them in front of the benchmark, they’ll get 100%. The actual data is around 98% or 99% across 2 people.

That’s been one of the flaws of v1: we haven’t had strong assertions of human capability. That’s something we’re fixing in v2. We’ve been working on v2 for years. We put a lot of effort into building it last summer, and we’re putting the final touches on it now. We’re going to launch it with the 2025 ARC Prize.

One of the things we have with v2 is strong human-study baseline testing. We want to confidently assert that every puzzle in the benchmark is solvable by humans, which justifies the “easy for humans” claim.

I think that’s the guiding principle for ARC in the future. This is true for v1, and it will be true for v2, v3, and all future versions: ARC should represent things that are easy for humans and hard for AI. The gap we’re driving toward is zero.

If you could get that gap to zero, I think it would be hard for anyone to claim we don’t have AGI. If you can’t find a single thing that humans find easy but computers find hard, that’s a reasonable goal to set and a target to shoot for.

This contrasts with how a lot of other frontier benchmarks work. Dan Hendrycks’s Humanity’s Last Exam and Epoch AI’s FrontierMath are making ever more difficult benchmarks. I don’t discount that effort at all; I think it’s useful. But I think there’s something more important to understand from a capability-assertion standpoint: what do humans still find easy? That’s more revealing of what we’re still missing in these systems.

I totally agree. It’s funny, though. Throughout my lifetime, and especially in the last year or 2, there’s been this line that a benchmark represents human-level performance, and that once we get there, we’ll have AGI. Then, over and over and over, the systems beat the benchmarks.

It’s hard to make this claim with certainty, but do you feel like ARC could be the last benchmark of this genre? Is there something else out there that AI could solve, leaving some other benchmark that humans can do easily but AI can’t?

What I can say more confidently is that we’re going to drive the gap between easy for humans and hard for AI to zero. Whether that constitutes AGI is up to the world to decide. I personally think it does.

That’s the design philosophy we have for v2, and it will be the design philosophy for v3 and future versions. I’ll say that v2 will probably look fairly similar to v1 in terms of domain because we’ve been working on it for a long time. It will be harder for computers but still easy for humans.

5. How OpenAI’s O-series and DeepSeek are redefining AI capabilities

V3 will likely look very different from what we’ve done so far. I think it will still look like ARC, but it will test different capabilities that humans find quite easy and that current benchmarks don’t test. It will also likely include a formal way of measuring efficiency, which I think will be important for determining when we’ve reached AGI.

Can you describe what R1 and R1-Zero are and why they do well on ARC?

R1-Zero and R1 are trained on chains of thought and then generate them. They basically generate a single chain of thought and then give us an answer.

The important thing to understand is why there are 2 models and why R1-Zero and R1 are separate. DeepSeek didn’t have to release R1-Zero. In my view, R1-Zero is the more important system for understanding R1.

What does the Zero stand for?

The Zero stands for no human data in the training loop, like AlphaZero. They’re training it purely with reinforcement learning, using domains such as math and coding to create verifiers. DeepSeek-V3 generates a chain of thought, and then they can use a formal domain—literally running a computer program—to provide feedback.

Before we go down that path, they obviously had an earlier training step in which there was human data, right? How does it learn language otherwise?

There’s a foundation model, DeepSeek-V3, and it was trained on human data. But this isn’t a foregone conclusion. I think it’s going to be an engineering tradeoff that AI system developers have to make in the future: how much knowledge do we put in the foundation model we start from, and how much do we have the system generate itself at runtime?

My expectation is that you’ll use these reasoning-style systems to generate new knowledge and add it back into an ever-growing corpus of knowledge. You’ll probably use that corpus as the starting point for future training runs and inference systems.

So yes, R1-Zero was trained on human data through DeepSeek-V3. But that’s probably not going to remain the permanent case for these systems. If R1-Zero can bootstrap itself up in knowledge from first principles—math, for example—give it the basic arithmetic operators and let it bootstrap up calculus, that’s something a system like R1-Zero could potentially do using reinforcement learning with no humans in the loop.

They train a foundation model on human data. What’s the next step they take with R1-Zero specifically?

They generate a single chain of thought, step by step, and then use the entire chain of thought to ground a final response from the same DeepSeek-V3 model.

How is the feedback incorporated? Is it done offline at training time?

What I just described is test time. A user inputs a query, the model generates one very long chain of thought, and then it gives you a final answer.

The feedback loops happen offline at training time. The developers did that months ago. They took the DeepSeek-V3 model and had it generate lots of chains of thought. Then they had another computer program look at those chains and provide the training-loop feedback: was that a good thought or a bad thought?

That reinforcement-learning signal was used to fine-tune the ultimate R1-Zero model. In the case of R1, they also allowed human experts to label the chains of thought.

How do they have the original model that knows whether a thought is good or bad? How do you get started?

That’s what differs between R1-Zero and o1. o1 uses a grader model—a separate neural model that provides feedback and says things like “good job” or “bad job.”

R1 doesn’t do that. It uses a purely symbolic verifier. It takes the potential code the model has output, tries to run it symbolically, and uses that to provide a feedback signal about whether it’s good or bad.

That’s generally what you’re seeing in the reinforcement-learning space. You want to get a final answer, then symbolically verify it and make a 100% reliable assertion about whether it was right or wrong. You can do that in domains where you can run a computer program to verify the result: a Python program, a math equation, or a piece of code. Does the code compile? Does it produce the answer you expect?

This differs from some of the o1 work, where they’re using a process reward model—a separate, neural, trained model. Again, this is all informed speculation. DeepSeek hasn’t shared all of these details, so I’m trying to understand the systems based on what they’ve released.

My understanding is that o1 has a separate language model providing feedback during training, whereas R1 uses a symbolic verifier.

6. AI generalization vs. memorization: What’s changing?

Is it possible that there are domains, such as computer programming, where it’s easy to do chain of thought and verify the result, but other domains where verification is harder? Would that cause these models to specialize in domains where it’s easier to tell whether the result is accurate?

That’s the bet. Can R1-Zero scale up without adding humans to the loop? The evidence we have is that o3, which does much better on ARC, required supervised fine-tuning from humans to work at a high enough degree of efficiency to make it tractable.

We don’t yet have evidence that you can bootstrap a purely reinforcement-learning-based language-model chain-of-thought system to get there. That’s probably what you’ll see happen in 2025. I would expect that to be a major area of work.

Another obvious thing people have noticed is that these models were much cheaper to train—by orders of magnitude. Do you have a thought about why that is?

I haven’t looked deeply into this. The only commentary I can offer is that people are often comparing the sticker price—the commercial pricing rate of o1 versus DeepSeek’s commercial rate for hosting its model.

Because R1 is open-source, you can run it on your own infrastructure. You don’t have to pay the commercial rate. DeepSeek is giving you something that’s close to the actual cost, whereas OpenAI has a margin. OpenAI has researchers to support and future research to invest in. They’re building a business.

My informed speculation is that there’s quite a bit of margin built into o1’s inference costs right now in order to fund future R&D. The true costs are probably more comparable than most people expect, at least for R1 and R1-Zero versus o1.

There’s also a debate about how much DeepSeek-V3 cost to train. People see the headline that it cost $5 million to train R1-Zero and assume that number obscures the fact that a lot of money went into training the foundation model. That’s accurate. I have no idea how much it cost to train the foundation model, and I don’t know what GPT-4’s chain-of-thought cost was either.

I’d make this single comment about inference costs: I’d broadly put the R systems and the o systems in a similar bucket from a cost basis.

Let’s take a step back. I first met you more than a decade ago. You’re the founder of a company called Zapier, which I’ve long admired. Some people may not have heard of it. Do you want to talk about what Zapier does?

Zapier is an automation company. We try to deliver automation software that’s very easy for nontechnical people to use, primarily to automate parts of their businesses.

We’re used predominantly by individuals, people automating their own lives, small teams, and individual teams within large organizations. It’s designed to be very easy to use, in contrast with the automation systems that historically existed.

We’re used today by 3 or 4 million businesses in the United States, and we have a substantial international customer base as well. The core idea is that you can connect apps and use them together. Businesses use software such as Gmail, Slack, Salesforce, or Google Workspace, and they build internal workflows around those processes.

They often have humans who have to shepherd data between different applications and make decisions. Zapier can automate that completely, and it’s easy enough for line-of-business users that you don’t have to involve an engineer.

7. The role of program synthesis in AI’s future

One notable thing about Zapier is that you raised a tiny amount of money from venture capitalists, then got off the VC train. Almost nobody does that, and you grew to incredible scale. That’s been an amazing return for your initial investors. You were even unusual for raising money in the first place.

We debated whether to raise money in the first place. The 3 founders—me, Bryan Helmig, and Wade Foster—were all from the Midwest. I grew up in St. Louis, and there really wasn’t a venture-capital market in the Midwest in 2010.

The question was, how do you build the business? You build a useful service or product, sell it, and use the proceeds to invest back into the business. That was our basic approach.

We went through Y Combinator, and we were debating whether to raise money. Everyone else was going to raise money after Demo Day. We had office hours with Sam Altman, and we asked, “Should we raise or not?”

The question he asked us was, “What’s the constraint on the business?” It was a good question because the real constraint at Zapier was that Bryan and I were waking up and doing support until noon every day. That took time away from making the product better and reducing the amount of support we needed in the first place.

We realized we should hire a support person, but we didn’t have enough cash on hand to do that. So we decided to raise a small round to hire someone as soon as possible and give ourselves more product time.

The funny, or sad, anecdote is that by the time we raised the round, got the money in the bank, found the person, hired them, started payroll, and they received their first paycheck, revenue had caught up enough that we could have paid them directly.

I’m pretty sure you could trace the million dollars we raised through its lineage all the way to today. It did allow us to get started, though, and I think that was honestly the biggest value we got out of Y Combinator: the activation moment to go full-time on the company.

You were also remote-first from the beginning, right?

Yes. After Y Combinator, we were a globally remote team and have been since 2011. The only other companies at the time that I knew of that were fully remote were Automattic, the company behind WordPress, and 37signals.

The third interesting thing about Zapier, from my perspective, is that you were very early to use language models. You were one of the first companies I saw with real LLM use cases. Can you describe what those were and what that experience was like?

This goes back to the chain-of-thought paper that came out in January 2022. My background is engineering. I studied mechanical engineering and did optimization research, which turns out to use essentially the same math as a lot of this machine-learning work.

I didn’t figure that out until 2017, but as soon as I did, I thought, “I know how this works.” It demystified the technology for me, and I started paying more attention to the research side.

Zapier was growing, though, and I had other priorities. We had to grow the company, introduce new products, and so on, so I wasn’t paying that much attention. I read the GPT-2 paper—or the release—played with it, read the GPT-3 paper, and gave a presentation to the company about it. Then I thought, “Cool technology. We can do some basic things around the margin, but that’s it.”

Then the chain-of-thought paper came out in January 2022, and that made me think this might be extremely relevant to what Zapier’s customers were trying to do. I went to Wade, who was CEO, and said, “Wade, I need you to take back all of my responsibilities. You need to run product and engineering because I need to go do AI research and figure out what this means for our business and our customers.”

For 6 to 12 months, Bryan, our CTO, and I coded all day long and tried to understand what this technology could and couldn’t do. By the summer of 2022—4 or 5 months before ChatGPT came out—we had built a lot of the early prototypes.

8. Reliability as the key challenge in AI automation

We had built tree-of-thoughts prototypes and an internal version of ChatGPT using this technology. We had essentially prototyped all the foundational pieces in 3 or 4 months and identified the most obvious place Zapier could start: tool use.

Could we equip language models with the tools on Zapier’s platform—all the actions and search endpoints—and allow them to do more? The models had frozen weights and couldn’t take action in the world, but Zapier could provide them with tools.

That activated us very early to start building and delivering AI products. It also gave me an early perspective on the limitations of this paradigm. I spent time talking to hundreds of Zapier customers who were trying to deploy this AI technology in the middle of their automations.

Zapier has been deploying AI agents for 2 years now, so I’ve heard what people want from the technology and where it doesn’t work. The number-one problem they consistently tell me is that the promise is there and they understand what it could do for their business, but they don’t trust it enough to go hands-off.

Zapier is an automation product, which is different from ChatGPT. With ChatGPT, you type on a keyboard and get a response that you can audit. Zapier is running on a server online. You’re not watching it; you’re monitoring it.

The feedback was, “I don’t trust it because the reliability isn’t high enough for me to take a human out of the loop.” That feedback was consistent from GPT-3.5 through GPT-4 and GPT-4o. It didn’t match the scaling-law hype that was so strong in 2023 and 2024, and it didn’t match my lived reality.

I had 2 sets of facts that didn’t fit together, and I was trying to understand why. That’s when I rediscovered François Chollet through his appearance on the Lex Fridman Podcast. I had been thinking about ARC-AGI, but I really dug into it, read the paper “On the Measure of Intelligence,” which you published in 2019, and had my aha moment.

I thought that paper did a good job articulating the promise of this technology and why it’s impressive, while also explaining the fundamental limits we would hit by simply scaling up pretraining and memorization. That explained what I was hearing from customers.

Once I reached that conclusion, I thought ARC-AGI was clearly the most important benchmark in the world. More people should know about it. Going into last summer, it was still relatively obscure.

You created a prize for the benchmark. Do you want to talk about that?

That was the phase where you introduced me to François, so thank you for doing that. I hope the world gets a lot of leverage out of that introduction.

I had been surveying AI researchers in the Bay Area for 6 months to a year, asking, “Have you heard of ARC-AGI? I think this is the most important benchmark.” The awareness rate was relatively low. Maybe 10% to 20% of the people I met had heard of it.

Many of the people who thought they had heard of it had actually heard about an older version from the Allen Institute that had long been beaten by language models. ARC-AGI was a relatively obscure benchmark.

François and the ARC Prize team had been running a small version of a contest for several years, so there was evidence that it was a robust benchmark. It wasn’t completely obscure, but awareness in the AI industry was low.

I thought this was direct, concrete evidence that there were fundamental limits to scaling up pretraining. Every other benchmark in the world was saturating faster and faster, while ARC-AGI wasn’t. I concluded that awareness was the problem.

After your introduction, I flew to Seattle and had lunch with François. I pitched him some ideas about how to raise awareness. It was a fun conversation, but I also had some critical questions about why he thought awareness was so low and why he wasn’t working on it more. He had good answers to all of them.

9. The cost and efficiency of training frontier AI models

At the very end of my notes, I had written, “Pitch ARC Prize.” I had recently seen Nat Friedman and Daniel Gross run the Vesuvius Challenge, which had been very successful at raising awareness around an obscure problem and increasing interest in it.

I thought we could do something similar. That’s where the ARC Prize came from.

What happened when you launched the ARC Prize? Was it successful in getting new engagement?

Going into June 2024, maybe 10% of AI researchers had heard about ARC-AGI. By the end of 2024, everyone in technology had probably heard about it, so I think we solved the awareness problem.

My honest reaction is that I’ve been continually surprised by how much energy there has been around the benchmark. Here’s a concrete example. The contest ended in early November, and we had a number of teams on the leaderboard.

One of the requirements for winning the cash prize was that teams open-source their progress. The goal was to rebaseline progress in the community each year. The number-one team was a runaway winner. They had been at the top of the leaderboard all summer with a score around 55%.

The week before the contest ended, they emailed us and said, “We’re not sure we want to open-source our solution.” I thought, “Oh no. Is this going to work? Did we get something fundamentally wrong about how to structure the contest so that it helps the community make progress?”

Then, in the 72 hours before the contest closed, 2 other teams shot up the leaderboard from around 10th place to 2nd place. The number-two team was suddenly right behind the number-one team. There were also 3 papers published within 24 hours of the contest ending. They had timed their ARC research papers to drop at the same time and enter the paper contest.

There was a phenomenal amount of energy at the end of the contest that we hadn’t seen building. Something similar happened when OpenAI’s o1 model came out. There was an incredible outpouring of demand for us to test o1 on ARC-AGI. Thousands of people were asking us to test it, so we did.

It was important that we did, but these moments have surprised me in terms of the benchmark’s relevance and how much awareness we’ve been able to create around it.

What insights have come out of wrestling with the prize or the challenge? In AI, what approaches have worked? What have we learned about building intelligent systems?

10. Closing thoughts on AI’s trajectory

The classic way people tried to beat ARC for 4 years was through pure program synthesis. You build a domain-specific language—little Python functions or transformations—and have a human look at the puzzles and guess what the transformations could be.

Then you do a brute-force search over all possible combinations of those transformations to find one that matches the input and output. You apply it to the test case. That was the classic approach until this year.

It doesn’t work well. It’s slow, inefficient, and brute-force because all the generality is built into the domain-specific language that the human creates. At ARC 2024, there were a couple of new approaches that were interesting to see.

11. NDIA: Mike’s new AI research lab and its mission

The first was an induction-based approach. You generate lots and lots of Python programs using language models, then search over them. You also guide and inform the language model’s program generation using the input from the puzzle.

Ryan Greenblatt got an early score of around 40% using a technique like this. It was very inefficient—you had to generate hundreds of millions of Python programs—but it showed promise for this program-induction approach.

How do you pick among the programs? You generate them, but how do you pick which one looks promising?

You run them. The dataset has the answer, so you can evaluate whether a program produces the correct output.

This is true of math, coding, and ARC-AGI. It’s where o1, R1, and o3 dominate: in domains that are easy to verify. If a computer can check the answer quickly, with strong and exact correctness, these techniques work.

The other major approach was test-time training. The ARC dataset has a public set and a private set. The private set is what the Kaggle contest and the cash prize are attached to. Very few people have seen it, and you aren’t allowed to see it when you submit your test.

That gives us strong guarantees around the system’s ability to adapt and reduces the chance of cheating or fitting to the private set. What people figured out was that they could take the private dataset inside Kaggle and use it as a starting point to generate lots of similar data through data augmentation.

They might change the colors or mirror the grid—things that don’t change the semantic rule but generate lots of nearby permutations. They then fine-tune a model using several thousand or tens of thousands of those examples.

That worked. I think it was the approach that got one of the top systems to around 50%. Now we have 2 broad sets of evidence about how to use language models to adapt to novelty.

One is chain-of-thought search, which is what o1 Pro and o3 do. They perform lots of sampling and search at each thought step. The other is test-time training, where you take the situation and create data augmentations around it, feed those back into the model, and perform inference.

That’s a form of knowledge recomposition.

If ARC were broader and not just in this domain of pixels and changing colors, would that break the approach? It seems like you’ve constrained the domain by choosing that format.

I remember reading Douglas Hofstadter’s book as a kid, where he makes sequences of numbers and you try to guess the next number. That seems like a variant of an ARC reasoning challenge in a different structure.

That’s an underappreciated aspect of ARC, or at least a common misconception. ARC is not really a visual benchmark. I understand why people think that, because we render it visually. The intuition is that AI systems aren’t good at ARC because they aren’t good at visual domains yet.

ARC should be thought of more as a program-synthesis benchmark than as a visual benchmark. Classic program synthesis is exactly what you described: given a sequence of integers and an output sequence, figure out the rule that maps one sequence to the other.

There’s even a database called the On-Line Encyclopedia of Integer Sequences. Researchers have created hundreds of thousands of sequences that can be used for this kind of research.

You’re given an input sequence and an output sequence, and your goal as an engineer is to create a computer program that automatically figures out the rule. You might be surprised, but this is a very hard challenge.

Depending on the complexity of the program, how much hidden state there is, and how many hidden variables there are, programs can be long or complex. The longer the program gets, the more you run into the problem of having to enumerate possibilities.

Classically, researchers try to brute-force it. They search over every possible program that could potentially exist and check each one: “Let me plug in the input and see whether I get the output.” It’s extremely inefficient. The search scales exponentially, so it’s never really going to work in a reasonable amount of time.

That’s the rough form of the problem: you’re trying to create programs. You can map this onto o1 and o3 because you can get a better feel for how those systems work. You give the model a question, it thinks for a while, builds a large chain of thought, and gives you an answer.

One way to think about that chain of thought is as a program. It’s a natural-language program, but it is a program. It has individual steps, and each step transforms the latent space from the thought before it to the thought after it.

You’re constructing a program. What o1 Pro and o3 do is program recombination. They search over the space of possible programs, similar to classic program synthesis, trying to find the right one.

The big difference between program synthesis and deep learning is that program synthesis can learn out of distribution. It can find programs that generalize to data outside the training distribution. Deep learning can’t do that in the same way.

Deep learning is a paradigm where you give the model a lot of data, embed it on a high-dimensional manifold, and make quick, approximate judgments about new data near that manifold. You won’t get exact guarantees. You need a lot of data, and you get in-distribution accuracy.

With program synthesis, you’re looking for a program. You might need only a couple of examples. You don’t need a million or 100,000; you may need 3 examples to find the contours of the rule. Once you find the rule and have the program, it works for any input, no matter what input you give it.

It requires very little data and generalizes out of domain, but the problem is combinatorial explosion. You can’t search through all possible programs efficiently. In o1’s case, the system searches through possible reasoning chains.

How do you do this well?

The insight is to use the pros and cons of both sides and merge them together. You want to use the upside of deep learning—making quick, approximate sketches—to inform a search process so that the search isn’t brute-force.

Humans don’t sit here and think through 1,000 Python programs to solve an ARC puzzle. We use our intuition to generate a couple of possible answers, then symbolically verify them in our heads. We run through the steps and ask, “Is this right? Is this right? Is this right?”

If it isn’t right, we go back to the deep-learning part of our brain and say, “Give me some more ideas.” There’s a smooth back-and-forth between the 2 systems. We think that’s the fundamental substrate we can construct.

My co-founder Sean has done a lot of work on SWE-bench, and he got the highest score there. It’s a very different domain, but he’s using o3 and o1 to generate programs, run them, and try to figure out which one is best. The runtimes are expensive here, though.

I think cost is going to be a major factor. This is something the AI research field hasn’t fully reckoned with yet.

When we released the o3 results, we had to report them on a 2D graph where the x-axis was effectively token cost or cost per task. We’re entering a paradigm where you can spend more money to get higher accuracy and reliability. The curve is still logarithmic, so it isn’t linear, but you can spend more money to get a better answer in many domains.

That means you can’t report a single benchmark number anymore. ARC still needs to fix this for next year. We need a leaderboard with some kind of efficiency dimension because you can’t simply say, “That system got 75% on the benchmark.” You also have to ask how long it took and how much it cost.

Those are the questions we’ll need to answer in order to guide research toward efficiency. There are still a couple of human bottlenecks in these reasoning systems. Once we get past those, efficiency will be the main thing we have to figure out.

If you can use compute to get better answers, compute costs reliably go down. The shape of that curve matters too, doesn’t it?

I’m not sure I follow.

The slope, even on a log scale, of the tradeoff between cost and performance matters. I’d love to spend more money to get better answers in many domains.

There are several use cases where that’s going to start working this year. One of the major ones is agents.

Coming back to the Zapier anecdote, the number-one blocker for deploying automation with agents is that reliability and trust aren’t high enough. What does increasing the ability to adapt to novelty mean in practice? It’s not just a cute benchmark result.

It means you can more consistently get the same answer—not necessarily the right answer, but a consistent answer—from these systems. That gives humans the ability to steer them and control their behavior more precisely than before. It raises the reliability bar.

A lot of use cases where people want agents weren’t blocked by cost. People were willing to pay up to the rate of human labor. They were blocked because the systems didn’t work well enough. We’ll start unlocking those use cases by plugging systems like o1, o1 Pro, o3, and R1 into the planning steps of agent systems.

One thing in the back of my head during this interview is that we’re taking for granted that this kind of reasoning is important. I wonder if we’re losing people by showing these toy problems. What does this actually translate to in the real world?

I’ll show you. I have a funny story that led me down this AI-benchmarking path a couple of years ago.

When we were first building some of our AI agents at Zapier, we had one that I set up. Zapier has been a strong, longtime partner of OpenAI. I’ve been part of 3 major OpenAI launches, which has been fun. We use Slack, and we had a shared Slack channel with OpenAI.

When we were building some of our early agent prototypes, we built a system where we wanted the agent to automatically send a Slack message based on lead-management information coming from HubSpot. Our sales team was doing transcriptions, and those were feeding into the process.

The agent had 2 main functions. It needed to choose a Slack channel to send the customer information to, and it needed to write the body of the Slack message. We had the agent guess both based on the inbound message.

Our system let us give the agent hints about how to fill in those 2 fields. We could say, “For the Slack channel, use the testing channel. For the message, grab the lead’s first name, last name, phone number, and email, and build a little lead widget.”

When we first turned it on, we had a Slack channel called #testing. The name of the channel we had with OpenAI was something like #openai-partner-testing. The agent picked the partnership channel a couple of times and started sending customer information there.

That wasn’t good. This was a production system, and it was an important partnership. We didn’t want customer information shared there. We immediately turned it off, scrubbed the messages, and fixed the situation.

That was my first realization that reliability, overrides, and control would be extremely important for building trust. The first time you see something like that, you say, “Turn this thing off. Get it 10 feet away from me. I can’t deal with this. It’s putting my business in jeopardy.”

It made us realize early on that, in order to deploy AI agents, we needed to offer users hardcoded control over what the system was allowed to guess. With Zapier AI agents, you can allow the agent to guess if you want to, but the default is for you to choose a channel.

That gives you some certainty and guarantees. Or you can say, “Here are the 3 channels I want you to choose from.” It allows you to build in a hard guarantee.

All of this was an early insight that reliability matters enormously for deploying agents that automate tasks businesses and users care about. That is the kind of problem we’ll start to see solved with o1, o3, and R1.

If you build an agent and tell it to send something to the testing channel, and it works the first 3 times, you’ll expect it to keep working. That will become the truth. In the past, with the stochastic nature of pure language-model systems, you couldn’t make that guarantee without adding guardrails on top.

Why did you start Ndea as its own organization rather than work with OpenAI, given your close relationship with them? Why did you feel it needed to be a new organization?

The reality is that, if you look at the frontier of what most of these companies are working on, I don’t think they share our view about the importance of program synthesis.

I’ve spoken with people at the frontier labs, including OpenAI, and shown them the o3 results. I think the view is that this is still a deep-learning paradigm with some special search layered on top, and that the search isn’t the important part.

There are people at these companies who understand the importance of program synthesis, but I think the broad industry view is still that deep learning is scaling. We fundamentally disagree with that.

Program synthesis is at least 50% of the equation. Maybe that isn’t 50% from a compute-budget standpoint, but if you measure where the ideas for future, highly efficient AGI systems come from, half of them will come from program synthesis in some form.

AGI is such an important technology that anyone with a unique, differentiated idea should try it. That’s why we launched the ARC Prize: to inspire more people to try new things and return to the kind of AI research environment we had in 2018 and 2019, when everyone was trying different ideas.

I do think the ARC Prize has helped shift the Overton window. We’re starting to see more fundamental innovation from small startups. One of the biggest surprises for me was that I expected individual researchers and perhaps some large labs to work on ARC.

Instead, around 7 or 8 startups came to me during or after the contest and said they had changed their research direction or pivoted to work on ARC. That was exciting because it suggested the prize was having the impact we hoped for: getting people to explore more and increasing the overall probability that we figure this out.

We also have long-term ambitions for what we want to do with the technology, and those differ from the ambitions of many of the major players. But, purely from a technology standpoint, we started Ndea because we have a differentiated view that we think has a good chance of success and could increase the probability of getting to AGI quickly.

Are you also oriented toward making a product and making money?

I have a strong view on that. Ndea is a research lab. We launched it a few weeks ago. François and I started it as an intelligent science lab. Our view is that reaching AGI requires combining deep learning and program synthesis. We think o3 beating ARC-AGI v1 may mark the start of a 5-to-10-year scaling journey on program synthesis, similar to AlexNet beating ImageNet in 2012. We don’t yet have the Transformer for program synthesis, and the field is much smaller than deep learning, so we need to grow it. We don’t have a product offering in the near term. Our goal is to get to AGI by our definition: reducing the gap between things that are easy for humans and hard for computers to zero.

You shouldn’t expect us to make products before that. We may use some of our prototypes to advance the frontiers of science, though. Building AGI is step 1 for us. What really excites me is leveraging the technology to accelerate the pace of innovation at the frontier of different scientific fields.

AGI is going to solve a lot of problems. It’s already solving problems for Zapier customers, and it isn’t even AGI yet. People are going to use this technology to solve many problems, and that’s great. I fully support it.

What excites me more is accelerating into an unknown future. Consider the printing press. When it was introduced in the 1400s, about 600 years ago, there was fear and skepticism, but also excitement about the proliferation of knowledge, sharing knowledge globally, and exchanging ideas freely.

If you had asked people then to predict what 2025 would look like, they would probably have had a hard time imagining Wikipedia, AI trained on Wikipedia, or the fact that we can talk to computers. We’re so far down the technology tree from those initial moments that catalyzed everything.

That’s what excites me. It’s almost more of an adventure than a problem-solving exercise. There will be exciting things in the future that I can’t even tell you about yet.

To get there faster, the main constraint is that we need computers that can work autonomously to innovate. That’s the bottleneck on creating AGI, and I’m excited to help accelerate it.

Do you have a point of view on the timeline for reaching the definition you described—getting rid of all the gaps between what’s easy for humans and what’s hard for computers?

There’s another part of that definition worth considering: efficiency. I think we’ll probably reach that definition before we reach it at human efficiency.

I’d add an asterisk: we’ll have to say there are no remaining tasks that are easy for humans and hard for AI, and that we’re doing them at human-level efficiency.

What does human-level efficiency mean? Can you put a dollar amount on it? Is it your salary?

That’s literally something we’re debating right now: how to measure efficiency. I don’t think the industry knows. ARC Prize is trying to figure this out too.

You want something you can compare between humans and computers. FLOPs aren’t a great measure. You could use dollars because you can pay a computer to do a certain amount of work for a certain amount of money, and you can pay a human to do the same amount of work.

Time is interesting from a wall-clock perspective because computers can make time faster through parallelization, which is something they’re good at. Honestly, I don’t know the answer.

I’d probably argue that dollars are the best measure today. Everything fits into dollars: how much you pay a human for their labor rate and how much the compute costs. Dollars also track the price-performance improvements as compute becomes more efficient.

Let’s say human-level efficiency means that we’re paying humans to do these easy tasks that computers find hard, and paying computers to do the same tasks.

Something like that. I’ll add that asterisk.

My expectation, if I had to make some wild guesses, is that ARC-AGI v2 will be durable for 12 to 18 months. That’s my best guess based on how it’s performing on all the frontier systems we’ve tested.

It won’t be another 5-year benchmark, as v1 was, because v2 is in the same domain. It’s raising the bar for computers without raising the bar for humans. It should still be interesting and useful as a gradient—a tool that points research in a direction.

What I’m excited about is v3, which we’ve started designing and prototyping for next year. Our design goal for v3 is that it will be durable for 3 years. That’s our hope and goal, although we’ll obviously have to compare it with reality.

My expectation is that we won’t have solved “easy for humans, hard for AI” at human-level efficiency for at least the next 3 or 4 years.

One of the hardest things about making predictions is that it’s easy to make predictions around smooth scaling curves. It’s extremely hard to make predictions around step-function changes in capability.

If you have a smooth curve, you can track it and make future guesses. But if you look at ARC-AGI v1, it went from zero to 4% over 5 years, and then from 4% to 85% in 2 months. That’s extremely hard to predict.

There are 2 reasons. You don’t know whether the technology to do it exists in the world yet, and you don’t know whether the ideas exist yet. Even if they do exist, you don’t know whether someone has put them together into a system that can demonstrate the capability.

There are a lot of unknown variables when you try to predict when step-function capability jumps will happen. I don’t believe we’re in a pure scaling regime where you can draw a smooth curve over system sizes and project it into the future.

I don’t think we’ll have a job at that point. I treat this much more empirically. You can make informed guesses about when step functions might happen, but I think it’s in the public interest to understand that this is the reality: we’re still looking for step-function capability leaps, and we don’t know when they’ll appear.

Awesome. I think that’s a good stopping point for this interview. We covered a lot of content and a lot of wide-ranging topics. Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for future episodes.