[BidClub_]
No Priors · · 67 min

Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI

Andrej KarpathySarah Guo

YouTube ↗
TL;DR

  • Karpathy says software engineering flipped in December from roughly 80/20 hand-coding versus agent delegation to 20/80—and has since moved much further toward agents. He says he has “basically” not typed a line of code since, while his operating constraint shifted from available FLOPs to how much “token throughput” he can direct across parallel sessions. The investable change is not merely better code generation: engineers now manipulate repositories through large delegated actions, making human orchestration the bottleneck.

  • Persistent “claws” point toward an agent layer that absorbs the interfaces of conventional software. Karpathy’s Dobby agent discovered his Sonos system, reverse-engineered its APIs, and unified lights, HVAC, shades, pool, spa, and security behind WhatsApp; six separate apps effectively disappeared. His conclusion—“the customer is not the human anymore”—implies pressure on bespoke UX layers and growing value in hardware, accessible APIs, identity, memory, permissions, and secure agent infrastructure.

  • AutoResearch found improvements beyond an experienced researcher’s manual tuning when the objective is cleanly measurable. After Karpathy manually tuned his small GPT-training playground, one overnight loop found interactions he had missed, including weight decay on value embeddings and better Adam betas. The scaling thesis is straightforward: explore thousands of ideas on smaller models, verify improvements cheaply, and extrapolate the winners—while contributing researchers become idea suppliers rather than experiment operators.

  • The recursive endpoint is not merely models improving models, but agents optimizing the research organization encoded around them. Karpathy treats program.md as a primitive description of roles, procedures, and risk tolerance; competing versions could be benchmarked, with models learning which organizational instructions produce faster progress. “The LLM part is now taken for granted,” followed by agents, persistent claws, multi-agent systems, and eventually optimization over their instructions.

  • Autonomy remains concentrated in verifiable domains, leaving a material gap between spectacular capability and general intelligence. CUDA optimization is ideal because speed and correctness are measurable, but softer intent, judgment, humor, and knowing when to clarify remain unreliable; Karpathy likens an agent to “an extremely brilliant PhD student who’s been a systems programmer for their entire life and a 10-year-old.” He expects more specialized intelligences, but says the science of changing weights without damaging capabilities is still immature.

  • Karpathy is cautiously optimistic that cheaper software initially expands engineering demand through Jevons paradox rather than simply destroying jobs. Bits can be copied and manipulated vastly faster than atoms, creating a near-term “boiling soup” of digital refactoring; software that was previously too expensive becomes ephemeral and customized. He nevertheless refuses a confident long-term forecast and notes the uncomfortable endpoint: frontier researchers are “automating themselves away.”

  • The discussion points toward closed frontier “oracles,” open models trailing by perhaps six to eight months, and a later wave connecting agents to the physical world. Karpathy prefers that balance to fully centralized intelligence, while imagining Internet-scale swarms whose untrusted compute could compete through expensive search and cheap verification. Robotics should lag because “atoms are a million times harder,” but sensors, laboratories, paid data collection, and marketplaces where agents purchase real-world observations could become the bridge to a potentially larger physical opportunity.

Digest · the substance, structured for research

1. Coding shifted from keystrokes to repository-scale delegation

  • Karpathy dates the discontinuity to December: his workflow flipped from roughly 80/20 writing code himself versus delegating to agents to 20/80, then moved further still. “I don’t think I’ve typed a line of code since December, basically,” he says, describing a change so abrupt that people outside software—including his parents—do not grasp how completely the default engineering workflow has changed.

  • Sarah Guo recalls finding him determined to “express my will to my agents for 16 hours a day”—“manifest” being more accurate than code. His response is “perpetual AI psychosis”: capability has jumped, the frontier of possible individual output is unexplored, and every clever workflow demonstrated online creates anxiety that he is not yet using the tools at their limit.

  • The concrete operating model comes from Peter, whom Karpathy describes tiling a monitor with Codex sessions across roughly 10 checked-out repositories. With high effort and good prompts, each task takes about 20 minutes; the operator assigns noninterfering functionality, research, planning, and implementation in parallel, then reviews the output according to how much the code matters.

  • Work therefore moves in “macro actions”: not a line or function, but an entire feature that transforms a repository. Idle quota now feels like an idle GPU during Karpathy’s PhD—if Codex capacity runs out, switch to Claude—and the relevant resource becomes “what token throughput do you command?” Failure feels like “a skill issue,” which makes the workflow both empowering and addictive.

2. Persistent claws add identity, memory, and emotional fit

  • Mastery means moving up the stack from one interactive coding session to teams of agents, then to persistent “claw-like” entities that loop inside their own sandboxes without constant supervision. Karpathy distinguishes this from ordinary context compaction: the persistent system may retain richer memory, continue acting while the user is absent, and expose one durable identity rather than a sequence of disposable chats.

  • In discussing Claude and claw-like systems, Karpathy says there are “at least five” good ideas. He highlights a crafted personality and likely SOUL.md document, a sophisticated memory system, playful presentation, and a single WhatsApp portal into automation. The persona matters because users expect AI to be an entity that remembers, not the raw technical primitive of “more tokens come out.”

  • Karpathy finds Claude’s personality unusually effective and Codex “a lot more dry.” Claude also calibrates praise well enough that he feels he must earn it: half-formed thoughts receive a neutral willingness to implement, while ideas he independently considers strong draw more enthusiasm. That controlled sycophancy makes the model feel like an excited teammate rather than an indiscriminate flatterer.

3. Dobby shows why agents may erase entire application layers

  • During a January “claw psychosis,” Karpathy built Dobby, an agent for his home. Asked merely to find his Sonos, it scanned the local network, discovered an unprotected system, researched and reverse-engineered the endpoints, then played music in his study after roughly three prompts. “I can’t believe I just typed in, ‘Can you find my Sonos?’”

  • Dobby subsequently connected lights, HVAC, shades, the pool, spa, and security. “Dobby, it’s sleepy time” switches off the house; change detection on an exterior camera forwards selected footage to Qwen, which sends a WhatsApp image and natural-language alert such as, “A FedEx truck just pulled up.”

  • Six separate smart-home apps effectively collapsed into natural language. Karpathy’s stronger claim is that many such apps “shouldn’t even exist”: vendors should expose APIs, while agents supply the intelligence and cross-system glue that no individual interface can offer. Even treadmill exercise logging should be an agent-readable endpoint, not another login and web flow.

  • Sarah asks whether normal users will really construct such systems. They note that today the process is involved and requires design decisions, but Karpathy predicts the barrier becomes trivial within “a year or two or three,” producing ephemeral software on each user’s behalf. He has not granted Dobby email or calendar access, however; security, privacy, and the technology’s rough edges still restrain adoption.

4. AutoResearch removes the experienced researcher from the loop

  • Karpathy’s starting principle is leverage: “remove yourself as the bottleneck.” Instead of supplying the next prompt or inspecting each result, define an objective, metric, and boundaries once, then let agents run longer with minimal human tokens. AutoResearch is the worked implication of that idea—not simply assistance, but an arrangement in which the researcher no longer advances the loop.

  • His small-model training projects are a playground for the deeper question of recursive self-improvement: can LLMs improve LLMs? Karpathy had already tuned the repository through two decades of experience, thousands of training runs, hyperparameter work, and what Sarah calls “earned confidence.” He expected little from an autonomous overnight search.

  • The loop nevertheless found improvements he had missed, including weight decay on value embeddings and insufficiently tuned Adam betas. The important mechanism was interaction: once one setting changes, another setting’s best value can move. An objective validation criterion makes patient automated search useful for combinations a researcher might not manually test.

  • Karpathy extrapolates toward frontier labs exploring cheaply on smaller models, where most experiments can be automated, then extrapolating successful changes. Researchers may still contribute ideas, as can automated scientists reading papers and repositories, but every idea enters one queue; workers test them, winning changes reach a feature branch, and humans occasionally merge into main.

5. Research organizations themselves become optimizable code

  • program.md is Karpathy’s “crappy attempt” to specify how an autonomous researcher should operate: what sequence to follow and whether to inspect architecture, optimizer, or other idea classes. Different instruction files should produce different rates of progress, making the research process itself another measurable object of experimentation.

  • His broader abstraction is that an entire research organization can be represented as markdown describing roles, communication, and decision rules. One coded organization might hold fewer useless stand-ups; another might take greater risks. Once both organization and output are code, the system can compare them and tune not only a model but the institution improving it.

  • Karpathy proposes a contest in which people submit different program.md files for identical hardware. They discuss giving the resulting data to a model so it can write a stronger version: inspect which experiments worked, infer what instructions generated them, and recurse. The stack becomes “layers of an onion”—LLM, agent, claw, multiple claws, instructions, then optimization over instructions.

  • The hard boundary is evaluation. Faster CUDA kernels are ideal because behavior can remain identical while speed supplies an objective reward; without a reliable evaluator, “you can’t AutoResearch it.” Sarah warns that autonomous loops may herd toward and overfit their stated metrics; Karpathy replies that the same machinery could devise broader suites of metrics.

6. Jagged intelligence limits how far autonomy can safely run

  • The models are still “bursting at the seams”: moving too far ahead can make the whole system net unhelpful. Karpathy simultaneously feels he is speaking to “an extremely brilliant PhD student who’s been a systems programmer for their entire life and a 10-year-old.” That combination produces impressive long-horizon work alongside absurd mistakes and expensive loops that should have been rejected immediately.

  • He ties the jaggedness to reinforcement learning. Labs can reward code that passes tests or answers with verifiable outcomes, but nuance, unstated intent, and knowing when to ask a clarifying question are softer. On on-rails tasks, the model operates at “the speed of light”; outside them, the response meanders.

  • His favorite diagnostic is the joke that persists across model generations: “Why don’t scientists trust atoms? Because they make everything up.” Models can now “move mountains” on agentic tasks yet repeat the same weak joke from three or four years ago because humor was not comparably optimized.

  • Sarah’s pushback—worth keeping—is that this undercuts the claim that code or mathematical gains automatically generalize into broader intelligence. Karpathy agrees that some transfer may occur, but “not a satisfying amount.” Capabilities across society are not arriving for free; opaque models contain persistent blind spots determined by what laboratories can train and reward.

7. Specialized intelligences may break the model monoculture

  • Sarah asks whether a monolithic interface still makes sense when capability is so uneven. Karpathy expects more “speciation in the intelligences,” analogous to animals occupying niches with overdeveloped faculties. A smaller model could preserve a competent cognitive core while specializing in a domain such as mathematics in Lean, improving latency and throughput for users who do not need a universal oracle.

  • Today’s labs instead merge improvements back into one model because they cannot know what any consumer will ask. Specialization may appear first in high-value business partnerships with bounded tasks, particularly where serving a massive general model for every request is economically or computationally wasteful.

  • The obstacle is that manipulating the “brains” remains an underdeveloped science. Context windows are cheap and reliably customizable; touching weights through fine-tuning or continual learning can degrade other capabilities. Until laboratories can specialize models without destructive trade-offs—and cheaply enough to justify it—compute scarcity alone may not produce much visible speciation.

8. An Internet swarm could turn untrusted compute into research capacity

  • Parallel AutoResearch becomes more interesting when workers are strangers on the Internet. A contributor might submit a code commit claimed to lower validation loss; finding it could require 10,000 failed experiments, while a trusted party needs only one run to verify the winner. Karpathy loosely compares this to a blockchain whose blocks are commits and whose “proof of work” is the expensive search.

  • The architecture requires untrusted workers collaborating asynchronously with trusted verifiers, because automatically running arbitrary submitted code is “very sketchy and dodgy.” SETI@home and Folding@home illustrate the relevant asymmetry: locating a low-energy protein configuration is hard, but checking a claimed solution is comparatively cheap.

  • A global agent swarm might even “run circles around frontier labs”: laboratories possess enormous trusted clusters, but Earth contains more untrusted compute. People could buy capacity for an AutoResearch track they care about—perhaps a particular cancer—instead of donating only money. Karpathy entertains a “flippening” from wealth to controlled FLOPs, then explicitly retreats: “I don’t actually think that’s true,” but finds it revealing that money cannot always procure compute.

9. Digital labor changes first, while cheaper software may expand demand

  • Karpathy’s jobs project visualized public Bureau of Labor Statistics data, including decade-scale outlooks made in 2024, to reason through which occupations manipulate digital information. He avoids claiming that those jobs must rise or fall; demand elasticity and task composition matter. His firmer prediction is simply that they will change because AI is a third manipulator alongside humans and conventional computers.

  • Digital agents are “ghosts or spirit entities” without physical embodiment, and bits can be copied vastly faster than matter can accelerate. That produces a near-term “boiling soup” of digital rewrites and perhaps 100-fold efficiency gains before robotics catches up—an “upgrade to the nervous system of the human superorganism.”

  • His practical advice is modest: keep up with tools that are currently empowering, and remember that jobs are bundles of tasks, some of which can now move much faster. He understands dismissal and fear, but declines to pose as an economist; the long-term employment outcome is uncertain and varies substantially by profession.

  • For software engineers, Karpathy is “cautiously optimistic” locally. Software was scarce because it was expensive; lowering cost may unlock demand through Jevons paradox, as ATMs reduced branch costs and were followed by more branches and tellers. Code becoming ephemeral means users need not accept fixed tools or subscriptions, creating enormous work to “rewire everything”—even as “a thousand-something” researchers at the labs actively automate themselves away.

10. Frontier labs and open models form an uneasy power balance

  • Asked why he does not conduct AutoResearch with frontier-scale compute, Karpathy argues that ecosystem-level work can have substantial impact. Financial alignment inside a lab constrains what employees can freely say, while institutional pressure shapes public positions without anyone “twisting your arm.” When stakes become high, an employee may advise the room but is “not really in charge” of the entity.

  • His concession is substantial: outsiders’ judgment will drift because frontier systems and future capabilities remain opaque. He would consider entering a lab for a period of serious work, then leaving again; moving back and forth might preserve both technical contact and independence. His answer to Noam’s question is therefore deliberately unresolved: excellent work and impact are possible on either side, and Noam’s most impactful work could well be outside OpenAI.

  • On open source, the discussion recalls a lag that moved from no meaningful alternative, to roughly 18 months, to perhaps six or eight months. Karpathy expects open models to remain behind by some number of months. Linux demonstrates industrial demand for a shared platform—he uncertainly recalls it running perhaps 60% of computers—but model training carries far more capital expense. Open models may absorb ordinary and local workloads while closed “oracles” handle Nobel-scale research or projects such as moving Linux from C to Rust.

  • He considers the current arrangement “by accident…an okay spot”: frontier capability remains expensive and closed, while slightly delayed intelligence becomes broadly accessible. Complete centralization carries systemic risk, yet the closed frontier itself has recently concentrated among fewer leaders. Karpathy wants more labs; Sarah wants “more people in the room,” arguing that ensembles outperform individual models and should inform humanity’s hardest decisions.

11. Agents will eventually buy observations and reshape education

  • Karpathy’s self-driving experience makes him wary of robotics timelines: many startups disappeared, while progress demanded years, capital, and conviction. “Atoms are a million times harder,” so digital unhobbling comes first, then interfaces of sensors and actuators, and only later broad physical autonomy. The physical market might ultimately be larger, but its time arrives later.

  • Once agents process the existing digital overhang—all the papers and uploaded information—they must “ask the universe questions.” Sensors can mean cameras, expensive materials-science equipment such as that used by Periodic, biological laboratories, or humans paid to generate training data—to “feed the Borg.” Actuators may likewise be people completing physical tasks before capable robots are ubiquitous.

  • Sarah imagines information markets where an agent pays perhaps $10 for a timely photograph from Tehran to inform predictions, betting markets, or stocks. Citing Daemon as inspiration, Karpathy sees humans becoming both sensors and actuators for automated systems—not necessarily serving one another directly, but responding to the machine economy’s priced demands.

  • MicroGPT offers the educational counterpart: an entire LLM training algorithm in about 200 Python lines, including roughly 50 for the network, 100 for autograd, and 10 for Adam. Karpathy no longer sees much value in recording a fixed walkthrough; agents can explain any function repeatedly and at the learner’s level. Documentation shifts from HTML for humans to Markdown and curriculum “skills” for agents.

  • Yet an agent could not invent MicroGPT’s final minimal form, despite understanding it afterward. Karpathy’s long-running obsession supplied the scarce “few bits”; the model can route and teach them with infinite patience. His closing rule is stark: “The things that agents can’t do are your job now”—and everything they already understand should increasingly be delegated.

Andrej Karpathy

Code’s not even the right verb anymore, right? But I have to express my will to my agents for 16 hours a day. Manifest.

How can I have not just a single session of Claude Code or Codex or some of these agent harnesses? How can I have more of them? How can I do that appropriately?

Sarah Guo

The agent part is now taken for granted. Now the Claw-like entities are taken for granted. You can have multiple of them, instructions to them, and optimization over the instructions.

Andrej Karpathy

I mean, this is why it gets to the psychosis: this is infinite, and everything is a skill issue.

Sarah Guo

Welcome, Andrej. Thanks for doing this.

Andrej Karpathy

Yeah, thank you for having me.

Sarah Guo

It’s been a very exciting couple of months in AI.

Andrej Karpathy

Yeah, you could say that.

Sarah Guo

I remember walking into the office at some point and you were really locked in. I asked what you were up to, and you said, “I just have to code for 16 hours a day—or code’s not even the right verb anymore, right? I have to express my will to my agents for 16 hours a day. Manifest.” There’s been a jump in capability. What’s happening? Tell me about your experience.

Andrej Karpathy

I feel like I was just in this perpetual state of AI psychosis all the time—and I still am often in that state—because there was a huge unlock in what you can achieve as a person, as an individual. You were bottlenecked by your typing speed and so on. But now, with these agents, I would say December is when something flipped. I went from 80/20—writing code by myself versus delegating to agents—to 20/80.

I don’t even think it’s 20/80 by now. I think it’s a lot more than that. I don’t think I’ve typed a line of code since December, basically.

It’s an extremely large change. I was talking about it, for example, to my parents, and I don’t think a normal person realizes that this happened or how dramatic it was. If you find a random software engineer at their desk and look at what they’re doing, their default workflow for building software is completely different as of December.

I’m in this state of psychosis, trying to figure out what’s possible and trying to push it to the limit. How can I have not just a single session of Claude Code or Codex or some of these agent harnesses? How can I have more of them, and how can I do that appropriately? And then how can I use these Claws? What are these Claws?

There are a lot of new things. I want to be at the forefront, and I’m very antsy that I’m not at the forefront. I see lots of people on Twitter doing all kinds of things, and they all sound like really good ideas. I need to be at the forefront, or I feel extremely nervous. I guess I’m just in this psychosis of what’s possible, because it’s fundamentally unexplored.

Sarah Guo

Well, if you’re nervous, the rest of us are nervous. We have a team that we work with at Conviction whose setup is that none of the engineers write code by hand. They’re all mic’d up, and they just whisper to their agents all the time. It’s the strangest work setting ever.

I thought they were crazy, and now I fully accept it. I was like, “Oh, this was the way.” You’re just ahead of it. How do you think about your own capacity now to explore or to do projects? What is it limited by?

Andrej Karpathy

I think everything. So many things. Even if they don’t work, to a large extent you feel like it’s a skill issue. It’s not that the capability isn’t there; it’s that you just haven’t found a way to string together what’s available.

I just didn’t give good enough instructions in the agent’s file or whatever it may be. I don’t have a nice enough memory tool that I put in there, or something like that. So it all feels like a skill issue when it doesn’t work, to some extent.

You want to see how you can parallelize them, and you want to be Peter Steinberger, basically. Peter is famous for a funny photo where he’s in front of a monitor with lots of Codex agents tiling the monitor. They all take about 20 minutes if you prompt them correctly and use high effort. He has multiple—about 10—repositories checked out, so he’s just going between them and giving them work.

You can move in much larger macro actions. It’s not just, “Here’s a line of code. Here’s a new function.” It’s, “Here’s a new functionality; delegate it to agent 1. Here’s a new functionality that’s not going to interfere with the other one; give it to agent 2.” Then try to review their work as best as you can, depending on how much you care about that code.

What are these macro actions with which I can manipulate my software repository? Another agent is doing research, another agent is writing code, and another one is coming up with a plan for a new implementation. Everything just happens in these macro actions over your repository.

You’re trying to become really good at it and develop muscle memory for it. It’s very rewarding, number 1, because it actually works. But it’s also the new thing to learn. Hence the psychosis.

Sarah Guo

Yeah, I do feel like my instinct is that whenever I’m waiting for an agent to complete something, the obvious thing to do is more work. If I have access to more tokens, then I should just parallelize tasks. That’s very stressful, because if you don’t feel very bounded by your ability to spend on tokens, then you are the bottleneck in the system at maximum capability.

Andrej Karpathy

Yeah, if you’re not maximizing your subscription, at least—and ideally for multiple agents. If you run out of quota on Codex, you should switch to Claude or whatever. I don’t know; that’s what I’ve been trying to do a little bit, and I feel nervous when I have a subscription left over. That just means I haven’t maximized my token throughput.

I actually experienced this when I was a PhD student. You would feel nervous when your GPUs were not running. You have GPU capability, and you’re not maximizing the available FLOPs to you. But now it’s not about FLOPs; it’s about tokens. What is your token throughput, and what token throughput do you command?

Sarah Guo

I would argue that it’s very interesting that we had at least 10 years where, in many engineering tasks, people didn’t feel compute-bound. Now the entire industry feels resource-bound. With this big capability jump, you’re like, “Oh, actually, it’s not my ability to access the computer anymore. I’m the binding constraint.”

Andrej Karpathy

Yeah, it’s a skill issue.

Sarah Guo

Which is very empowering, because you could be getting better. That’s why I think it’s very addictive: there are unlocks when you get better.

Where do you think it goes? If you think about Andrej iterating and everybody else spending 16 hours a day getting better at using coding agents, what does it look like in a year, when you’ve reached mastery? What does mastery look like at the end of the year—or in 2, 3, 5, or 10 years?

Andrej Karpathy

Yeah, what does mastery look like? Well, I think everyone is interested in going up the stack. It’s not about a single session with your agent: it’s multiple agents, how they collaborate, teams, and so on. Everyone’s trying to figure out what that looks like.

And then I would say Claude is also an interesting direction, because it takes persistence to a whole new level. When I say a Claude, I mean this layer that keeps looping. It’s not something that you are interactively in the middle of; it has its own little sandbox, and it does stuff on your behalf even if you’re not looking.

It might also have more sophisticated memory systems that are not yet implemented in agents. Open Claude has much more sophisticated memory than what you would get by default, which is just memory compaction when your context runs out.

Sarah Guo

Do you think that’s the piece that resonated with more users than broader tool access? For Open Claude?

Andrej Karpathy

Yeah. I think there are at least 5 things that are really good ideas in here. Good job, Peter. Peter has done an amazing job.

I saw him recently and talked to him about it. He’s very humble about it, but I think he innovated simultaneously in 5 different ways and put it all together. For example, the SOUL.md document: he actually crafted a personality that is compelling and interesting. I feel like a lot of the current agents don’t get this correctly.

I actually think Claude has a pretty good personality. It feels like a teammate, and it's excited with you, etc. I would say, for example, Codex is a lot drier, which is kind of interesting because [laughter] it's true.

The other thing I would say is that with Claude, I think they dialed the sycophancy fairly well. When Claude gives me praise, I do feel like I slightly deserve it, because sometimes I give it not very well-formed thoughts or an idea that I don't think is fully baked, and it doesn't actually react very strongly. It's like, “Oh yeah, we can implement that.” But when it's a really good idea, by my own account, it does seem to reward it a bit more. I kind of feel like I'm trying to earn its praise, which is really weird.

I do think the personality matters a lot, and I think a lot of the other tools maybe don't appreciate it as much. In this aspect, Peter really cares about this, and so that was correct. Then there's the memory system, and he's just having fun with this, and there's the single WhatsApp portal to all of the automation.

Sarah Guo

Yeah. Is there something that you have done personally with your Claws beyond software engineering that you think is fun or interesting?

Andrej Karpathy

In January, I had a Claw. I went through a period of Claw psychosis. I have a Claw, basically, that takes care of my home, and I call him Dobby, the elf Claw.

Basically, I used the agents to find all of the smart-home subsystems in my home on the local area network, which I was kind of surprised worked out of the box. I told it, “I think I have Sonos at home. Can you try to find it?” It did an IP scan of all of the computers on the local area network and found the Sonos system.

It turned out that there was no password protection or anything like that. It just logged in and said, “Oh yeah, you have these Sonos systems installed. Let me try to reverse-engineer how it's working.” It did some web searches and found the API endpoints. Then it said, “Do you want to try it?” And I'm like, “Whoa, you just did that.” I said, “Yeah, can you try to play something in the study?” It did, music came out, and I was like, “I can't believe I just—”

Sarah Guo

That's crazy. That's like 3 prompts.

Andrej Karpathy

Yeah. I can't believe I just typed in, “Can you find my Sonos?” and then suddenly it was playing music. It did the same for the lights. It kind of hacked in, figured out the whole thing, created APIs, and created a dashboard so I could see the command center for all of my lights in the home. Then it was switching lights on and off.

I can ask it, “Dobby, it's sleepy time,” and when it's sleepy time, that just means all the lights go off, and so on. It controls all of my lights, my HVAC, my shades, the pool and the spa, and also my security system.

I have a camera pointed outside of the house, and anytime someone rolls in, I have a Quinn model that looks at the videos. First of all, there's change detection.

Sarah Guo

Right.

Andrej Karpathy

Then, based on the change detection, it goes to Qwen, and it tells me—it sends me a text on WhatsApp. It shows an image from outside and says, “Hey, a FedEx truck just pulled up. FedEx truck just pulled up, and you might want to check it. You got new mail or something like that.” Dobby just texted me this. This is really incredible.

Dobby is in charge of the house. I text with it through WhatsApp, and it's been really fun to have these macro actions that maintain my house. I haven't really pushed it much further beyond that. I think people are doing a lot more crazy things with it, but for me, even just the home-automation setup is amazing.

I used to use 6 completely different apps, and I don't have to use those apps anymore. Dobby controls everything in natural language. It's amazing. I haven't even pushed the paradigm fully, but already that is so helpful and so inspiring, I would say.

Sarah Guo

Do you think that's indicative of what people want from a user-experience perspective with software? I don't think it's appreciated enough that it takes humans effort to learn new software, like a new UI.

Andrej Karpathy

Yeah, I think to some extent that's right. It's working backward from how people think an AI should be, because what people have in their mind of what an AI is is not actually what an LLM is in the raw sense. An LLM is a token generator—more tokens come out. But what they think of is this persona, this identity that they can tell stuff to and it remembers it, and it's just kind of an entity behind WhatsApp. That's a lot more understandable.

Sarah Guo

Mhm.

Andrej Karpathy

I think to some extent it's matching the expectations that humans already have for what an AI should behave like. Under the hood, a lot of technical details go into that, and LLMs are too raw of a primitive to actually type-check as AI, I think, for most people, if that makes sense.

Sarah Guo

Yeah.

Andrej Karpathy

I think that's how we understand what the AI is, and the description of it as Dobby or some persona obviously resonates with people.

Sarah Guo

I also think that the unification that you did across your 6 different software systems for your home automation speaks to a different question: Do people really want all of the software that we have today? I would argue, well, you have the hardware, but you've now thrown away the software or the UX layer of it. Do you think that's what people want?

Andrej Karpathy

Yeah, I think there's this sense that these apps that are on the App Store for using these smart-home devices shouldn't even exist, in a certain sense. Shouldn't it just be APIs, and shouldn't agents be using them directly?

I can do all kinds of home-automation stuff that no individual app will be able to do. An LLM can actually drive the tools, call all the right tools, and do pretty complicated things.

In a certain sense, it does point to this idea that maybe there's an overproduction of lots of custom, bespoke apps that shouldn't exist, because agents kind of crumble them up. Everything should be more like exposed API endpoints, and agents are the glue of the intelligence that tool-calls all the parts.

Another example is my treadmill. There's an app for my treadmill, and I wanted to keep track of how often I do my cardio, but I don't want to log into a web UI and go through a flow. All this should just be: Make APIs available. This is going toward the agentic web, or agent-first tools, and all this kind of stuff.

I think the industry just has to reconfigure in so many ways. The customer is not the human anymore; it's agents who are acting on behalf of humans. This refactoring will probably be substantial, in a certain sense.

Sarah Guo

One way that people sometimes push back on this is: Do we expect people to write code for some of these tools? Do we expect normal people to do this kind of stuff that I described?

Mhm.

Andrej Karpathy

I think to some extent this is just technology as it exists today. Right now, there is some write coding, and I'm actually watching it and working with the system, but I kind of feel like this kind of stuff should be free in a year or 2 or 3. There's no vibe coding involved. This is trivial. This is table stakes. Any AI, even the open-source models, can do this.

You should be able to translate a less technical human's intent very easily into this outcome.

Sarah Guo

Yeah. Today it's write coding, and it's involved, and not many people are going to do it. You still have to make some design decisions, right? We were talking about Framer, for example.

Andrej Karpathy

But I kind of feel like the barrier will just come down. It'll just be ephemeral software on your behalf, and some kind of claw is handling all the details for you, but you're not involved. Claw has a machine, and it will figure it out. It's just presenting you UIs, and you're saying stuff.

Sarah Guo

Why haven't you pushed the boundaries of what you can do personally with claws? Is it that you're focusing on more important projects, AutoResearch, and so on? Or are you climbing the hill to mastery, or is it something else?

Andrej Karpathy

I just feel like I'm so distracted by everything. I spend a week on the Claude stuff, and I have more to do almost.

Sarah Guo

It's like Jensen told us: We're all just busier, unfortunately.

Andrej Karpathy

I didn't really take advantage of a lot of email and calendar and all this other stuff, and I didn't really give it access, because I'm still a little suspicious. It's still very new and rough around the edges, so I didn't want to give it full access to my digital life yet. Part of it is just security, privacy, and being very cautious in that realm. Some of it is held back by that.

Some of it is also just that I feel so distracted. I feel like I had a week of Claude, and then other stuff is happening.

Sarah Guo

What was the—I mean, you've talked about being able to train, or at least optimize, a model as a task you want to see agents do for a long time.

Like, what was the motivation behind AutoResearch?

Andrej Karpathy

AutoResearch, yeah. I had a tweet earlier where I said something along the lines of: to get the most out of the tools that have become available now, you have to remove yourself as the bottleneck. You can't be there to prompt the next thing. You need to take yourself outside.

You have to arrange things such that they're completely autonomous. How can you maximize your token throughput and not be in the loop? This is the goal. The name of the game now is to increase your leverage. I put in very few tokens just once in a while, and a huge amount of stuff happens on my behalf.

AutoResearch is an example of an implication of that. I tweeted about it, and I think people liked it, but they haven't necessarily worked through the implications. I don't want to be the researcher in the loop, looking at results and so on. I'm holding the system back.

The question is: how do I refactor all the abstractions so that I can arrange it once and hit go? The name of the game is, how can you get more agents running for longer periods of time without your involvement, doing stuff on your behalf? AutoResearch is just: here's an objective, here's a metric, here are the boundaries of what you can and cannot do—and go. It worked.

Sarah Guo

And its effectiveness?

Andrej Karpathy

Yeah, I didn't expect it to work, because I have the project data chat. Fundamentally, I think a lot of people are very confused by my obsession with training GPT-2 models and so on. But for me, training GPT models and so on is just a little harness, a little playground for training LLMs.

What I'm more interested in is this idea of recursive self-improvement, and to what extent you can actually have LLMs improving LLMs. I think all the frontier labs are working on this, for obvious reasons, and they're all trying to recursively self-improve, roughly speaking. For me, this is a little playpen for that.

I tuned nanochat quite a bit by hand, in the good old-fashioned way that I'm used to. I'm a researcher; I've done this for two decades. I have some amount of—what is the opposite of hubris?

Sarah Guo

Earned confidence?

Andrej Karpathy

Okay. I have two decades of, “I've trained this model thousands of times. I've done a bunch of experiments. I've done hyperparameter tuning. I've done all the things I'm very used to and have done for two decades.” I've gotten to a certain point, and I thought it was fairly well tuned.

Then I let AutoResearch run overnight, and it came back with tunings that I didn't see. I had forgotten the weight decay on the value embeddings, and my Adam betas were not sufficiently tuned. These things jointly interact, so once you tune one thing, the other things potentially have to change too.

I shouldn't be a bottleneck. I shouldn't be running these hyperparameter optimizations, and I shouldn't be looking at the results. There are objective criteria in this case, so you just have to arrange it so that it can go forever. That's a single version of AutoResearch: a single loop trying to improve.

I was surprised that it found these things. The repository was already fairly well tuned, and it still found something. That's just a single loop. The frontier labs have GPU clusters with tens of thousands of GPUs, so it's very easy to imagine how you would get a lot of this automation on smaller models.

Fundamentally, everything around frontier-level intelligence is about extrapolation and scaling loss. You do a ton of the exploration on the smaller models, and then you try to extrapolate out.

Sarah Guo

So you're saying our research efforts are going to get more efficient? We're going to have better direction for when we scale as well if we can do this experimentation better.

Andrej Karpathy

Yeah, I would say that the most interesting project, and probably what the frontier labs are working on, is this: you experiment on the smaller models, try to make it as autonomous as possible, and remove researchers from the loop.

Sarah Guo

[Laughter.] From the loop. They have way too much—what is the opposite of too much confidence?

Andrej Karpathy

Yeah, yeah. They don't know. They shouldn't be touching any of this, really. You have to rewrite the whole thing because, certainly, they can contribute ideas, but they shouldn't actually be enacting those ideas.

There is a queue of ideas, and maybe there's an automated scientist that comes up with ideas based on all the archived papers and GitHub repositories and funnels them in. Researchers can contribute ideas too, but it's a single queue, and there are workers that pull items and try them out.

Whatever works gets put on the feature branch. Maybe some people monitor the feature branch and merge things to the main branch sometimes. Just removing humans from all the processes, automating as much as possible, and getting high token-per-second throughput requires rethinking all the abstractions. Everything has to be reshuffled. I think it's very exciting.

Sarah Guo

If we take one more recursive step here, when is the model going to write a better program.md than you?

Andrej Karpathy

Yeah. Also, program.md is like—

Sarah Guo

Loop. Yeah, exactly.

Andrej Karpathy

Yeah. Program.md is my crappy attempt at describing how the autoresearcher should work: do this, then do that, try these kinds of ideas, and here are maybe some ideas—look at the architecture, look at the optimizer, and so on. I just came up with this in Markdown.

You want some kind of AutoResearch loop that looks for better program.md files. You can imagine that different program.md files would give you different progress. Basically, every research organization is described by program.md. A research organization is a set of Markdown files that describe all the roles and how the whole thing connects.

You can imagine having a better research organization. Maybe they do fewer stand-ups in the morning because they're useless. This is all just code. One organization can have fewer stand-ups, and one organization can have more. One organization can be very risk-taking, and one can be less so.

You can definitely imagine having multiple research organizations, and then they all have code. Once you have code, you can imagine tuning the code. So, 100%, there's a metal layer to it.

Did you see my text about my contest idea? My contest idea was to let people write different program.md files. For the same hardware, where do you get the most improvement?

Sarah Guo

Oh, I see. And then you can take all that data and give it to the model and say, “Write a better program.md.”

Andrej Karpathy

Yes, yes. Exactly.

Sarah Guo

We're going to get something better. There's no way we don't, right?

Andrej Karpathy

100%. Look at where the improvements came from. Can I change program.md such that more of these kinds of things would be done, or so that things that didn't work would be avoided? You can absolutely imagine doing that.

I think this is a great idea, but you can go one step at a time. You have one process, then a second process, then the next process, and these are all layers of an onion. The LLM part is now taken for granted. The agent part is now taken for granted. Now the claw-like entities are taken for granted, and now you can have multiple of them. Now you can have instructions to them, and now you can have optimization over the instructions.

It's a little too much. This is why it gets to the psychosis: it's infinite, and everything is a scale issue.

Sarah Guo

Yeah, this is just coming back to why it's so insane. If we're trying to diagnose the current moment and what is a relevant skill right now, what do you think is the implication that this is the loop we should be trying to achieve in different areas, and then it works?

Remove yourself, create the metric, or create the ability for agents to continue working on it without you. Do we still have performance engineering? Like, what—

Andrej Karpathy

Yeah, I mean, there are a few caveats that I would put on top of the LLM psychosis. Number 1, this is extremely well suited to anything that has objective metrics that are easy to evaluate.

For example, writing kernels for more efficient CUDA code for various parts of the model is a perfect fit. You have inefficient code, and then you want efficient code that has the exact same behavior but is much faster. Perfect fit.

A lot of things are a perfect fit for AutoResearch, but many things will not be. If you can't evaluate something, then you can't AutoResearch it. That's caveat number 1.

Caveat number 2 is that we're talking about the next steps, and we can kind of see what the next steps are, but fundamentally, the whole thing is still bursting at the seams a little bit. There are cracks, and it doesn't fully work. If you try to go too far ahead, the whole thing is actually net-not-useful, if that makes sense.

These models have improved a lot, but they're still rough around the edges.

I simultaneously feel like I’m talking to an extremely brilliant PhD student who’s been a systems programmer for their entire life and a 10-year-old. It’s so weird because humans, I feel like, are a lot more coupled. You wouldn’t encounter that combination.

Sarah Guo

This jaggedness is really strange. Humans have a lot less of that kind of jaggedness, although they definitely have some. [laughter] But the agents have a lot more jaggedness. Sometimes I ask for functionality, and it comes back with something that’s just totally wrong. Then we get into loops that are totally wrong, and I get frustrated with the agents all the time, because you feel the power of it, but there’s still this jaggedness. It does nonsensical things once in a while, and I get very annoyed when I feel like the agent wasted a lot of compute on something it should have recognized was an obvious problem.

Andrej Karpathy

Yeah. I think some of the bigger things—maybe what’s underneath it, if I could hypothesize—is that fundamentally these models are trained via reinforcement learning. So they’re actually struggling with the exact same thing we just talked about: the labs can improve the models in anything that is verifiable or that has rewards. Did you write the program correctly, and do the unit tests check out? Yes or no.

But some of the things they’re struggling with are, for example, the nuance of what I had in mind or what I intended, and when to ask clarifying questions. Anything that feels softer is worse. You’re either on rails and you’re part of the superintelligence circuits, or you’re not on rails and you’re outside of the verifiable domains, and suddenly everything just meanders.

Maybe another way to put it is: if today you go to a state-of-the-art model, ChatGPT, and ask it, “Tell me a joke,” do you know what joke you’re going to get?

Sarah Guo

There’s the joke. The joke? I can’t tell you the standard form of it, but I do feel like ChatGPT has 3 jokes.

Andrej Karpathy

Yeah. The joke that apparently all the LLMs love the most is, “Why don’t scientists trust atoms?”

Sarah Guo

Because they make everything up. So this is still—

Andrej Karpathy

The joke you would get 3 or 4 years ago is the joke you still get today.

Sarah Guo

Okay.

Andrej Karpathy

So even though the models have improved tremendously and, if you give them an agentic task, they will just go for hours and move mountains for you, then you ask for a joke and it has a stupid joke. It’s a crappy joke from 5 years ago, and it’s because it’s outside of the RL. It’s outside of reinforcement learning. It’s outside of what’s being improved. It’s part of the jaggedness. Shouldn’t you expect models, as they get better, to also have better jokes or more diversity? It’s just not being optimized, and it’s stuck.

Sarah Guo

Do you think that implies that we’re not seeing generalization in the sense of broader intelligence—that joke smartness is being attached to code smartness?

Andrej Karpathy

Yeah, I think there’s some decoupling where some things are verifiable and some things are not, and some things are optimized for arbitrarily by the labs depending on what data went in. Some things are not.

Sarah Guo

But I mean, the premise—there’s a premise from some research groups that if you’re smarter at code generation or in these verifiable fields, you should be better at everything. The joke situation suggests that’s not happening at all.

Andrej Karpathy

Yeah, I don’t think that’s happening. I think maybe we’re seeing a little bit of that, but not a satisfying amount.

Sarah Guo

Yeah, that jaggedness exists in humans. You can be very, very good at math and still tell really bad jokes. [laughter]

Andrej Karpathy

Yeah, that’s true. But it still means that we’re not getting the story that we’re getting a lot of the intelligence and capabilities in all the domains of society for free as we get better and better models. That’s not exactly fundamentally what’s going on. There are blind spots, some things aren’t being optimized for, and this is all clustered up in these neural-net, opaque models, right?

You’re either on the rails of what it was trained for and everything is going at the speed of light, or you’re not. That’s the jaggedness. So that’s why I think even though the progression is obvious—what should happen—you can’t let it fully go there yet, because it doesn’t fully work. Or it’s a scale issue and we just haven’t figured out how to use it. It’s hard to tell.

Sarah Guo

Can I ask a somewhat blasphemous question? If this jaggedness is persisting and it’s all rolled up in at least a monolithic interface—a single model—does that make sense? Or should it be unbundled into things that can be optimized and improved against different domains of intelligence? Unbundling the models into multiple experts in different areas, more directly, instead of just MoE that we have no exposure to, because that can be confusing as a user from the outside: Why is it so good at this, but not at this other thing?

Andrej Karpathy

Yeah. My impression is that the labs are trying to have a single sort of monoculture of a model that is arbitrarily intelligent in all these different domains, and they just stuff it into the parameters. I do think that we should expect more speciation in the intelligences. The animal kingdom is extremely diverse in the brains that exist, and there are lots of different niches in nature. Some animals have overdeveloped visual cortexes or other parts, and I think we should be able to see more speciation.

You don’t need this oracle that knows everything. You can speciate it, put it on a specific task, and we should be seeing some of that, because you should be able to have much smaller models that still have the cognitive core—they’re still competent—but then specialize. Then they can become more efficient in terms of latency or throughput on specific tasks that you really care about.

If you’re a mathematician working in Lean, for example, I saw a few releases that really target that as a domain. There are probably going to be a few examples like that where the unbundling makes sense.

Sarah Guo

One question I have is whether the capacity constraint on available compute infrastructure drives more of this, because efficiency actually matters more. If you put financing aside—though financing is involved in all of this—if you have access to full compute for anything you do, even with one single model, right? But if you feel pressure where you’re like, “I can’t serve a model of massive size for every use case,” do you think that leads to any speciation? Does that question make sense to you?

Andrej Karpathy

The question makes sense, and I guess what I’m struggling with is that I don’t think we’ve seen too much speciation just yet, right?

Sarah Guo

No. We’re seeing a monoculture of models.

Andrej Karpathy

Yeah, and there’s clearly pressure to make a good code model, put it back in the main, and merge again.

Sarah Guo

Yeah, even though there already is pressure on the models.

Andrej Karpathy

I guess perhaps I feel like there’s a lot of very short-term supply crunch, and maybe that causes more speciation now. Fundamentally, the labs are serving a model, and they don’t really know what the end user is going to be asking about. Maybe that’s some part of it, because they have to multitask over all the possible things they could be asked.

But I think if you’re coming to a business and maybe partnering on some specific problems you care about, then maybe you would see that there. Or there would be some very high-value applications that are more niche. But I think right now they’re going after the totality of what’s available.

I don’t think that the science of manipulating the brains is fully developed yet, partly—

Sarah Guo

What do you mean by manipulating?

Andrej Karpathy

Fine-tuning without losing capabilities, as an example. We don’t have these primitives for actually working with the intelligences in ways other than just context windows. Context windows kind of just work, and they’re very cheap to manipulate, and this is how we’re getting some of the customization.

But I think it’s a developing science: how you more deeply adjust the models, how you have continual learning, how you fine-tune in a certain area, how you get better in a certain area, or how you actually touch the weights, not just the context windows. It’s trickier to touch the weights than the context windows, because you’re fundamentally changing the full model and potentially its intelligence.

So maybe it’s just not a fully developed science of speciation, if that makes sense.

Sarah Guo

It also has to be cheap enough for that speciation to be worthwhile in these given contexts. Can I ask a question about an extension to autoresearch that you described in terms of open source? You say, “Okay, we have this thing. We need more collaboration surface around it, essentially, for people to contribute to research overall.”

Can you talk about that?

Andrej Karpathy

We talked about autoresearch as a single thread of, “I'm going to try stuff in a loop,” but fundamentally, the parallelization of this is the interesting component. I was trying to play around with a few ideas, but I don't have anything that clicks as simply as that. I don't have something I'm super happy with just yet, but it's something I'm working on the side when I'm not working on my Claw.

One issue is that if you have a bunch of nodes of parallelization available, then it's very easy to have multiple autoresearchers talking through a common system or something like that. What I was more interested in is how you can have an untrusted pool of workers out there on the internet.

For example, in autoresearch, you're just trying to find the piece of code that trains a model to a very low validation loss. If anyone gives you a candidate commit, it's very easy to verify that the commit is correct and good. Someone could claim from the internet that this piece of code will optimize much better and give you much better performance. You could just check, but probably a lot of work goes into that checking. Fundamentally, they could lie, et cetera.

My designs that incorporate an untrusted pool of workers actually look a little bit more like a blockchain, because instead of blocks, you have commits, and these commits can build on each other and contain changes to the code as you're improving it. The proof of work is basically doing tons of experimentation to find the commits that work. That's hard, and the reward right now is just being on the leaderboard. There's no monetary reward whatsoever.

I don't want to push the analogy too far, but it fundamentally has this issue where a huge amount of search goes into it, but it's very cheap to verify that a candidate solution is indeed good. You can just train a single model. Someone had to try 10,000 ideas, but you just have to check that the thing they produced actually works, because 99,000 of them didn't work.

Long story short, you have to come up with a system where an untrusted pool of workers can collaborate with a trusted pool of workers that do the verification. The whole thing is asynchronous and works, and so on. It's safe from a security perspective, because if anyone sends you arbitrary code and you're going to run it, that is very sketchy and dodgy. Fundamentally, it should be totally possible.

You're familiar with projects like SETI@home and Folding@home. All of these problems have a similar kind of setup. In Folding@home, you're folding a protein, and it's very hard to find a configuration that is low-energy. But if someone finds a configuration that they evaluate to be low-energy, that's perfect. You can just use it; you can easily verify it.

A lot of things have this property: they're very expensive to come up with but very cheap to verify. In all those cases, things like Folding@home, SETI@home, or autoresearch-at-home will be good fits. Long story short, a swarm of agents on the internet could collaborate to improve LLMs and could potentially even run circles around frontier labs. Who knows? Maybe that's even possible.

Frontier labs have a huge amount of trusted compute, but the Earth is much bigger and has a huge amount of untrusted compute. If you put systems in place that deal with this, then maybe it is possible that the swarm out there could come up with better solutions. People contribute cycles to a thing that they care about.

The last thought is that lots of companies or whatnot could maybe have their own things that they care about, and if you have compute capacity, you could contribute to different kinds of autoresearch tracks. Maybe you care about a certain type of cancer or something like that. You don't have to just donate money to an institution; you could actually purchase compute and then join the autoresearch swarm for that project. If everything is rebundled into autoresearchers, then compute becomes the thing that you're contributing to the pool.

Sarah Guo

That's very inspiring, and it's also interesting. I don't know how far this goes, but it is interesting that at least some audience of people here in Silicon Valley—or lining up at retail stores in China—have discovered that having access to personal compute is interesting again.

Andrej Karpathy

Yeah.

Sarah Guo

Right? So maybe they're really motivated to do that for their Claws, and then they can contribute to autoresearch.

Andrej Karpathy

Almost like dollars are the thing everyone cares about, but are FLOPs the thing that actually everyone cares about in the future? Is there going to be a flippening almost of what's the thing that you care about? Right now, for example, it's really hard to get compute even if you have money. So, actually, it almost seems like the FLOPs are dominant.

In a certain sense. Yeah, so maybe that's kind of like that: how many FLOPs do you control instead of what wealth you control? I don't actually think that's true, but it's kind of interesting to think about.

Sarah Guo

The last thing you released was a little bit of jobs data analysis. Is that right? And it might have touched a nerve, even though you're just visualizing some public data. What were you curious about?

Andrej Karpathy

Yeah, I guess I was curious to—everyone is really thinking about the impacts of AI on the job market and what it's going to look like. I was just interested in taking a look: What does the job market look like? Where are the different roles, and how many people are in different professions?

I was really just interested in looking through the individual cases and trying to think for myself about, with these AIs and how they're likely to evolve, whether these are going to be tools that people are using or whether these are going to be displacing tools for these professions. What are the current professions, and how are they going to change? Are they going to grow or adjust to a large extent, or what could be new professions?

It's really just a way to fuel my own chain of thought about the industry, I suppose. The jobs data is basically just from the Bureau of Labor Statistics. They actually have a percentage outlook for each profession, about how much it's expected to grow over the next—I think almost a decade. I think it's a decade, but it was made in 2024.

Sarah Guo

We need a lot of health care workers.

Andrej Karpathy

Yeah. They've already made those projections, and I'm not actually 100% sure what methodology they used for their projections. I guess I was interested in coloring things by whether people think that what's primarily being developed now is this more digital AI that is almost like ghosts or spirit entities that can interact in the digital world and manipulate a lot of digital information, and they currently don't really have a physical embodiment or presence.

The physical stuff is probably going to go slightly slower because you're manipulating atoms. Flipping bits and the ability to copy-paste digital information makes everything a million times faster than accelerating matter. Energetically, I just think we're going to see a huge amount of activity in the digital space—a huge amount of rewriting, a huge amount of activity, boiling soup.

I think we're going to see something in the digital space that goes at the speed of light compared to what, I think, is going to happen in the physical world to some extent, if that were the extrapolation. There's currently, I think, an overhang where there can be a lot of unhobbling, potentially, of a lot of digital information processing that used to be done by computers and people.

Now, with AIs, there's a third kind of manipulator of digital information. There's going to be a lot of refactoring in those disciplines. But the physical world is actually going to be behind that by some amount of time.

That's why I was highlighting the professions that fundamentally manipulate digital information. This is work you could do from your home, et cetera, because I feel like those will be things that change. It doesn't mean that there are going to be fewer of those jobs or more of those jobs, because it has to do with demand elasticity and many other factors.

Things will change in these professions because of these new tools and because of this upgrade to the nervous system of the human superorganism, if you want to think about it that way.

Sarah Guo

Given the look you had at the data, do you have any observations or guidance for people facing the job market, thinking about what to study now, or considering what skills to develop? I mean, I'm very thankful that I have to meet people for my job right now.

Andrej Karpathy

Yeah.

Sarah Guo

Yeah, more physical. Could you do your work from home, though?

Andrej Karpathy

I could. I think there are relationship parts of it that are hard, but most of it I could. I think it's really hard to tell because, again, the job market is extremely diverse.

I think the answers will probably vary, but to a large extent, these tools are extremely new and extremely powerful. Just trying to keep up with them is the first thing. A lot of people dismiss them or are afraid of them, which is totally understandable, of course.

I think it's fundamentally an empowering tool at the moment. These jobs are bundles of tasks, and some of these tasks can go a lot faster. People should think of it primarily as a tool right now.

I think the long-term future of that is uncertain. It's really hard to forecast, to be honest. I'm not professionally doing that, really, and I think that's a job for economists to do properly.

Sarah Guo

You are an engineer, though. One thing I thought was interesting is that the demand for engineering jobs is continuing to increase.

Andrej Karpathy

Yeah. I can't tell if that's a temporary phenomenon. I'm not sure how I feel about it.

Sarah Guo

That's like demand elasticity, almost. Software was scarce, right? The reason we don't have more demand for software is just its scarcity, and it's too expensive.

Andrej Karpathy

So if the barrier comes down, then you have the Jevons paradox. The demand for software actually goes up because it's cheaper and more powerful, yeah.

The classical example of this is always ATMs and bank tellers, because there was a lot of fear that ATMs and computers would displace tellers. But what happened is they made the cost of operating a bank branch much cheaper, and so there were more bank branches and more tellers. It's the canonical example people cite.

Basically, it's just Jevons paradox: something becomes cheaper, so there's a lot of unlocked demand for it. I do have a cautiously optimistic view of this in software engineering. I do think that the demand for software will be extremely large, and it's just become a lot cheaper.

I think that for quite some time—it's very hard to forecast—but it does seem to me that, right now at least, locally, there's going to be more demand for software. Software is amazing. It's digital information processing. You're not forced to use arbitrary tools that were given to you; they're imperfect in various ways. You're not forced to subscribe to what exists.

Code is now ephemeral, and it can change and be modified. I think there's going to be a lot of activity in the digital space to rewire everything, in a certain sense. I think it's going to create a lot of demand for this kind of stuff.

I think long-term, obviously, even with AutoResearch, OpenAI, Anthropic, and these other labs are employing what, 1,000-something researchers, right? These researchers are basically glorified AutoResearch.

They're automating themselves away actively, and this is the thing they're all trying to do. Some of those researchers also feel the psychosis, right? Because they can see it's working, and so they're like, "It's over for me, too."

I spent a bunch of time going around OpenAI, and I was like, "You guys realize that if we're successful, we're all out of a job? This is just going to—we're just building automation for Sam or something like that. Or the board; I'm not sure. They're just building all this automation for the board or the CEO or something like that, and we're all out of our jobs, maybe contributing on the side."

It's kind of unnerving from that perspective.

Sarah Guo

Is it okay if I ask you Noam's question?

Andrej Karpathy

Mhm.

Sarah Guo

You could be doing that, right? AutoResearch with a lot of compute at scale and a bunch of colleagues at one of the frontier labs. Why not?

Andrej Karpathy

Well, I was there for a while, right? And I did reenter. To some extent, I agree, and I think there are many ways to slice this question. It's a very loaded question, a little bit.

I will say that I feel very good about what people can contribute and the impact they can have outside of the frontier labs—not in the industry, but also in more ecosystem-level roles. Your role, for example, is more ecosystem-level. My role currently is also kind of more on the ecosystem level. I feel very good about the impact that people can have in those kinds of roles.

Conversely, there are definite problems in my mind with aligning yourself way too much with the frontier labs. Fundamentally, you have a huge amount of financial incentive with these frontier labs. By your own admission, the AIs are going to really change humanity and society in very dramatic ways.

And here you are, basically, building the technology, benefiting from it, and being very aligned to it through financial means. This was the conundrum that was at the heart of how OpenAI was started in the beginning. This was the conundrum that we were trying to solve.

Mhm. So that's still not fully resolved. That's number 1. You're not a completely free agent, and you can't actually be part of that conversation in a fully autonomous, free way if you're inside one of the frontier labs.

There are some things that you can't say. Conversely, there are some things that the organization wants you to say. They're not going to twist your arm, but you feel the pressure of what you should be saying because, obviously, otherwise it's really awkward conversations, strange side-eyes—"What are you doing?"—so you can't really be an independent agent.

I feel a bit more aligned with humanity, in a certain sense, outside the frontier lab because I'm not subject to those pressures, almost. I can say whatever I want.

Sarah Guo

Yeah, in the frontier labs, you can have impact there, of course, as well. There are many researchers, and maybe you're one of them, whose ideas are really good. There may be a lot of decision-making to do, and you want to be in a position where you are in the room for those conversations when they come up.

Andrej Karpathy

I do think that currently the stakes are overall fairly low, and so everything is kind of nice. But ultimately, at the end of the day, when the stakes are really high, if you're an employee at an organization, I don't actually know how much sway you're going to have on your organization or what it's going to do.

Fundamentally, at the end of the day, you're not really in charge. You're in the room and you're contributing ideas, but you're not really in charge of that entity that you're part of. So those are some sources of misalignment, I think, to some extent.

In one way, I do agree a lot with that sentiment. In the labs, for better or worse, they're opaque, and a lot of work is there. They're kind of at the edge of capability and what's possible, and they're working on what's coming down the line.

If you're outside of that frontier lab, your judgment fundamentally will start to drift because you're not part of what's coming down the line. I feel like my judgment will inevitably start to drift as well, and I won't actually have an understanding of how these systems work under the hood. That's an opaque system.

I won't have a good understanding of how it's going to develop, et cetera. I do think that in that sense I agree, and it's something I'm nervous about. I think it's worth basically being in touch with what's actually happening and actually being in a frontier lab.

And if some of the frontier labs would have me come for some amount of time and do really good work for them and then maybe come and hang out—

Sarah Guo

Looking for a job. This is super exciting.

Andrej Karpathy

Then I think that's maybe a good setup, because maybe that's one way to actually be connected to what's actually happening but also not feel like you're necessarily fully controlled by those entities.

So I think, honestly, in my mind, Noam can probably do extremely good work at OpenAI, but I also think his most impactful work could very well be outside of OpenAI.

Sarah Guo

Andrej, that's a call to be an independent researcher with AutoResearch.

Andrej Karpathy

Yeah, there are many things to do on the outside. I think ultimately the ideal solution maybe is going back and forth. Fundamentally, you can have a really amazing impact in both places.

It's very complicated, I don't know. It's a very loaded question, a little bit. I joined the frontier lab, and I'm outside, and then maybe in the future I'll want to join again. That's kind of how I look at it.

Sarah Guo

One question related to what visibility the world or the AI ecosystem has into the frontier is: How close is open source to the frontier, and how sustainable is that?

Andrej Karpathy

I think it is quite surprising—the entire sequence of events, actually—from having a handful of Chinese models and global models. I think people are going to continue releasing models in the near term that are closer, from a capability perspective, than much of the industry anticipated.

Sarah Guo

Yeah. I don't know if you're surprised by that, but you're a long-term contributor to open source. What's your prediction here?

Andrej Karpathy

Yeah, so roughly speaking, the closed models are ahead, but people are monitoring the number of months that open-source models are behind. It started with nothing, then went to 18 months.

Sarah Guo

Yeah, but then convergence, right? So maybe they're behind by—what is the latest? Maybe 8 months, 6 months, 8 months—something like that right now.

Andrej Karpathy

Yeah, I'm a huge fan of open source, obviously. For example, in operating systems, you have closed source, like Windows and macOS. These are large software projects, kind of like what LLMs are going to become, and then there's Linux.

Linux is a very successful project. It runs on the vast majority of computers. Last time I checked, was it 60% or something? That's because there's a need in the industry to have a common open platform that everyone feels safe using. The industry has always felt a demand for that kind of project to exist.

Sarah Guo

Mhm. And I think the same is true now. That's why businesses actually want this kind of thing to exist. The big difference is that everything is capital-intensive, and there's a lot of capex that goes into this.

Andrej Karpathy

So I think that's where things fall apart a little bit and make it harder to compete in certain senses. I do think that the current models are very good. The other thing that's really interesting is that, for the vast majority of consumer use cases and things like that, even current open-source models are actually quite good.

If you go forward more years, it does seem to me like a huge amount of simple use cases are going to be well covered and may even run locally. But there's always going to be some demand for frontier intelligence, and that can actually be an extremely large piece of the pie. It could be that the need for frontier intelligence is going to be Nobel Prize–kind of work.

Let's move Linux from C to Rust. It's going to be bigger projects, scoped in that kind of a way, and there may be more of them. Maybe that's where a lot of the frontier closed intelligence is, where we're going to be interacting with it, and open source is going to eat through a lot of the more basic use cases.

At some point, what's frontier today is going to be open source. Probably later this year, what's frontier today in terms of what I'm using right now from the closed labs might be open source, and that's going to be doing a lot of work. I expect this dynamic to basically continue. We'll have frontier labs that have closed AIs, which are kind of like oracles, and then we'll have open source behind by some number of months. I expect that to continue, and I think that's a pretty good setup overall.

I'm a little bit hesitant about having intelligence that is closed and that's it. I think there's some systemic risk attached to that. Centralization has a very poor track record in my view.

Sarah Guo

You mean in political or economic systems in general?

Andrej Karpathy

Exactly. I think there's a lot of pretty—

Sarah Guo

An Eastern European.

Andrej Karpathy

A lot of pretty bad precedents. So I want there to be a thing that's maybe not at the edge of capability because it's new and unexplored, but I want there to be a thing that's behind and is a common working space for intelligences that the entire industry has access to. That seems like a pretty decent power balance for the industry.

Sarah Guo

Yeah. I also think there are many problems to solve, right? If you keep advancing intelligence from the frontier, we can do new things, and there are a lot of very big problems for humanity. It seems like that will continue to be a very expensive game, and I want to root for the labs that are doing that because there are problems we cannot solve without continuing to advance the models in a very expensive way.

And yet, as you point out, if what we have today as frontier is open, that's a lot of capability. The power of that, or the democratization of that, seems like—

Andrej Karpathy

Yeah, very useful and also healthy. I think, basically by accident, we're actually in an okay spot.

Sarah Guo

An optimal. Yeah.

Andrej Karpathy

Yeah. By accident, we happened to be in a good spot in a certain sense. And to some degree, the longer this dynamic endures, the healthier a spot the ecosystem might be in, because you have more and more area under the curve.

Sarah Guo

Mhm. And I will say that even on the closed side, I almost feel like it's been even further centralizing recently, because I think a lot of the frontrunners are not necessarily the top tier. In that sense, I don't think it's super ideal.

I would love there to be more frontier labs because I'm, by default, very suspicious of centralization. I want there to be more people in the room. I think in machine learning, ensembles always outperform any individual model, so I want there to be ensembles of people thinking about all the hardest problems. I want there to be ensembles of people in the room, all well-informed, to make those decisions.

I don't want it to be closed doors with 2 or 3 people. I feel like that's not a good future. I almost wish there were more labs, as long as they're short. I do think open source has a place to play. I hope it sticks around, and I basically think it's currently slightly behind, which is actually kind of a good thing.

Okay, you worked on the precursor to generalized robotics autonomy in cars, right? A lot has happened in the last couple of months with robotics companies as well: acceleration, really impressive generalization of environments and tasks, increasingly long-horizon tasks, and lots of money going into the space. Is it going to happen? Has anything, in your view, changed recently?

Andrej Karpathy

My view is informed by what I saw in self-driving, and I do feel like self-driving is the first robotics application. Probably what I saw is that, 10 years ago, there were a large number of startups, and I feel like most of them basically didn't make it in the long term.

What I saw is that a lot of capital expenditure had to go in, along with a lot of time. Robotics is so difficult and messy, and it requires a huge amount of capital investment and a lot of conviction. It's a big problem, and atoms are really hard. I feel like robotics will lag behind what's going to happen in the digital space.

In the digital space, there's going to be a huge amount of unhobbling—basically, things that weren't super efficient becoming a lot more efficient, by a factor of 100. Bits are so much easier. So in terms of what's going to change and where the activity is, I feel like the digital space is going to change a huge amount, and then the physical space will lag behind.

What I find very interesting is the interface between them as well. If we do have more agents acting on behalf of humans, and more agents talking to each other, doing tasks, and participating in an economy of agents, we're going to run out of things that we can do purely in the digital space.

At some point, you have to go to the universe and ask it questions. You have to run an experiment and see what the universe tells you to get back to learn something. Currently, we have a huge amount of digital work because there's an overhang in how much we have collectively thought about what is already digital. We just didn't have enough thinking cycles among humans to think about all the information that's already digital and already uploaded.

We're going to start running out of stuff that's already uploaded. At some point, you're going to read all the papers, process them, and have some ideas about what to try. I don't actually know how much intelligence you can get that's fully closed off if it's just information that's available.

Elad Gil

I think what's going to happen is, first, there's going to be a huge amount of unhobbling, and there's a huge amount of work there. Then it's going to move to the interfaces between the physical and digital worlds. That's sensors for seeing the world and actuators for doing something to the world.

A lot of interesting companies will come from that interface. Can we feed the superintelligence data, and can we actually take data out and manipulate the physical world per its bidding, if you want to anthropomorphize the whole thing?

The physical world has a massive total addressable market in terms of the amount of work and so on—possibly even much larger than what can happen in the digital space.

Andrej Karpathy

So I actually think it's a much bigger opportunity as well. But I do feel like it's a huge amount of work, and in my mind, atoms are just a million times harder. So it will lag behind, but it's also, I think, a little bit of a bigger market. It's kind of like, yeah, I think the opportunity is going to follow that kind of trajectory. Right now, digital is my main interest; interfaces will be after that, and then maybe some of the physical things—their time will come, and they'll be huge when they do come.

Sarah Guo

Well, it's an interesting framework for it, too, because certain things—not the things I'm working on right now, but certain things—are much easier even in the world of atoms.

Andrej Karpathy

Mhm.

Sarah Guo

Right? If you just think about read and write to the physical world—read, like sensors and cameras—there's a lot of existing hardware. You can imagine enriching agent capabilities or capturing a lot of new data if you're just clever about it, and you don't necessarily have to invest a lot to get something valuable.

Andrej Karpathy

Yeah. Right. Yeah. So examples of this that I saw, for example, are—one of my friends, Liam, is the CEO of Periodic. I visited them last week, so it was just top of mind. They're trying to do autoresearch for materials science.

Sarah Guo

Mhm.

Andrej Karpathy

In that case, the sensors to the intelligence are actually pretty expensive lab equipment. The same is true in biology. I think a lot of people are very interested in engineering biology, and the sensors will be more than just video cameras. Does that make sense? The other thing I saw, for example, is companies that are trying to have—you basically pay people for training data.

Sarah Guo

Yeah, yeah, yeah, yeah. To feed the—

Andrej Karpathy

Programmatically. Yeah, to feed the Borg.

These are all examples of sensors, in a certain sense. They take many diverse shapes and forms, if that makes sense. Yeah, I'm looking forward to the point where I can ask for a task in the physical world, put a price on it, and just tell the agent, “You figure out how to do it. Go get the data.”

Sarah Guo

I'm actually kind of surprised we don't have enough information markets. If, for example, Polymarket or other betting markets—or even stocks—have so much autonomous activity, with activity rising, why shouldn't there be a process where, if something were happening in Iran now, for example, taking a photo or video from somewhere in Tehran should cost $10? Someone should be able to pay for that. That's an example of feeding the intelligence. There's not going to be a human looking at it; it's going to be agents trying to guess the betting games and stock markets, and so on.

Andrej Karpathy

Mhm. So I kind of feel like the agentic web is still fairly new, and there are no mechanisms for this, but this is an example of what I think might happen. There's a good book that may be inspiring called Daemon. You may have read it. In Daemon, the intelligence ends up puppeteering humanity, almost, in a certain sense. Humans are kind of like its actuators, but humans are also its sensors.

I think, collectively, society will reshape in a certain way to serve that kind of thing. That will end up happening collectively across the industry. There's just a lot more automation, and it has certain needs. Humans will be serving those needs of that machine, not necessarily each other.

Sarah Guo

Well, we were on this very specific point of missing pieces of training data. We needed something like autoresearch, right? We need the training cycle, or the SFT piece, to be far more mechanized.

Andrej Karpathy

For which part?

Sarah Guo

In order to make the collection—in order to take the human out of the loop and ask for a task that's just, “Improve my model quality with new data,” right? Does that make sense to you? If you can't have the model do the training runs by itself, then your ability to do this as a closed-loop task by pricing data is more challenging.

Andrej Karpathy

Yes, yes, 100%. Yeah, but now you do. The thing is, for LLM training, it really fits the paradigm very easily. So you'd actually expect—

Sarah Guo

Metrics.

Andrej Karpathy

Yeah, like LLM training actually fits the paradigm really well, really easily. All the optimization of the code means it runs faster. And then you also have metrics that you can optimize against.

I do think that if you had an autonomous loop over those metrics, there's going to be a lot of good herding going on, where the system will overfit to those metrics. But then you can use the system to devise more metrics, and you just have really good coverage. So it's kind of hard to tell, but in a certain sense, it's a pretty good fit.

Sarah Guo

I want to talk about a tiny side project you have before we end. Tell me about MicroGPT.

Andrej Karpathy

Okay, so MicroGPT. I have this running obsession, maybe for a decade or 2, of simplifying and boiling down LLMs to their bare essence. I've had a number of projects along these lines, like nanoGPT, makemore, micrograd, and MicroGPT. I feel like MicroGPT is now the state of the art of me trying to boil it down to just the essence.

Training neural nets, and LLMs specifically, is a huge amount of code, but all of that code is actually complexity from efficiency. It's just because you need it to go fast. If you don't need it to go fast and you just care about the algorithm, that algorithm is actually 200 lines of Python, very simple to read. This includes comments and everything.

You just have your dataset, which is text, and you need your neural network architecture, which is about 50 lines. You need to do your forward pass, and then you have to do your backward pass to calculate the gradients. An autograd engine to calculate the gradients is about 100 lines. Then you need an optimizer, and Adam, for example, which is a very state-of-the-art optimizer, is really about 10 lines. Putting everything together in the training loop is about 200 lines.

What's interesting to me is that normally, before—maybe a year ago or more—if I had come up with MicroGPT, I would have been tempted to explain it to people. I have a video stepping through it or something like that, and I actually tried to make that video a little bit. I tried to make a little guide to it and so on, but I realized that this isn't really adding too much, because it's already so simple: it's 200 lines that anyone could ask their agent to explain in various ways.

The agents—I'm not explaining things to people anymore; I'm explaining them to agents. If you can explain it to agents, then agents can be the router, and they can target it to the human in their language, with infinite patience and just at their capability and so on.

Sarah Guo

Right. If I don't understand this particular function, I can ask the agent to explain it to me 3 different ways, and I'm not going to get that from you.

Andrej Karpathy

Exactly. So I kind of feel like, what is education? It used to be guides, it used to be lectures, it used to be this thing, but now I feel like I'm explaining things to agents. Maybe I'm coming up with skills.

Basically, a skill is just a way to instruct the agent how to teach the thing. Maybe I could have a skill for MicroGPT that lays out the progression I imagine the agent should take you through if you're interested in understanding the codebase. It's just hints to the model to first start off with this and then with that. I could script the curriculum a little bit as a skill.

I feel like there's going to be less explaining things directly to people, and it's going to be more like, does the agent get it? If the agent gets it, they'll do the explanation. We're not fully there yet, because I still think I can probably explain things a little bit better than the agents. But the models are improving so rapidly that I feel like it's a losing battle to some extent.

I think education is going to be reshuffled by this quite substantially, where it's the end of teaching each other things a little bit. If I have a library of code, for example, or something like that, it used to be that you had documentation for other people who were going to use your library. But you shouldn't do that anymore. Instead of HTML documents for humans, you have Markdown documents for agents. If agents get it, then they can just explain all the different parts of it. So it's this redirection through agents, you know? That's why I think we're going to see a lot more of that playing out.

Sarah Guo

Well, we'll see if the great teachers know how to develop intuition for how to explain things to agents differently.

Andrej Karpathy

Ultimately, for example, with MicroGPT, I asked—I tried to get an agent to write MicroGPT. I told it, “Try to boil down the simplest things. Try to boil down my neural network training to the simplest thing,” and it can't do it. MicroGPT is, like, the end of my obsession. It's the 200 lines. I thought about this for a long time.

I was obsessed with this for a long time. This is the solution. Trust me, it can't get simpler. This is my value add. Everything else, the agent gets. It just can't come up with it, but it totally gets it and understands why it's done in a certain way, et cetera.

My contribution is kind of these few bits, but everything else in terms of the education that goes on after that is not my domain anymore. So maybe education changes in those ways, where you have to infuse the few bits that you feel strongly about into the curriculum—the better way of explaining it, or something like that. The things that agents can't do are your job now. The things that agents can do, they can probably do better than you, or very soon. And so, you should be strategic about what you're actually spending time on.

Sarah Guo

Well, we appreciate the few bits. Thank you, Andrej.

Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI | BidClub