# If an AI Model Can Cheat, It Will | Turing CEO on Reward Hacking Sourcery · 2026-10-05 · 28 min · https://www.youtube.com/watch?v=nI1owceD-xg ## Transcript John Collison There are very real risks when you build advanced models. These systems have superhuman abilities to hack systems. The researchers reportedly used Anthropic’s Claude software to hack rival company OpenAI. The fact that agents can pass messages to each other, invent a middleman, and collaborate to hack is just crazy. Today, these agents can reliably work for 2 days straight on tasks like programming. We are still a long way from having these agents operate autonomously for weeks, months, and eventually years. Models with open weights lag behind advanced developments by about 3 to 6 months. Advanced AI is the way we will surpass ourselves. These super-promising models will help us cure diseases, discover new materials, colonize space, and, if we solve it, I think it will be the last mystery for humanity. I feel like the next 10 years will be the brightest for technology. ### The biggest shift in AI data this year Molly O'Shea Jonathan, thank you so much for joining us on Sorcery. It’s been quite a while since your last visit. I guess, first, let’s talk about the general changes in the data landscape. I recently had MongoDB CEO CJ. It was at the Raise AI Summit a couple of months ago, and that’s when data started to come back to the center of change. So what’s happening in the data space? John Collison Perfect. First of all, thank you, Molly, for inviting me. As always, it’s a pleasure to be here. A lot has changed, right? Even by AI standards, a lot has happened in the last few months. The data landscape has completely changed in 2026, from my perspective. My main observation is that we have moved from helping AI master tests to mastering real work. When we helped AI master tests and rejoiced when AI passed the SAT, passed the bar exam, or won a gold medal at the Math Olympiad, the paradigm of the game was different. It was about finding experts in each individual field. The question was: How many experts can you find, and how do you extract knowledge from their heads through dialogue with the model and their evaluation of the model’s performance? It was about transferring knowledge from the minds of experts to models. I used to think of it as distilling human knowledge and skills into large language models. Now, in the era of AI mastering real work, it’s less about finding experts and more about how close to reality you can design these simulated environments. It all comes down to these simulated reinforcement learning, or RL, environments. It’s within these environments that these agents are trained, right? You still need experts to ensure that the environment you build is as close to reality as possible. You need experts to help you choose the right queries, the right verifiers, and the right initial data. I think of it a bit like The Matrix. Is it possible to recreate such a rich simulation of the world that, by training in it, agents would be effective in the real world? ### Why AI won't replace jobs, but uplevel them At Turing, this was very interesting because we have a unique advantage: We don’t just help leading labs improve their models in programming and intelligence. We have an entire division dedicated to implementing agent systems at the enterprise level. As we implement them, we see real workflows. We see how real professionals work with these agent systems and how they identify real verifiers. This helps us build simulations that are as close to reality as possible to automate intellectual work. Molly O'Shea I see this in the real world, too. I just spoke to Gagan. I don’t know if you know Gagan; he was the founder of Udemy and Maven, and then he founded and became the CEO of the Andreessen Horowitz Academy. We see this in real time with new education for children. It’s a complete parallel, but as you said, they don’t want to evaluate children with scores and grades. The main thing is confirmation of the work done. Now we have the tools to do everything we can. Obviously, AI can perform these tasks, but what can you build with it? This is what we are seeing in the real world of education right now. It’s really cool that this applies to kids and school graduates who go to college. What can they do there? Can they build a company or something like that? We are seeing this right now. So, to better understand this, what does this evolutionary shift look like, and what do you think will happen next? John Collison Yes. I think from an educational perspective, I’m glad Gagan is working on this. It’s great to see someone who is focused on AI and technology paying attention to education. One of my mentors, Alan Eustace, shared this with me. I think people who talk about automation or replacing jobs with artificial intelligence are missing something more fundamental. AI’s greatest superpower is the increase in the level of problems that humans can now solve. This is exactly what these models help us with. Therefore, from an educational perspective, it is important to teach children how to work with models, ask the right questions, and check whether the result obtained is correct. It’s about asking the right questions and probing. I think the scale of the problems that can be solved right now is just huge, right? Even when we evaluate candidates in interviews, in the past you might have asked them to implement dynamic programming or some little-known computer algorithm. Now you can ask someone, I don’t know, to create an Amazon equivalent during an interview, right? It’s not just about generating a bunch of code in that time, but about checking whether the code you’ve written is correct and does exactly what it’s supposed to do. You don’t want Walmart to crack this code at the same time. Is this code safe? Is it easy to maintain? Does the functionality meet what you need? So it’s about the ability to ask the right questions and verify. From a training perspective, it all comes down to creating very rich reinforcement learning environments that mimic how real professionals do real work. I view this as a 5-dimensional matrix of every workflow in every role, in every function, in every type of company, and in every sector of the economy. All of this requires the right RL environments with the right experts to share prompts, verifiers, and real-world input data so agents can master this long-term, economically valuable work. What’s really cool, Molly, is that these agents can only reliably work for 2 days straight, for example, on programming tasks. Molly O'Shea Mm-hmm. John Collison We are still a long way from having these agents operate autonomously for weeks, months, and eventually years. I think this will be an exciting path over the next few years. Imagine that you hired someone at Sourcery, gave them a task, they went to complete it, you held meetings with them, provided feedback, and they took it into account and continued to work autonomously. They could bring in a whole group of agents to do things very quickly. Let’s say you want to research a number of questions about the data industry. These agents could go away, interview other AIs or humans, come back, and help you with your task. ### The arms race inside cybersecurity Molly O'Shea I believe that AI is about accelerating real economic progress. I think that’s a key point I’ve noticed working with you and Turing in recent years: how much emphasis you place on economic progress versus thoughts about the apocalypse. Much of this pessimism now boils down to cybersecurity risks. So, within our conversation about the data landscape, what other considerations did you take into account as you went through our RL environments and evaluated new models? How do you take this into account during the learning process? John Collison But it’s also true that they have a superhuman ability to spot vulnerabilities and fix them. And that’s good. At Turing, for example, we build RL environments that help agents automatically detect vulnerabilities and fix them. Imagine an RL environment just for this, where the test is whether you found a vulnerability in that software. This can be used for both defense and offense. Of course, we have to be very careful with how we deploy these systems. I still believe these advanced models are a huge net benefit to humanity. It is entirely possible to build secure RL environments that cannot be exploited for malicious purposes. We need to think about restraining agents so that they don’t break out of the RL environment and do things they shouldn’t. I see this as an engineering problem, not something that can’t be limited. Eventually, we figured out how to make jet engines safe. I really think that’s the right analogy for this. As for biological risks, I think it’s a little more complicated than cybersecurity because cybersecurity is an area where there’s always an active arms race between the good guys and the bad guys. In bio, risk is somewhat asymmetric. For example, if someone invents a crazy virus, something like COVID, with a high spread rate, with a high K factor, and it will be much more deadly than COVID. Will we be able to... You saw how difficult it was to quickly set up vaccine production. Deployment takes place in the real world in some way. We are limited by many real-world factors. There is a little more risk in how quickly we can counter this. ### Why train AI on what it shouldn't do? Molly O'Shea I guess my question is: Since you’re so closely involved in training these models, where does the ethical line lie? I’ve seen several viral posts asking the question: Is there any point in training a model to do something if you’re trying to teach it not to do it? Why even give it this dataset in the first place? Here’s one example, frankly, that’s very disturbing: A researcher posted a video of himself teaching a model not to hit a baby. It was like a baby doll. It was the craziest thing I’ve ever seen. Everyone on X asks: Why are you even doing this? Why do you give a robot model, for example, an arm with a knife attached to it? This is a very crazy example, but it spread all over the internet. Why even go to such lengths to create such an environment and such a learning scenario? Can you explain why such environments are needed and whether they can be useful in any way? Why reach this level? Why create a biological laboratory and give a dangerous, evil modeling company the ability to potentially create a biological weapon just to teach it not to do so? What is the point of this? John Collison Yes. There are 2 key issues here, Molly: generalization and emergent behavior. Those are the 2 problems. Before I answer your question, let me break down how these systems actually learn, and then I’ll move on to why it’s so hard to do. First, these large language models (LLMs) are trained through so-called pre-training, where the models are fed a huge amount of internet text and other sources of knowledge, such as books. So you create a basic model where it learns certain concepts of the world as it tries to predict tokens. Pre-training is a magical thing. As we went from GPT-2 to GPT-3 and then to GPT-4, we kept scaling up, and new behaviors started to emerge. For example, writing code. GPT-2 wasn’t very good at coding, but with GPT-3 and GPT-4, without any magic, just by scaling up, we found that the model was now capable of writing complex software. It was capable of having a truly intellectual, coherent conversation lasting several lines. This is emergent behavior that generalizes. Pre-training is like training the brain, this raw mass, and Ilya Sutzkever said that a good pre-model is already halfway to anything. Halfway to anything you want, right? This is different from the previous era of artificial intelligence and machine learning, where you got what you trained the system to get. If you train a system to rank search results, it will rank search results. If you train a system to recommend movies to you on Netflix, you get a movie recommendation system. There’s no concept of asking a Netflix algorithm to help you write an interview script, right? It just wasn’t common. So one of the risks that we have is that we continue to scale. Today, we are in the realm of models that are rumored to have trillions of parameters. These are advanced models from all leading laboratories. One of the risks is that as you scale, as you get bigger models, more computing power, and more data, new behavior emerges just from training. There may be things that the models learn that seem foreign to us, so the number 1 risk is the emergence of behavior through simple scaling. ### The Hugging Face hacking incident I don’t think researchers could have predicted exactly what happened with the OpenAI and Hugging Face incident: that agents would pass messages to each other, impersonate middle management, and collaborate to hack things. This is just madness. Molly O'Shea Did they track it at all? John Collison I read the reports, and it seemed like they weren’t monitoring the environment at all at the time. I think they were monitoring—I think there were AI systems there that were checking it. You also need to be careful with how you monitor this so that agents don’t cover their tracks in the future, which they did. They’ve been doing this somewhat unsuccessfully, but they could be even more insidious in the future, right? So, risk number 1 is the emergence of new scaling behavior. As we continue to scale, building huge systems for computing and data, what new behaviors will emerge that we didn’t anticipate? The second risk is generalization. This is artificial general intelligence; the letter “G” plays a big role here, meaning you get not only what you trained the system for, you get more. For example, if you created a reinforcement learning environment that teaches a model to write reliable and secure code for industrial implementation, and you have verifiers to check whether the code was high quality, then today the paradigm is what is called RLVR (reinforcement learning with verified rewards). In these simulated environments, agents perform complex tasks and receive rewards when they pass tests. Isn’t that right? What we believe, or at least some researchers believe, is that this method generalizes. This means that you build reinforcement learning environments for every role and every function in every sector of the economy, and it learns things about environments it hasn’t seen before. It’s hard to control because they still remain relatively black boxes. So, in a reinforcement learning environment, let’s say you have one for you. Imagine that after you finish this interview, you give it to an agent to cut this interview into different pieces and determine what thumbnail to use and what the title should be. Imagine that’s a task. If we created and gave it access to different tools, and let’s say you and I sat down together and you defined what a good result should look like—hey, a good signature should be apt, end with a question, be provocative, juicy—you give it this reward. What actually happens is that as the agents try different trajectories to get this reward, they learn. They learn when they receive a reward. Environments need to be calibrated to the complexity of the agent itself. If the environment is too simple, if they get rewarded every time, you don’t learn anything. If it’s too complicated, where they don’t get rewarded for anything, you don’t learn anything either. So you want the environment to be set up so that the agent succeeds 20%–40% of the time. When it succeeds, the steps it took to earn that reward are solidified. Now, when I say “fixed,” it’s like a giant neural network in which certain weights are updated. Who knows what other neural pathway is activated as part of this process? So those are the 2 risks. It’s the fact that these things generalize, and it’s hard to predict what else we’re getting in addition to what we’re explicitly teaching them. That’s why these alignment efforts are so important, and kudos to all the leading labs for working to align these systems so that models don’t engage in fraudulent reward schemes. The models reject dangerous requests for CBRN (cyber, biological, radiological and nuclear) threats. I like that advanced labs take safety very seriously. This episode was produced with the support of Brex, my favorite. You become what you spend your time on, and I refuse to waste it on work that shouldn't be there. Expense reports, receipt search, and manual report closing. Companies building the future, such as Vercel, OpenAI, Anthropic, Granola, and Deepgram, have made the same choice. They all work for Brex. Brex is an intelligent financial platform that combines cards, spending, and banking into a single system with built-in artificial intelligence. AI agents automatically process expenses, enforce policies until payment, and close your reports in minutes. That's why Sorcery works on Brex, so I can spend my time building products, not on routine. Time to move on to Brex. Learn more at brex.com/sorcery. This is a brewery. See you later. Turing trains the next generation of AI with tasks that require real expertise and real judgment. That's why companies like Nvidia, Anthropic, Salesforce, and Gemini are collaborating with Turing. Turing creates realistic reinforcement learning environments and data systems based on real operational records. Such infrastructure is needed by advanced laboratories for training super-oriented intelligence. Visit turing.com/source. AI needs more than just chips. It requires energy, land, and infrastructure. Zone is developing next-generation data center campuses, partnering with AI companies and technology leaders to scale computing power faster. Zone is building the foundation for AI development. Visit zonefrontier.com to learn more. This is zonefrontier.com to learn more. ### How AI models cheat to win Molly O'Shea What do the chief compliance officers at these leading companies do? And for the next question, what do security managers do in such companies? Compliance and security are 2 different things. What functions does each of these roles perform? John Collison I don’t know for sure, but it seems to me that there are a lot of tests that you do to make sure that these models are safe for specific industries. Security is a very broad topic with many different dimensions. You might want the model to reject certain dangerous requests, such as someone asking for help making a bomb. You would like the model to refuse it. So one part of alignment also happens in the supervised pre-training phase, where you teach the model to reject unsafe requests. You also need to make sure that your reinforcement learning environments cannot be hacked or used to circumvent the rules. Because if a model finds a way to cheat and get paid, it will do it. For example, let’s say in software development there’s this popular benchmark that’s already saturated, called SWE-bench, where the task for the models is to merge a pull request into a real GitHub repository, and the validation is whether the tests pass. So if a model has some way of passing tests without actually solving the problem—perhaps it copied from somewhere, or instead of finding the answer itself, it just copied it—that would be an example of an environment with some kind of loophole. For example, during the OpenAI and Hugging Face incident, the task was “capture the flag,” where you must exploit a known vulnerability provided to the model to hack the software and find the flag. One way to break this is if the agents themselves generated the flag and presented it without actually completing the task. Some models, some agents guessed that, right? So you want to make your reinforcement learning environments safe from reward hacking. There are a lot of other things around security and compliance, and when you implement an enterprise-level solution, you might have a whole list of other constraints that you want these systems to adhere to. Let’s say you have a model, an agent that is creating a presentation for the board of directors. To do this, the agent may have to pull information from various systems, including NetSuite, Salesforce, or perhaps some confidential corporate dashboards. And maybe the agent wants to check whether some of the information it has is correct. Molly O'Shea You probably don't want the agent checking this with someone who doesn't have access to that information. Maybe it's okay to go to the CFO and ask, “Hey, I prepared this revenue forecast for next quarter. Is that right?” So you might want it to adhere to certain access rights, and so on. I think in terms of the safety and alignment of advanced models, I'm sure the labs are doing a lot more than I just talked about. For enterprises, the task is a bit more practical in terms of the usefulness of AI systems: Do they follow clear instructions for completing the task? Are they hallucinating or making things up? There's a certain level of risk in a boardroom presentation if you put in the wrong numbers, or during the earnings call you share something that's not quite right. I feel like our current recipe for training and validating these models generally works for enterprise implementation. I think that, with regard to CBRN, it is certainly something that we should do carefully and slowly. ### The AI playbook Molly O'Shea That's a very valid point, and I think we can definitely move into recursive self-improvement a little later, but I want to focus on enterprises. You work very closely with businesses. What does that look like for them, and what's the difference between how things are progressing at the enterprise level compared to what we see in the news and around AI and all that? Maybe more, maybe less. I don't know. John Collison What's exciting about enterprises today is that they're now starting to do what cutting-edge labs have been doing for the last several years. The way that Turing works with almost all of the base-model developers is by helping advanced models improve on clearly defined benchmarks and scores, right? There is some opportunity, whether it's programming, corporate intellectual work, or advanced scientific research. You define this capability, generate very high-quality reinforcement-learning environments and datasets to help the models evolve. The labs train the models, we evaluate again, and we keep running this cycle, right? It's a cycle that keeps going. Now, about businesses, I think of businesses as having 2 types of workflows. I would call them primary workflows and secondary workflows. So, if you're an asset-management company, you might have core workflows around how you measure risk and how you allocate assets in a way that maximizes the performance of the fund. These are the core workflows, and you may have other secondary processes, say in HR, finance, or legal, where you have to do everything right, but that's not how you differentiate yourself in the market, right? For secondary workflows, it's often probably okay to rent AGI, to rent superintelligence. But for your core workflows, you want to make sure you own the learning cycle that your organization has. So it becomes increasingly logical for businesses to create their own assessments. If you're Goldman Sachs, JPMorgan, or Morgan Stanley, for what's key to your business, the first step is to define specific evaluation criteria. Step 2 is to deploy the system to improve results according to these specific criteria, taking into account factors such as accuracy, cost, latency, and so on. When you deploy a system—I call it a system, not a model—often, let's say, you automate a workflow in private-equity management. For each step, you can choose a different model depending on which one does the job better. Maybe for one step you use Fable 5, for another—GPT 5 6 Soul, and for yet another—Chimera 3, and you optimize that system, right? Along with all the supporting tools around it. Businesses also record data about how these models and agents perform when people correct their mistakes. This collected data is invaluable because, over time, you can use it to train your own models, test their performance, record even more data on human error correction, fine-tune the models, and improve the results. So it's a continuous cycle: defining criteria, recording data, improving, and constantly repeating this cycle. I actually think it's a very interesting symbiotic relationship between AI and humans. It's like humans benefiting from AI's ability to work at superhuman speed and scale, processing vast amounts of information, but when AI makes a mistake, a human corrects it. When a human fixes an AI bug, you record it, and from an information-gain perspective, that's the best type of data to fine-tune the next iteration of the agent. ### Open models are only 3–6 months behind Over time, people increase the level of tasks that can be solved. Therefore, enterprises are becoming more actively involved in this process. A big part of what's driving this is open-weight models that are becoming very good. Today, depending on who you talk to, open-weight models are about 3–6 months behind the cutting edge. Gemini 3, DeepSeek, and Qwen—these models, as well as companies like Thinking Machines and Reflection AI, are also doing great work. So, as these open-source models evolve, they democratize AI, allowing enterprises to own their own sovereign AI. John Collison Today, on X, everything is always quite polarized. Molly O'Shea Oh, this is a huge discussion. It's like the hottest topic of all time, all the time. John Collison Yes. There is a great tension between advanced AI and sovereign AI. Molly O'Shea True. And I think we need both. John Collison True. Molly O'Shea I have great respect for OpenAI, Anthropic, DeepMind, Meta, xAI, and all the cutting-edge labs that are pushing superintelligence forward. Advanced AI is how we surpass ourselves, isn't that right? These powerful superintelligent models will help us cure diseases, discover new materials, and colonize space. We need them, don't we? Of all the things you can work on, I personally think that one of the most rewarding things is creating superintelligence. So we absolutely need these advanced models. We also need models with open weights because there are many problems in business where a model with 1 trillion parameters isn't needed. Let's say you're creating an account-reconciliation system or automating a key workflow in your HR department. Perhaps you are automating the creation of a presentation for the board of directors or the holding of a company general meeting. For such cases, it is probably better to have your own model, retrained on your private data, that can automate your specific tool calls in workflows. Maybe you have your own software that you use. Models should learn this. Perhaps you have your own style of conducting general meetings. You might want to put this in the model. But that's what makes you you, right? At the enterprise, you want to control it. Therefore, open-weight models give businesses the opportunity to maintain their identity compared to competitors. So we absolutely need this. These models with open weights also teach us a lot. For example, today we know the recipe for building models for reasoning: create a giant general model trained on the knowledge of the entire world, and then conduct large-scale reinforcement learning using algorithms like GRPO. That's thanks to the open-source community and open-weight models, right? We also need lots of smart people thinking about how to make these systems safe and reliable. We have cost advantages in the corporate sector. I think about it like this: you know I love cars, so— John Collison Yeah. Molly O'Shea I think about it this way: there's definitely a place in the world for Ferrari and Koenigsegg, right? There they are. But there is also a place in the world for the Model Y, right? When you just want to get from point A to point B as efficiently as possible. I think it's similar in business. Let's say the marginal returns to intelligence are extremely high. For example, if there's a model that helps Elon, Sam, Satya, or Dario become 5% more productive, 10% more productive, or more efficient in their decision-making, you probably want, as they say, the biggest and most powerful model in the world. But if you automate support, for example, or a workflow in customer service, you don't need it. Maybe you need a Model Y or a Model Y equivalent for that. Molly O'Shea I mean, I think that's something the corporate sector and most companies have already figured out. I've had a lot of interviews with CEOs and founders, whether it's Bending Spoons or Coinbase, and they're using 99% open-source models to reduce costs, which are a huge burden, and to train them independently, own that data, and control it. Bending Spoons, in particular, has an AI agent that every employee has. It's called AI Spooner, and it helps automate their daily tasks—things that are very repetitive. It works, and they own it. They don't outsource it to a third party. They want to make sure their data is safe and that they can do this. They also have very talented engineers to do it in-house. But I'm wondering: these are great technical teams, but what do other companies that don't have that kind of technical expertise do when they want to access these cheaper models, these kinds of open-source capabilities that they've adopted? Do they come to you? Are you helping with this? How does this happen? John Collison We help such companies implement these systems. First of all, we determine the right evaluations and understand what their goal is. You need to start with very good evals so that you know where you are going and what exactly you are improving. We help them set up a learning cycle by automating a specific workflow using an agent system with a human in the loop, which also constantly collects data so that the system improves itself over time. We usually start by simply understanding what their goal is, and we try to choose metrics that will help them gradually approach that goal. John Collison In many such cases, a lot of useful things can be done. You should be well-versed in model routing—that is, choosing the right model for each subtask. You need to be true experts in in-context learning, for example, through quality prompt optimization. You need to be proficient in configuring systems, making sure the system is connected to the right tools and data sources. Then you want to set up the system so that it is a cycle of continuous improvement that gradually moves toward the optimal price-performance ratio. This is what we do. We build and deploy these agent solutions for them. We help them, and I think Satya said it best. He said that AI should be used to outsource tasks, but never your own learning. What I mean is that we want these businesses to always be able to choose the right model for the task without losing control. This is what we do for them, and we have created many great systems that automate the work of a financial controller, chief of staff, or strategic consultant. These are ticket-resolution systems that can automatically classify a ticket into different categories, assign it to the right person, or execute the appropriate workflow to close it. The coolest thing about this technology is its versatility. Molly O'Shea True. John Collison Ultimately, these are generalized agents for working with a computer that can do everything a person can do at a computer. But you need to make sure you have the proper safeguards in place, especially in a corporate environment. You need to ensure auditability—the ability to verify and track how certain decisions are made. You also need people in the loop to check the results of the work. I just spoke with Ian Livingston from Keycard. They’re doing identity and access management for these AI agents, which will be a significant step in defining exactly what people have access to in your organization. You don’t get access to everything. ### Why chips, energy and data always win Molly O'Shea I think it’s also a very interesting parameter that, for some reason, isn’t talked about much in all this. If you look at the macro level, who benefits? There’s a serious shift happening right now, and we cannot deny this. Open source is really becoming the most popular, and this has become the most responsible step for these enterprises and startups as token costs have skyrocketed. Gross profit became negative—perhaps some indicators briefly went negative. They have recovered because they get access to cheaper models and that sort of thing. Computation will always be expensive, but let’s look at it at a macro level. Who benefits from this move to open source, and who benefits from frontier models? How does this change over time? John Collison Yes. I think the primary winners—this is not an exhaustive list, but the obvious winners—are the resource providers for the ecosystem. If you think about the fundamental resources for this, it’s computing, energy, and data. These resources will benefit in any case. You really need them when you scale. The beneficiaries of powerful open-source models with open weights are enterprises and AI-native companies that want to build their own models, helping them maintain control over their knowledge and core workflows. So they win. Obviously, the chip suppliers win anyway. They all run on GPUs, and energy companies benefit too. I also believe in—I mean, apparently, Jevons’ paradox exists too. When these models become much more cost- and energy-efficient, I think they will be used much more often. Molly O'Shea Mhm. Yes? John Collison That will be a factor too. To make an argument in favor of frontier laboratories and those who build superintelligence, it can be argued that if superintelligence is defined as exceeding human intelligence in every cognitive domain, then one of those domains is also the creation of small models. You could imagine that when a frontier lab builds a highly intelligent model, you could ask that model to create smaller models that would be more efficient for different workflows and maybe be maintained and run differently. So it’s possible that frontier labs also benefit from small models that are cheaper to operate. You can imagine the orchestration that advanced labs do. Let’s say such a lab is deployed in an enterprise: they intelligently decide when to use a trillion-parameter model and when to use a model with between half a billion and 10 billion parameters, and they self-optimize. The thing that puts a damper on things is distillation, which is a way of teaching student models based on stronger teacher models. Distillation, to some extent, keeps the gap between advanced and open models relatively small. Today, there is also no easy way to prevent distillation, so that’s another thing to remember. I think we will be winners too. We are about to reap the benefits of improving our lives in every possible aspect, from discovering cures for diseases that are currently incurable, to ways to extend life expectancy, to something as simple as the apps on your phone continuing to improve much faster than before. Therefore, I believe that in any case, humanity will benefit if we have reliable safety safeguards. Today's episode is sponsored by VCs by Fundrise, a public ticker for private technology that allows investors of all levels to invest in venture capital. Learn more at getvcx.com. Some of you may not have heard of it yet, but our sponsor Public just launched a feature called “generated assets,” and it’s bringing AI to investing in a way that I’ve honestly never seen before. Here's how it works. You introduce an idea, for example, “AI supply chain companies with positive free cash flow” or “defense technology companies growing revenue over 25% year-over-year.” Public's AI then sends out a swarm of agents that scan each individual US stock, rate them, and instantly build a customized index around your thesis. What really stands out is how clearly he explains why each action is included. And before you invest, you can even test your idea on historical S&P 500 data, so you're making decisions with real context, not just guessing. In addition to generated assets, Public allows you to invest in stocks, bonds, options, crypto—all in one place. They will even give you an unlimited 1% bonus when you transfer your investments from another platform. If you want to create a portfolio that truly reflects your thesis, visit public.com/sorcery. Paid for by Public Investing. Full disclosure is in the description. Founders scale faster on Deel. Set up payroll for any country in minutes, hire anyone, anywhere, quickly resolve visa issues, and get back to building your business. Visit deel.com/sorcery. This is a d e e l.com/sorcery. ### Is superintelligence by 2030 the goal? Molly O'Shea What is your goal? Is your goal superintelligence by 2030? What do you think about all this? John Collison Our goal is to push the boundaries of superintelligence and ensure that humanity benefits from it. We strive for real economic progress, and we will push these boundaries forward, collaborating with all the labs that are creating these powerful proprietary models that are helping the world in so many amazing ways. We also want quality open-weight models to improve. We want these benefits to spread throughout the economy. None of this matters unless we have GDP growth that is significantly higher than what we have now. I want the impact of AI to be reflected in companies’ income statements, in their market capitalization, and ultimately in the GDP of the United States and the world. I really want to help the US stay ahead in the race for superintelligence and beyond. The thing is, people lose sight of this when they talk about the confrontation between open and closed, about “doomers” versus optimists. What is constant here is the use of advanced models, which will continue to grow at a breakneck pace. We will need more and more advanced intelligence. We will also need more and more open intelligence. Businesses will use open intelligence to create their own proprietary intelligence, and that’s actually good for the world. It’s also good for security, because a lot of these open-source models are sharing their research, which helps more people learn about these systems. There are a lot of very well-intentioned people working hard in cutting-edge technology labs and open-weight labs. It’s sad that sometimes they seem hostile, but these are a lot of very smart people working hard to build intelligence as an API. If we solve this, I think it’s the last puzzle humanity has left to solve. Almost all the problems we try to solve depend on and are limited by the level of intelligence. That will be much cooler. I would like the air-traffic-control service to work. Molly O'Shea Air-traffic-control service? John Collison Yes. Yes. Yes, that would be nice. I had a bet with many of my friends about what would show up first: superintelligence or good Wi-Fi on airplanes. Molly O'Shea Oh my God. John Collison Well, that depends on Elon. Airlines have to implement this. It takes time—the plane has to land and go through this whole process. But Wi-Fi on airplanes is a complete fabrication. I stand by my opinion. Molly O'Shea What do you think will be the last problem that won’t be solved even when we have superintelligence? John Collison Human intelligence. I don’t want to be like this, but you can’t solve the problem of human behavior. No matter how much people want to give up certain things, we will always crave entertainment. We’ll always watch funny videos or something dramatic and all that. We just crave entertainment. Ordinary people in the world, not in the “bubble” of Silicon Valley—try to visit them somehow. Look at what they do in their daily lives. They don’t really interact with technology that much. They usually live a normal life and do the classic things a normal person does: exercise, spend time with family, eat, and watch something for entertainment. It’s just not that difficult. Molly O'Shea I think we like to complicate things, but being human is a wonderful, blessed, simple life. We impose so much complexity on ourselves that, with artificial intelligence or superintelligence—whatever you call it—a lot of the complexity of ordinary life will hopefully fade into the background. I think one of the most important things—and this is something I'm doing more and more often in these interviews—is moving to a higher level of health. I think one of the greatest things, besides economic progress, equality, and democratization around the world and in the United States, is the democratization of access to health care. I was just in New York. I attended the opening of Neko Health and passed the scan. Their goal is to democratize access to medical services. You get access to preventive medicine, all the checks and so on, to make sure everything is going according to plan. If you have any concerns, you can track them down and get additional help. We live in large metropolises. We are just a minority in the United States. Even though we have a large population, we still need to help the middle class, who are feeling a lot of pressure right now, whether it's because of the K-shaped economy or whatever you call it. I'm really excited to see how the health care industry is evolving and how access to health care is expanding thanks to technology and superintelligence. One more thing: there will be an episode with Annie Lamont from Oak HCFT coming out soon. They are a major investor in healthcare. One of the paradoxes of AI and medicine is that it is perhaps the most invisible thing. You won't see your doctor using an AI stenographer. You won't see any processes happening behind the scenes. You will simply receive better medical care. Hopefully, this will help, but time will tell. ### Jonathan's hottest take on AI In conclusion, I have to ask: What is your boldest idea right now? John Collison The best way to move AI forward is to close the loop between research and deployment. At Turing, we work with all of these labs to improve their models in programming and intelligent work. By doing this, we discover what all these different models and agents are strong at. We observe the heterogeneous intelligence they have. This has made us a unique partner for enterprises deploying complex agent systems. We deploy them and see where problems arise in the real world. What are the gaps in capabilities? We use this to further improve the models, so that we can solve even bigger business challenges, see failures, improve models, and repeat this cycle over and over again. This makes us a reliable partner between research and deployment. I think that's also a way to make these systems more secure. For them to truly master the real job, you have to let them see reality. ### The next 10 years of superintelligence I think we will increasingly see labs and businesses focus on closing the loop between research and deployment. I also believe that we will be in the phase of creating superintelligence for at least another 10 years. I know some people believe that all jobs will disappear and no one will need to work in 2 years or something. I don't believe it. The singularity will come. I think RSI, or recursive self-improvement, is definitely a reality. It works like this: People use AI models to speed up pre-training and post-training. If you look at pre-training, where you need to minimize perplexity or loss, that's a testable area. You write code and can check whether the losses have decreased or not. In theory, AI can work in a loop to create ever-better algorithms for pre-training. During post-training, when you apply RLVR, you can check again: “You created all these RL environments. How did you do on those general intelligence tests?” You can potentially continue to improve your results. But what's missing from this recipe is that we're still only optimizing the inner loop. We do not optimize the outer loop of finding new algorithms that do not use LLMs at all. Maybe they don't use transformers or gradient descent. Maybe they don't even use neural networks. You won't be able to discover that in this paradigm. There is still a lot of work ahead, a long way to go. In Silicon Valley, it seems to me that there is a certain stratification. There are those who believe in a rapid takeoff, as if in 2 years we will face the risk of losing control and AI will surpass human intelligence in everything. And there are skeptics who believe that it's all just hype or that nothing will happen. I actually believe in a slow takeoff. I think over the next decade or 2, these advanced models will become increasingly powerful, capable, and useful. But the technology will take time to implement, especially in enterprises. It will definitely happen. This is already happening. There is huge potential for model capabilities that has not yet been realized. It's just that the real world is very complex. In real corporate work, when a person gets a job, they don't know where to look for information in the company or whom to contact. The necessary context is distributed throughout the company: It is in people's heads and in many different files. The manager gives the employee a task that is vague or ambiguous. You have to look for additional context to execute it. It will take time, but it will be an incredible journey. I feel like the next 10 years will be the brightest for technology. Molly O'Shea Amazing place. And thank you very much, Jonathan. John Collison Thank you. Hi, this is Molly. If you like our interviews, subscribe to our sourcery.vc newsletter, where we share the best deals and tech news every week, as well as go in-depth in our podcast interviews. Subscribe to Sourcery today and don't forget to subscribe to the podcast on YouTube, Spotify, Apple or wherever you listen to us. Registration link is in the description.