[BidClub_]
Acquired · · 236 min

Google Part III: The AI Company. Google is amazingly well-positioned... will they win in AI? (Audio)

Ben GilbertDavid Rosenthal

YouTube ↗
TL;DR

  • Google enters the AI platform shift with the only end-to-end stack the hosts can identify: a frontier model in Gemini, millions of TPUs, a $50 billion-plus cloud business, global consumer distribution, and a search monopoly funding the whole effort. That makes AI Google’s “birthright,” but also creates the episode’s central innovator’s dilemma: the better direct-answer product may be structurally less profitable than the “giant cash printing business” it could replace.

  • The modern AI industry grew from research Google repeatedly invented, financed, productized internally, and then allowed to escape. Early probabilistic language models powered “Did you mean?” and AdSense; Google Brain’s cat paper launched the recommender era; Google acquired Geoffrey Hinton’s team and DeepMind; and eight Brain researchers published the 2017 transformer paper. Yet all eight transformer authors eventually left, while OpenAI, Anthropic, NVIDIA, Character.AI, and others commercialized the resulting platform shift.

  • Google’s decisive infrastructure advantage began when neural networks threatened to require “another Google” worth of data centers. It ordered 40,000 NVIDIA GPUs for $130 million, then designed and deployed the first TPU in only 15 months using reduced precision, a hard-drive-compatible form factor, and FPGA prototypes. The hosts cite an estimated 2–3 million TPUs today, approaching the scale of NVIDIA’s roughly 4 million GPU shipments last year and potentially making Google the industry’s lowest-cost producer of tokens.

  • ChatGPT converted AI overnight from a sustaining technology for search and ads into an existentially disruptive one. OpenAI’s improvised GPT-3.5 chat interface reached 1 million users in under a week, 30 million in a month, and 100 million within two months; Microsoft then declared “a new day for search” and wanted the world to know it had “made Google dance.” Google’s rushed Bard response was inferior and factually wrong in its launch material, contributing to an 8% one-day stock decline.

  • Sundar Pichai’s recovery has been “rapid but not rash”: merge Brain and DeepMind, appoint Demis Hassabis, standardize the company on Gemini, and ship at frontier-lab cadence without immediately destroying search economics. Gemini progressed from its December 2023 release to Gemini 1.5’s 1 million-token context window, Gemini 2.0, and Gemini 2.5 Pro; Google simultaneously layered AI Overviews and AI Mode onto search while keeping Gemini as a separate full-chat product. The result is a delicate “ballet” between adoption and cannibalization, not a wholesale replacement of google.com.

  • Google can finance the most capital-intensive race in business history from a franchise still producing $370 billion of annual revenue and $140 billion of earnings. It has invested aggressively in AI infrastructure while retaining $95 billion in cash and marketable securities, buying back stock, and paying a dividend; Google Cloud now exceeds a $50 billion annual run rate and grows about 30%. Rivals with strong models but no self-funding cash engine remain dependent on external capital and hyperscaler infrastructure.

  • Waymo is the clearest proof that Google can sustain a technically difficult AI bet long enough for “slowly and then all at once” to arrive. After roughly $10–15 billion of cumulative investment, Waymo has logged more than 100 million fully driverless miles, operates about 2,000 vehicles, and reports 91% fewer crashes involving serious injury or worse than comparable human driving. Against the CDC’s estimated $470 billion in total U.S. costs from crash deaths in 2022, the hosts now see a plausible “Google-sized opportunity” where they once saw a wild-goose chase.

  • The investment case turns on whether Google’s full-stack cost advantage and distribution can outweigh worse monetization and lower market share in AI. The bull case is that longer prompts expose more intent, Google can bundle AI across 150 million Google One subscribers, and its chips, data, YouTube corpus, applications, and private network compound; the bear case is that AI siphons off lucrative travel and health queries while supporting neither search-like ads nor 90% share. The unresolved question is stark: “Would they rather go bankrupt than lose at AI”—and will they still choose the mission if AI never becomes as good a business as search?

Digest · the substance, structured for research

1. Google owns the AI stack but may have to destroy its best business

  • Ben frames Google’s position as the textbook innovator’s dilemma: a government-defined search monopoly with roughly 90% share has invented a product that is better for many purposes, but has not found a way to make it “anywhere near as profitable” as the incumbent one.

  • The strategic assets are unusually complete: Gemini is a top-tier model, Google Cloud exceeds $50 billion in revenue, TPUs are the only AI-chip deployment the hosts consider comparable in scale to NVIDIA GPUs, and google.com remains “the front door to the internet” whenever users express intent.

  • One researcher’s formulation becomes the investor shorthand: without either a frontier foundational model or an AI chip, a participant “might just be a commodity in the AI market.” Google is the only company the hosts identify as having both—and it also owns the cloud and applications through which they are distributed.

2. Larry Page conceived Google as an AI company from the beginning

  • Larry’s father studied machine learning and artificial intelligence when the field was deeply unfashionable, and that contrarian orientation carried into Google. Even PageRank can be understood as a statistical, AI-adjacent system.

  • Larry’s 2000 formulation already contained the end state: “Artificial intelligence would be the ultimate version of Google.” The ultimate search engine would understand everything online, know exactly what the user wanted, and return “the right thing”; ordinary search was merely an incremental route toward that goal.

  • Google’s later dilemma therefore is not an unrelated disruption invading from outside. AI is simultaneously the company’s founding mission, its greatest research accomplishment, the mechanism behind billions in existing revenue, and a product shape that could dissolve the economics of its original implementation.

3. Compression became Google’s first theory of machine understanding

  • Around 2000–01, early engineer Georges Harik told Ben Gomes and new hire Noam Shazeer that compressing data might be technically equivalent to understanding it: if a system can reduce information and later reconstruct it, it must have captured the meaningful structure that survived the compression.

  • The hosts connect Harik’s school-test analogy directly to modern language models: the world’s knowledge is compressed into a comparatively small collection of parameters, then “uncompressed” into useful answers. David cautiously challenges the anthropomorphism—models may demonstrate understanding, or they may merely “mimic understanding.”

  • Harik and Shazeer ignored internal skeptics and pursued probabilistic natural-language models. Harik’s defense was pure early-Google culture: “Sanjay thinks it’s a good idea, and no one in the world is as smart as Sanjay,” referring to Sanjay Ghemawat.

  • Their first commercial proof was Google’s “Did you mean?” system, a next-token-prediction ancestor that inferred a likely intended query from observed corrections. Besides improving the experience, it prevented infrastructure from repeatedly serving useless mistyped searches that users immediately replaced.

4. PHIL turned language modeling into billions of dollars

  • Harik and Shazeer expanded the work into PHIL, the Probabilistic Hierarchical Inferential Learner, a “large” language model by early-2000s standards. By predicting sequences and extracting meaning, it became useful far beyond spelling correction.

  • When Susan Wojcicki and Jeff Dean prepared AdSense in 2003, PHIL supplied the missing understanding of third-party webpages. Dean reportedly implemented the system in a week, matching Google’s existing AdWords corpus to publisher content and creating billions of dollars of incremental revenue by massively expanding ad inventory.

  • By the mid-2000s, PHIL reportedly consumed 15% of Google’s entire data-center infrastructure across AdSense, spelling correction, and related applications. Language modeling was therefore already computationally expensive, production-grade, and directly tied to Google’s bottom line two decades before ChatGPT.

5. Jeff Dean converted a 12-hour research model into a 100-millisecond product

  • Google Translate’s chief architect, Franz Och, trained an enormous n-gram model on two trillion words from Google’s search index and achieved a spectacular BLEU score in a DARPA translation challenge. The catch was that translating one sentence took about 12 hours—acceptable when submissions were due days later, useless for consumers.

  • Dean rearchitected the work so words and sentence components could be processed in parallel across Google’s distributed CPU infrastructure instead of sequentially. After several months, average translation latency fell from 12 hours to roughly 100 milliseconds, and the research system shipped in Google Translate.

  • The result established a recurring Google playbook: pair frontier machine-learning research with infrastructure engineering until it becomes a product at global latency and scale. The same predictive machinery then improved search-query completion and AdWords quality scores, where better click-through-rate estimates translated directly into revenue.

6. Google built an academic talent monopoly before AI was fashionable

  • In 2007 Larry recruited Sebastian Thrun from Stanford’s Artificial Intelligence Laboratory, effectively replacing a startup financing process with Google signing bonuses. Thrun brought a network of professors, researchers, and students that had included future Meta product chief Chris Cox and, briefly, Stanford undergraduate Sam Altman.

  • After successful work on Street View and the Ground Truth maps project, Thrun persuaded Larry and Sergey to let academics work part-time at Google while retaining university posts. Researchers gained compensation, infrastructure, colleagues, and the chance to deploy work to millions; Google concentrated the scarce practitioners from highly specialized PhD lineages.

  • Thrun invited Geoffrey Hinton to give a 2007 Google talk on deep neural networks, then brought him in as a consultant and, later, a roughly 60-year-old “summer intern.” Neural nets remained heretical because earlier hype had failed, but Hinton argued that multilayer systems needed more compute, not a different intellectual foundation.

7. Google Brain proved asynchronous distributed neural networks could work

  • Around 2010–11, Andrew Ng and Jeff Dean decided to take a large swing on a deep-learning model within Google X’s Google Brain effort, after earlier projects called Brain on Borg and Cortex had disappointed. Their ambition was a genuinely large deep neural network running across Google’s highly parallel CPU infrastructure.

  • Dean named the training system DistBelief: partly a reference to distributed computation, partly because “no one thought it was going to work.” Conventional research favored tightly synchronized computation; DistBelief instead updated parameters asynchronously, sometimes using stale information, and nonetheless worked.

  • That systems breakthrough mattered as much as the model. Google could split both data and models across enormous fleets, turning an architecture regarded as computationally impractical into something its existing infrastructure could train.

  • The resulting “cat paper” used a nine-layer network, 16,000 CPU cores, and 10 million randomly selected, unlabeled YouTube frames. A high-level neuron became excited by cats without ever being told what a cat was—evidence that meaningful representations could emerge through unsupervised learning.

8. The cat neuron quietly launched the first AI era

  • YouTube’s immediate problem was weak metadata: uploaders described videos poorly, leaving search and recommendation systems unable to know what the underlying content contained. If a neural network could answer “cat or not a cat” directly from frames, it could classify almost anything that drove relevance.

  • The hosts trace a huge commercial lineage from that result: YouTube recommendations, feeds, dwell time, copyright matching, revenue sharing, explicit-content filtering, and eventually similar systems at Facebook, Instagram, TikTok, Reels, and Shorts. This was not laboratory curiosity; it reorganized how humanity spent leisure time.

  • David’s signature correction is that the AI era began in 2012, not with ChatGPT in 2022. For any company with a feed, classifier, or recommender, neural networks were already “shaping human existence” and generating hundreds of billions of dollars—only invisibly, inside existing products.

9. AlexNet made gaming GPUs the engine of modern AI

  • Hinton and Toronto graduate students Alex Krizhevsky and Ilya Sutskever entered the 2012 ImageNet competition using a deep neural network trained on two consumer NVIDIA GeForce GTX 580 cards. They rewrote the algorithms in CUDA instead of relying on the supercomputer-class CPUs then associated with advanced research.

  • ImageNet’s best previous systems misclassified about 25% of images; AlexNet cut the error rate to 15%, a 10-point absolute improvement and roughly 40% relative gain in one year. The discontinuity convinced a previously skeptical computer-science community that deep learning had crossed into practical superiority.

  • Jensen Huang’s description of AlexNet as AI’s “big bang moment” carries both technical and financial meaning: two off-the-shelf gaming cards unlocked the next research level and put NVIDIA on its path from a PC graphics supplier to the most valuable company in the world.

10. Google acquired Hinton’s team through a professor-run auction

  • Hinton, Krizhevsky, and Sutskever formed DNNResearch without a product—only the researchers who had won ImageNet. Baidu opened with a $12 million offer, prompting Hinton to construct a timed auction in which every new bid reset the clock for another hour.

  • Microsoft, Google, Baidu, and cash-constrained DeepMind participated while Hinton managed proceedings from a hotel room at Harrah’s Lake Tahoe during the NIPS conference. When the researchers decided Google was where they wanted to work, they stopped the auction at $44 million rather than maximizing the final price.

  • Krizhevsky and Sutskever asked Hinton to take 40% while each retained 30%. The trio joined Google Brain, where even percentage improvements in search, ads, Gmail, and YouTube could uncover “quite a bit of loose change in those couch cushions” across tens or hundreds of billions of revenue.

11. DeepMind pursued general intelligence while others pursued classifiers

  • DeepMind was founded in London in 2010 by Demis Hassabis, Shane Legg, and Mustafa Suleyman. Hassabis combined childhood chess prodigy, commercial game designer, failed gaming entrepreneur, computer scientist, and neuroscience PhD; Legg belonged to AI’s self-described “lunatic fringe” and helped popularize “artificial general intelligence.”

  • Their leap was conceptual: AlexNet and the cat paper classified patterns, while DeepMind proposed creating an intelligence that could generalize beyond narrow tasks and eventually become smarter than humans. Its enduring mission was “solve intelligence and use it to solve everything else.”

  • The website’s vague promise of general-purpose algorithms for “simulations, e-commerce, and games” reflected an honest absence of near-term products. What DeepMind primarily needed was enough capital and compute to conduct open-ended research—not a conventional startup plan with customers and unit economics.

12. Peter Thiel and Elon Musk funded the mission because it sounded impossible

  • Hassabis and Legg targeted the 2010 Singularity Summit because conventional London investors were unlikely to fund an AGI lab. Legg’s requirement was someone “crazy enough” not to sweat a few million dollars, who liked “super ambitious stuff” and would ignore professors saying it could not work.

  • Peter Thiel missed Hassabis’s carefully designed conference talk, so Hassabis approached him at an afterparty through their shared interest in chess. The next-day pitch led Founders Fund to anchor an approximately $2 million seed round—tiny by later AI standards, but enough to begin hiring and experimenting.

  • Through the PayPal network, Hassabis met Elon Musk at SpaceX. When Musk described Mars as humanity’s backup, Hassabis asked what happened if AI was the catastrophe: an AI could reach Mars through communications or any systems humans brought with them.

  • Musk reportedly sat silently, concluded “that’s probably true,” and soon invested. The hosts treat this as the moment his concern about safe AI flipped on, while also intensifying his belief that machine vision and learning could become central to Tesla’s autonomous-driving ambitions.

13. Google won DeepMind by offering mission, infrastructure, and separation

  • By late 2013 Mark Zuckerberg reportedly offered as much as $800 million for DeepMind, roughly twice what its founders would make from Google’s proposal. Facebook would accommodate unusual arrangements, but Zuckerberg would not grant Hassabis lasting independent control; DeepMind would become part of Facebook.

  • Musk countered with Tesla stock when Tesla’s market capitalization was around $20 billion—stock the hosts estimate later appreciated roughly 70-fold. Yet Musk wanted DeepMind focused on autonomous driving, while the founders refused to confine the lab to one product problem.

  • Larry learned about DeepMind after seeing its Atari Breakout system on a private plane with Musk and Luke Nosek. The system independently discovered a strategy of bouncing the ball up around the edges of the bricks and then along the top, and Larry immediately wanted to know who had built it.

  • Hassabis told the hosts that Larry simply “got it.” Google already had Brain working with product groups, so DeepMind could remain in London pursuing intelligence; Google also offered unmatched compute and an independent oversight board. The resulting January 2014 acquisition cost $550 million.

14. DeepMind paid back strategically before it produced a consumer product

  • One early internal application used neural networks to optimize data-center cooling. Google announced in July 2016 that DeepMind had reduced the energy required for cooling by 40%—a saving across Google’s infrastructure that the hosts believe could have justified the acquisition rapidly.

  • AlphaGo then demonstrated creative machine reasoning against world Go champion Lee Sedol. It won the first three games of a five-game match, and “move 37” initially appeared mistaken before revealing a strategy so inventive that human players began learning from the machine.

  • Go supplied the right test because a turn offers roughly 200 choices versus about 20 opening moves and 30–40 midgame choices in chess. Hassabis said that, as of 2017, even all the world’s computers running for a million years could not enumerate every variation, forcing AlphaGo to learn representations and strategies instead of brute-forcing the game.

15. OpenAI began as an explicit counterweight to Google’s talent concentration

  • Google’s DeepMind acquisition infuriated Musk, who had invested in the company and wanted its capabilities inside Tesla. In summer 2015, he and Sam Altman assembled leading researchers for a Rosewood Hotel dinner and asked what could lure them away from Google’s money, peers, academic freedom, and infrastructure.

  • Almost everyone answered “nothing.” Ilya Sutskever alone found the risk worth taking: “I felt like there were risks involved, but I also felt like it would be a very interesting thing to try”—a line the hosts call “the most Ilya quote of all time.”

  • Google reportedly countered with roughly double OpenAI’s offer, delivered personally by Jeff Dean, but Sutskever stayed committed. His decision supplied the activation energy for about seven researchers, Greg Brockman from Stripe, Altman, and Musk to form a nonprofit research lab.

  • OpenAI pledged to advance digital intelligence for humanity “unconstrained by a need to generate financial return.” Backers announced $1 billion of commitments from Musk, Altman, Reid Hoffman, Jessica Livingston, Thiel, and others, though only about $130 million was ultimately collected.

16. Early OpenAI copied DeepMind until its funding model broke

  • OpenAI initially behaved like a university or DeepMind-style lab: recruiting researchers, publishing, and building agents for Dota 2, a Rubik’s Cube, Universe, Atari titles, and open-world games. Emergent strategies were scientifically interesting, but no project clearly coalesced into a product or commercial engine.

  • Dario Amodei left Google Brain for OpenAI in early 2016 and, with Sutskever, led major research efforts before later founding Anthropic. Even with that talent, Musk increasingly saw a collection of experiments copying DeepMind rather than an AI program useful to Tesla or moving decisively toward its mission.

  • After Google published the transformer paper, Musk demanded either full control—potentially merging OpenAI into Tesla—or his complete departure with future funding. The board refused, and by early 2018 OpenAI had lost its primary financial backer just as scalable transformers made capital requirements explode.

17. Google’s GPU order signaled an industrial market before NVIDIA saw one

  • When Krizhevsky joined Google in 2013, he was astonished that machine-learning workloads still ran on CPUs. He bought a GPU machine locally, placed it in a nearby closet, attached it to Google’s network, and resumed training as he had in academia—now with Google paying for electricity.

  • In 2014 Jeff Dean and Alan Eustace proposed formally adding 40,000 NVIDIA GPUs to Google’s fleet for $130 million. Finance resisted, but Larry personally approved the order because “the future of Google is deep learning.”

  • The order was enormous relative to NVIDIA’s approximately $4 billion annual revenue and $10 billion market capitalization. To the hosts, it effectively revealed that neural networks were valuable enough in production for a sophisticated customer to invest nine figures, likely strengthening NVIDIA’s conviction to build aggressively for the market.

18. The TPU emerged when one feature threatened to require another Google

  • Neural-network speech recognition initially ran only on Nexus phones because Google lacked capacity for every Android device. Dean calculated that three minutes of daily use across one billion phones would require roughly twice Google’s existing data-center fleet: “We need another Google.”

  • Jonathan Ross’s FPGA work supplied a bridge to a custom application-specific integrated circuit optimized for matrix multiplication. The TPU traded generality for efficiency, using reduced numerical precision and software quantization so the same memory, transistors, and power could execute many more calculations.

  • The team designed, verified, fabricated, and deployed the first TPU in 15 months. FPGA prototypes validated the mathematics, while a hard-drive form factor let technicians remove a drive and insert a TPU without physically redesigning server racks—a quintessential Google infrastructure shortcut.

  • TPU V1 was “not great,” but later generations converged toward GPUs while adding capabilities. The hosts cite an estimated 2–3 million Google TPUs today versus roughly 4 million NVIDIA GPUs shipped last year, making TPUs a scale platform rather than an experimental internal chip.

19. The transformer solved memory by making attention parallel

  • Google’s neural translation overhaul initially relied on recurrent neural networks, then long short-term memory networks. LSTMs cut translation error by 60% in 2016 because they retained more context, but remained computationally intensive and poorly suited to the parallel hardware revolution Google had built.

  • Jakob Uszkoreit and colleagues explored broader “attention”: instead of processing only nearby words sequentially, let the model examine the entire passage when predicting each translated word. The hosts compare this to professional human translation, where understanding the whole source precedes rendering individual sentences.

  • The architecture was compute-heavy but extremely parallelizable. The group called it the “transformer” both because it transformed one information representation into another and because several members liked the children’s franchise.

  • Shazeer joined after the early implementation failed to beat LSTMs, rewrote the codebase from scratch, and returned saying, “It works now.” The transformer crushed the incumbent system, and results continued improving as the team enlarged the model—revealing a scalable architecture rather than a narrow translation trick.

20. Elegance and scaling laws displaced clever hand-built algorithms

  • Google Brain co-founder Greg Corrado emphasized how suspiciously simple the transformer looked: “Transformers are barely a neural network architecture.” Researchers often assumed something so elegant could not work, but Corrado came to view simplicity and resource efficiency as signals that a solution resembled what survives in nature.

  • The result anticipated Rich Sutton’s later “bitter lesson”: researchers prefer sophisticated domain-specific algorithms, yet across language, vision, and games, scalable architectures combined with more data and compute repeatedly win. The transformer became the cleanest mechanism yet for “more data, more energy, more compute, better results.”

  • The conceptual loop closed with Harik and Shazeer’s 2001 lunch. Seventeen years after discussing comprehension as compression, Shazeer co-authored a model that ingested enormous corpora, stored statistical structure in compressed parameters, and reconstructed contextually appropriate language.

21. Google used the transformer tactically instead of treating it as a platform reset

  • The eight researchers published “Attention Is All You Need” in 2017. As of 2025, the hosts cite more than 173,000 academic citations, making it the seventh-most-cited paper of the 21st century despite being substantially newer than those above it.

  • Google did not simply ignore the invention. Transformer models including BERT and MUM improved query comprehension and search quality, following Brain’s established playbook of feeding new AI capabilities into existing products. What Google failed to do was treat the architecture as a wholesale change in computing and user interface.

  • Shazeer argued that Google should consider replacing the search index and ten blue links with a giant transformer, exactly the cannibalizing move leadership could not justify. Between 2017 and ChatGPT, the company spent roughly five years improving its franchise rather than building the new one.

  • All eight paper authors eventually left to start or join AI companies. Shazeer founded Character.AI and returned only after Google’s approximately $2.7 billion licensing-and-hiring transaction—an expensive illustration of publishing the breakthrough, losing the talent, then buying some of it back.

22. OpenAI bet its entity on transformer scaling

  • In June 2018 OpenAI introduced GPT-1, applying transformer pre-training across large amounts of general internet text and then fine-tuning for specific tasks. Google’s BERT and an Allen Institute model emerged around the same period, but OpenAI treated the direction as an institution-defining bet.

  • Scaling immediately implied more data, compute, energy, and capital than the nonprofit could sustainably supply. The hosts leave open whether Musk’s departure caused the pivot or merely accelerated a direction already chosen; either way, financial necessity and the transformer arrived together.

  • Reid Hoffman connected Altman to Microsoft chief Satya Nadella. Discussions at the July 2018 Allen & Company conference produced Microsoft’s commitment to invest $1 billion in cash and Azure credits, plus an exclusive license to OpenAI’s technology for Microsoft products.

  • OpenAI created a captive for-profit limited partnership controlled by its nonprofit parent, allowing Microsoft to invest while preserving nominal mission control. The structure solved an immediate capital problem but originated the “for-profit nonprofit question mark” still being contested in 2025.

23. Microsoft supplied the cloud that an independent model lab could not build

  • Microsoft was attractive not merely because it had cash, but because Azure could supply the GPU infrastructure OpenAI lacked. A young research lab was not going to buy chips, secure power, construct data centers, and operate a global training fleet; it needed a hyperscaler.

  • The partnership revived Google’s historic rival in the one platform shift Google itself had created. Microsoft gained differentiated technology for Azure and its applications, while OpenAI gained the capital and compute required to scale without returning to Google.

  • GPT-2 arrived in 2019 as a promising API that could continue supplied prose but demanded developer skill and offered no consumer front door. GPT-3 in June 2020 was much more convincing—sometimes difficult to distinguish from human writing—but remained mostly infrastructure awaiting productization.

24. GitHub Copilot proved GPT could change a real workflow

  • In summer 2021 Microsoft launched GitHub Copilot using GPT-3, the first substantial productization of OpenAI technology. It turned a general model into an assistive interface embedded directly in software development rather than asking consumers to invent prompts for a raw API.

  • Adoption followed “slowly then all at once”: early engineers whispered that Copilot made them somewhat more efficient; later companies claimed AI generated the majority of their code. The episode treats this as a durable workflow transformation preceding the consumer chatbot explosion.

  • Microsoft subsequently invested another $2 billion. By the end of 2021, OpenAI had a scalable architecture, a cloud patron, and its first product proof—while Google’s market capitalization began falling from nearly $2 trillion toward $1 trillion amid rate increases and a broader risk-off collapse.

25. Google built chat before ChatGPT but could not responsibly ship it

  • Shazeer built Meena, an internal transformer chatbot arguably close to the eventual ChatGPT experience. It lacked post-training and reinforcement learning from human feedback, however, and could reportedly produce responses as unsafe as lists of people who “should die.”

  • The technical prototype collided with three constraints: direct answers threatened ad revenue, publisher disintermediation carried legal risk, and Google’s trusted brand could not tolerate an authoritative system that hallucinated. Even today, David says he verifies Claude’s answers through Google—the trust promise the chatbot might have damaged.

  • Google evolved Meena into LaMDA but still kept the conversational interface mostly internal. Shazeer left in 2021 after repeatedly advocating release, despite having authored the transformer and possessing an unusually strong internal record.

  • AI Test Kitchen exposed a constrained LaMDA chat experience in May 2022, before ChatGPT, but ended every conversation after five turns because longer interactions were more likely to derail. The caution was understandable after Microsoft’s Tay debacle, yet it prevented users from discovering the product’s full magic.

26. ChatGPT was an accidental consumer product with explosive proof of demand

  • OpenAI’s GPT-3.5 was useful but still lacked an intuitive interface. Altman reportedly asked for a chatbot, and within roughly a week someone made a conversational product by wrapping repeated API calls in a chat interface—an implementation so simple that the magnitude of the response surprised the company itself.

  • Anthropic may have accelerated the timing: OpenAI heard that Dario Amodei’s team was preparing what became Claude and wanted its own chat interface out first. Neither lab appears to have forecast the category’s immediate consumer scale.

  • ChatGPT launched as a “research preview” on November 30, 2022. It reached 1 million users in less than a week, 30 million by December 31, and 100 million registered users by the end of January 2023—the fastest product ever to cross that threshold.

  • Servers buckled, Microsoft and OpenAI scrambled for additional Azure capacity and financing, and OpenAI erected a quick paywall partly to control costly demand. A company forecasting B2B API licensing discovered a direct consumer business almost by accident: “the accidental consumer tech company.”

27. ChatGPT changed AI from Google’s advantage into Google’s emergency

  • Before November 2022, Google treated AI as a sustaining innovation: better recommendations, ads, translation, Gmail, Photos, and search quality favored the incumbent with the most capital and data. ChatGPT revealed a superior interface for many jobs performed by search and turned those same assets into castles requiring protection.

  • Sundar issued a company-wide “code red” in December, demanding native AI products at speed. The proper disruptive-response playbook, as the hosts describe it, was to ship a comparable new product while measuring where it complemented search and where it cannibalized the franchise.

  • Microsoft sharpened the threat with another $10 billion investment, saying it now owned 49% of OpenAI’s for-profit entity, and February 2023’s AI-powered Bing. Satya declared “a new day for search,” adding, “I want people to know that we made Google dance.”

  • Ben calls this the worst possible scenario for Google: its oldest enemy could leapfrog ten-blue-link search with genuinely differentiated technology rather than another imitation. Yet the new Bing launch itself “kind of misses,” giving Google time to organize a stronger response.

28. Bard’s failure forced Google to rebuild both model and organization

  • Google rapidly rebranded its LaMDA chatbot as Bard and shipped it in February 2023. A choreographed launch demonstration contained a factual error, the product felt clearly inferior to ChatGPT, and Alphabet shares fell 8% that day.

  • The hosts identify missing post-training and RLHF as the experiential gap: Bard lacked ChatGPT’s tone, appropriateness, and apparent discipline. Google replaced LaMDA with Brain’s PaLM model in May, but it remained behind GPT-3.5 while OpenAI released GPT-4 in March.

  • Sundar then made the organizational decision previous DeepMind agreements had resisted: merge Google Brain and DeepMind into Google DeepMind, appoint Hassabis to lead the combined AI organization, and subordinate separate charters and cultural friction to a single company-wide emergency.

  • The hosts read the transaction as DeepMind absorbing Brain’s responsibility, not a neutral federation. Reinforcement-learning expertise and frontier-model development moved under Hassabis, while Jeff Dean, Oriol Vinyals, and later the returning Shazeer joined the unified push.

29. Gemini became both Google’s common model and its rallying product

  • Sundar’s second decree was “one model for everything”: text, images, audio, and video, serving every internal product and external AI surface. Centralization reflected scaling economics—duplicating frontier training across multiple teams would waste even Google’s resources and fragment the data flywheel.

  • Google announced Gemini at I/O in May 2023 and offered early public access in December. Gemini 1.5 followed in February 2024 with a 1 million-token context window; Gemini 2.0 arrived in February 2025, and Gemini 2.5 Pro entered experimental mode in March before general availability in June.

  • Search simultaneously gained AI Overviews, initially through Labs and later broadly, plus AI Mode in March 2025. Google can run inference across an unfathomable number of queries, but deliberately exposes different experiences by query and user rather than redirecting google.com wholesale to Gemini.

  • Renaming both the model and consumer application Gemini mattered culturally. Ben’s reading is that Google was declaring the product to be the technology itself—like Gmail’s original appeal of speed, search, and storage—not a heavily designed social experience layered over the model.

30. Google has recovered frontier velocity without yet proving demand quality

  • Beyond the core chatbot, Google launched NotebookLM, AI-generated podcasting, Veo 3 video, Nano Banana imagery, Flow, and Genie 3’s prompt-driven world building. These categories exploit Google’s research and YouTube assets without directly cannibalizing lucrative search queries.

  • Google reports 450 million monthly Gemini users, a remarkable rise from zero, but the hosts question the definition. The total may include Nano Banana, AI Overviews, AI Mode, or incidental model interactions, echoing Meta’s inflated framing of users who accidentally touched Meta AI.

  • Whatever the precise engagement quality, Google moved from a disastrous Bard launch to a competitive model family and rapid release cadence while revenue reached record levels. David sees a familiar capability: Google navigated mobile successfully and may again be learning to absorb a platform shift.

31. Waymo shows why AI timelines are impossible to forecast

  • The lineage begins with DARPA’s 2004 Grand Challenge: a 132-mile autonomous desert race with a $1 million prize, no human intervention, and zero finishers among roughly 100 registered teams. A year later, 22 of 23 finalists passed the prior best distance and five completed the course.

  • Sebastian Thrun’s Stanford team won using nearly unmodified commodity hardware on a Volkswagen rather than Carnegie Mellon’s extensively rebuilt vehicles. Their principle was that every custom component could fail, so they accepted noisy sensors and performed the innovation in software.

  • A real-time machine-learning system combined precise short-range laser data with a wide, color camera view. The camera could see toward the horizon and identify the safe path, letting the vehicle anticipate turns and drive faster.

  • Larry later challenged Thrun to identify a technical reason self-driving was impossible. After sleeping on it, Thrun answered, “I realized what it was. I’m just afraid,” and began Project Chauffeur, Google X’s first project, in 2009.

32. Waymo’s second 99% took more than a decade

  • Larry defined the “Larry 1000,” about 1,000 difficult California miles including Tahoe, Lombard Street, Highway 1, and the Bay Bridge. A small team completed the benchmark within 18 months, proving feasibility far faster than it could prove a safe, commercial product.

  • The project used no deep learning for its first five years. Convolutional networks improved object perception around 2013; transformer-derived techniques later aided prediction and planning; Waymo became an Alphabet subsidiary in 2016 and raised $3.2 billion externally in March 2020.

  • The first public commercial service without a human driver launched in Phoenix in October 2020, eleven years after the project began. Subsequent rounds included $2.5 billion and $5.6 billion, while San Francisco turned the service from a demonstration into a daily transportation habit.

  • Sebastian originally preferred highway assistance, Eric Schmidt proposed buying Tesla for roughly $3 billion, and Larry favored robotaxis. The chosen path was hardest operationally, but it also produced a distinct experience: private conversations, children’s car seats, pets, and phone calls without negotiating with a driver.

33. Waymo may be a Google-sized business disguised as an expensive fleet

  • Waymo operates in Phoenix, San Francisco, Los Angeles, Austin, and Atlanta, with Tokyo planned. The hosts cite hundreds of thousands of paid weekly rides, more than 100 million fully driverless miles growing by 2 million weekly, over 10 million paid trips, and approximately 2,000 vehicles.

  • Its hardware stack includes 13 cameras, four lidar units, six radars, and external microphones. That is costlier than Tesla’s camera-only approach, but Waymo argues multiple sensing modalities are necessary to satisfy the safety and regulatory bar for true autonomy.

  • A recent Waymo study reported 91% fewer crashes involving serious injury or worse than comparable human driving. Against more than 40,000 annual U.S. road deaths—about 120 daily—the hosts ask why a demonstrated order-of-magnitude safety improvement is not discussed constantly.

  • The CDC estimated that U.S. crash deaths in 2022 resulted in $470 billion of total costs. A tenfold reduction implies more than $420 billion of potential annual savings, before counting new mobility and experience value; against that opportunity, Waymo’s estimated cumulative $10–15 billion burn now looks “really, really smart.”

34. AI competition helped preserve Google’s search monopoly

  • Since the prior Acquired episode, a federal judge found Google to be a monopoly in internet search but imposed no remedy requiring a Chrome divestiture or an end to tens of billions in default-distribution payments to Apple and others.

  • AI competition contributed to the restraint: OpenAI, Anthropic, and Perplexity had attracted tens of billions of funding, persuading the court that the market might produce viable challengers without weakening Google during the new race.

  • Ben finds that logic questionable because none of those rivals yet produces net income; their competitive capacity depends on investors continuing to finance enormous losses. The episode’s darkly comic domino chain is that Sutskever’s departure created OpenAI, whose existence then helped prevent Google’s breakup.

35. Google can fund AI offense while returning capital

  • Over the latest twelve months, Google generated approximately $370 billion in revenue and $140 billion in earnings. The hosts call it the most profitable technology company and second globally only to Saudi Aramco: “Let’s not forget Google is the best business ever.”

  • Alphabet’s market capitalization crossed $3 trillion after falling toward $1 trillion in 2022, placing it behind NVIDIA, Microsoft, and Apple. Cash and marketable securities declined from about $140 billion in 2021 to $95 billion as Google funded AI data centers, repurchased shares, and initiated dividends.

  • The paradox strengthens the bull case: Google says it intends to win “the most capital intensive race in business history,” yet its search franchise produces more cash than management needs for capex plus a safety cushion. Independent model laboratories possess no comparable self-funding mechanism.

  • Google also has more than 150 million Google One subscribers, growing nearly 50% year over year. Most remain on cheaper storage tiers, while premium AI begins around $20 monthly, but YouTube Premium, YouTube Music, storage, Play Store benefits, and even sports could support a scaled AI bundle.

36. Google Cloud became AI distribution after learning enterprise sales

  • Google App Engine launched in 2008 as an opinionated platform-as-a-service requiring prescribed languages, SDKs, and deployment patterns. AWS’s infrastructure-as-a-service model proved more useful, and Google did not launch Compute Engine—the foundation of Google Cloud Platform—until 2012.

  • The early business also withheld Google’s internal crown jewels and lacked an enterprise organization. Revenue was only about $4 billion in 2017, while the go-to-market team numbered roughly 150 and sat largely in California rather than near global customers.

  • Kubernetes and multicloud portability gave the third-place provider a counterposition, then former Oracle president Thomas Kurian arrived in late 2018 and added roughly 10,000 go-to-market employees. Revenue exceeded $13 billion in 2020, reached $26 billion in 2022, and the division became profitable in 2023.

  • Today Google Cloud exceeds a $50 billion annual run rate, grows about 30%, and is the fastest-growing major hyperscaler cited. AI workloads supply a tailwind, while abundant TPUs give customers an alternative to fighting for NVIDIA allocations—provided they accept Google’s stack rather than CUDA.

37. Google is the only company holding every major AI pillar

  • The hosts divide the market into models, chips, cloud infrastructure, and scaled applications. NVIDIA primarily has chips; Microsoft and Amazon primarily have clouds; Meta has applications; AMD has chips; OpenAI and Anthropic have models; Apple, in their blunt assessment, has “nothing.”

  • Google possesses all four: Gemini, TPUs, Google Cloud, and applications spanning Search, YouTube, Gmail, Maps, Docs, Chrome, and Android. AI Overviews alone create model usage at a scale a startup cannot manufacture, while Cloud makes TPUs accessible outside Google’s internal fleet.

  • A cloud business is essential to the chip strategy because Amazon and Microsoft were unlikely to adopt an unproven Google accelerator voluntarily. Google Cloud created initial demand and developer access; possible future TPU availability through neoclouds could broaden the ecosystem without turning Google into a conventional merchant-chip seller.

  • The likely objective is not NVIDIA-style hardware margin but CUDA-style ecosystem gravity. If users can find TPUs wherever workloads already run, Google can improve utilization, tooling, and model economics while reducing dependence on a supplier extracting scarcity pricing.

38. Distribution, data, and YouTube make the bull case unusually broad

  • Google still distributes products to nearly every internet user at the moment of intent. AI Overviews and AI Mode demonstrate that it can funnel search demand into new experiences, while Gemini has reached product quality the hosts consider broadly comparable with OpenAI and Anthropic.

  • Personalized data across Gmail, Maps, Docs, Chrome, and Android could create an assistant no independent lab can reproduce. Switching costs are currently low, but they rise sharply once an AI understands communications, schedules, documents, preferences, and history across the user’s digital life.

  • YouTube adds the internet’s unique scale corpus of long- and short-form user-generated video, plus the second-largest search destination and Google’s private high-bandwidth network. Models could identify every product in every video, make it instantly shoppable, generate new content, and monetize engagement with Google’s existing advertising machinery.

  • The company retains a deep research bench and is willing to spend billions to restore exceptional people. With Dean and Shazeer serving as Gemini’s co-technical leads, Ben’s response is simple: “You got Jeff Dean working on it, I’m in.”

39. Owning the accelerator could make Google the lowest-cost token producer

  • NVIDIA’s roughly 75–80% gross margins imply buyers pay approximately a 4x–5x markup over production cost. Google still pays partner Broadcom, reportedly around a 50% margin, but that resembles a twofold markup—a major difference when accelerators dominate total cost.

  • The hosts cite estimates that chips and depreciation represent more than half of AI data-center cost, research and software labor roughly 25–33%, and power only about 2–6%. Fast obsolescence makes five-year chip depreciation optimistic: NVIDIA itself quickly shifted enthusiasm from Hopper to Blackwell.

  • AI companies may achieve only about 50% gross margins rather than software’s familiar 80%. In that world, low-cost production matters far more than in prior technology waves; Google’s own chips, data centers, network, and utilization could determine who can vend tokens sustainably.

  • Token volume illustrates the scale: Google reportedly processed 10 trillion tokens across its products in April 2024, almost 500 trillion in April 2025, and 980 trillion by June. Training cost amortized over that inference base creates a structural advantage independent laboratories cannot easily match.

40. AI may eventually monetize more intent than search ever observed

  • Traditional searches average two or three words; AI prompts often exceed 20. Bill Gross’s argument is that richer language reveals far more precise intent, so a future advertising system should know “the crap out of what that user wants” and price relevant placements accordingly.

  • AI also moves offline or nonexistent interactions into measurable digital sessions: complex planning, explanation, creation, and extended time spent. If the interface supports commercial participation without destroying trust, the monetizable pie could become larger than search rather than merely cannibalizing it.

  • Waymo supplies additional upside outside the search-replacement frame, potentially expanding ride-hailing into private mobility, personally owned autonomy, transportation for blind and elderly people, and trucking. The hosts exclude speculative AGI from normal valuation analysis but acknowledge it as the “galaxy brain” upside.

41. The bear case starts with weaker monetization and fragmented share

  • Google’s current AI products have not demonstrated an ad shape comparable with search. The hosts estimate Google earns roughly $400 per U.S. user annually from a free product; only a thin segment will willingly pay approximately that amount each year for an AI subscription.

  • Search launched in 1998 and found AdWords within two years, while the current AI wave still lacks a comparably obvious value-capture system. High value creation does not guarantee that model providers, rather than users, applications, chipmakers, or clouds, retain the economics.

  • Google owns roughly 90% of search but may settle at only 25–50% of an AI market shared with OpenAI, Anthropic, Perplexity, xAI, Meta, and others. Even equivalent monetization per user would produce substantially less market power.

  • The first queries moving to chat may be the most lucrative: travel planning that once generated Expedia advertising, or health questions associated with expensive legal and medical leads. Financial erosion could begin beneath record headline revenue before becoming visible in consolidated results.

42. Incumbency removes the goodwill Google enjoyed during earlier transitions

  • Google’s chatbot began as “immediately obviously inferior” and improved toward parity, whereas 1998 search was immediately superior. The company is now defending an entrenched franchise instead of arriving as the beloved underdog with a radically better product.

  • Public and regulatory sentiment toward all large technology companies has deteriorated, but the hosts believe startups still receive more goodwill than Google. OpenAI and Anthropic had to behave like consequential institutions unusually early, yet they retain more of the challenger narrative.

  • The central risk is not technical incapacity; it is hesitation produced by profitability, publishers, trust, and shareholders. If AI Mode better fulfills Google’s mission, why is the company not moving much more aggressively? Because every percentage point of migration can affect one of history’s greatest businesses.

43. Google’s strongest AI powers are scale, brand, and cornered distribution

  • In Hamilton Helmer’s framework, scale economies dominate: Google amortizes training, hardware, networking, and software over search-scale inference, while high utilization improves the economics of every TPU and data center.

  • Google search functions as a cornered distribution resource, and the broader product portfolio creates future switching costs once Gemini becomes personalized. Brand cuts both ways, but for mainstream users, “I trust Google” may outweigh resentment of the incumbent.

  • Network effects are weak today because another Gemini user does not directly improve one person’s experience. Google also lacks counterpositioning—it is being counterpositioned—and the hosts do not yet see process power that reliably produces breakthroughs unavailable to peer laboratories.

  • That thinner power profile is revealing. Search appeared to possess nearly every durable advantage; AI currently gives Google several formidable ones, but not the overwhelming bundle that once entitled it to persistent 90% share.

44. Google’s outcome rests on whether it can remain rapid without becoming rash

  • Ben’s quintessence is “the most fascinating example of the innovator’s dilemma ever.” Larry and Sergey have said they would rather go bankrupt than lose in AI, but the real test arrives only if advancing Google’s information mission requires accepting a permanently worse business than search.

  • David argues that, despite its flat-footed opening, Google may now be threading the needle better than any other incumbent. It unified rival labs, standardized on one model, restored key talent, shipped rapidly, and used selective integration to protect the franchise rather than making an impulsive all-or-nothing switch.

  • The unresolved possibility is that search’s foundation is eroding even while financial statements remain superb. Record revenue can coexist temporarily with users moving their most valuable workflows elsewhere, particularly when Google itself is subsidizing the transition through expensive inference.

  • For now, the hosts credit Sundar with being “rapid but not rash.” Stewarding both a mission and a public-company profit engine is an unusually difficult dual mandate; whether Google can keep both will make this one of business history’s defining platform-transition cases.

Ben Gilbert

I went and looked at a studio—or a little office that I was going to turn into a studio—nearby, but it was not good at all. It had drop ceilings, so I could hear the guy in the office next to me. You would have been able to hear him talking on episodes.

David Rosenthal

Third co-host.

Ben Gilbert

Is it Howard?

David Rosenthal

No, it was like a lawyer. It seemed to be like talking through some horrible problem that I didn't want to listen to, but I could hear every word.

Ben Gilbert

Does he want millions of people listening to this conversation?

David Rosenthal

Right.

David Rosenthal

All right.

Ben Gilbert

All right. Let's do a podcast. Let's do a podcast.

Welcome to the fall 2025 season of Acquired, the podcast about great companies and the stories and playbooks behind them. I'm Ben Gilbert.

David Rosenthal

I'm David Rosenthal, and we are your hosts. Here's a dilemma. Imagine you have a profitable business. You make giant margins on every single unit you sell, and the market you compete in is also giant—one of the largest in the world, you might say. But on top of that, lucky for you, you are also a monopoly in that giant market, with 90% share and a lot of lock-in.

David Rosenthal

And when you say monopoly, you mean monopoly as defined by the U.S. government. That is correct. But then imagine this: In your research lab, your brilliant scientists come up with an invention. This particular invention, when combined with a whole bunch of your old inventions by all your other brilliant scientists, turns out to create a product that is much better for most purposes than your current product. So you launch the new product based on this new invention, right?

Ben Gilbert

Right. I mean, especially because, out of pure benevolence, your scientists had published research papers about how awesome the new invention is, and lots of the inventions before it, too. Now there are new startup competitors quickly commercializing that invention. So, of course, David, you change your whole product to be based on the new thing, right?

David Rosenthal

This sounds like a movie.

Ben Gilbert

Yes. But here is the problem: You haven't figured out how to make this new, incredible product anywhere near as profitable as your old, giant cash-printing business. So maybe you shouldn't launch that new product. David, this sounds like quite the dilemma to me.

Of course, listeners, this is Google today. In perhaps the most classic textbook case of the innovator's dilemma ever, the entire AI revolution that we are in right now is predicated on the invention of the Transformer by the Google Brain team in 2017. Think OpenAI and ChatGPT, Anthropic, NVIDIA hitting all-time highs—all the craziness right now depends on that one research paper published by Google in 2017.

And consider this: Not only did Google have the densest concentration of AI talent in the world 10 years ago that led to this breakthrough, but today they have just about the best collection of assets that you could possibly ask for. They've got a top-tier AI model with Gemini. They don't rely on some public cloud to host their model; they have their own in Google Cloud, which now does $50 billion in revenue. That is real scale.

They're a chip company with their Tensor Processing Units, or TPUs, which is the only real-scale deployment of AI chips in the world besides NVIDIA GPUs. Maybe AMD, maybe, but these are definitely the top 2. Somebody put it to me in research that if you don't have a foundational frontier model or you don't have an AI chip, you might just be a commodity in the AI market. Google is the only company that has both.

Google still has a crazy bench of talent. And despite ChatGPT becoming kind of the Kleenex of the era, Google does still own the textbox—the single one that is the front door to the internet for the vast majority of people anytime anyone has intent to do anything online. But the question remains: What should Google do strategically? Should they risk it all and lean into their birthright to win in artificial intelligence, or will protecting their gobs of profits from search hamstring them as the AI wave passes them by?

But perhaps first we must answer the question: How did Google get here? So, listeners, today we tell the story of Google, the AI company.

David Rosenthal

Woo.

Ben Gilbert

You like that, David? Was that good?

David Rosenthal

I love it. Did you hire a Hollywood screenwriting consultant without telling me?

Ben Gilbert

I wrote that 100% myself with no AI. Thank you very much.

David Rosenthal

No AI.

Ben Gilbert

David, Google, the AI company.

David Rosenthal

So, Ben, as you were alluding to in that fantastic intro—you’re really upping your game here—if we rewind 10 years from today, before the Transformer paper comes out, all of the following people, as we've talked about before, were Google employees.

Ilya Sutskever, founding chief scientist of OpenAI, who, along with Geoffrey Hinton and Alex Krizhevsky, had done the seminal AI work on AlexNet and published it a few years before. All 3 of them were Google employees, as was Dario Amodei, the founder of Anthropic; Andrej Karpathy, chief scientist at Tesla until recently; Andrew Ng; Sebastian Thrun; Noam Shazeer; and all the DeepMind folks—Demis Hassabis, Shane Legg, and Mustafa Suleyman. Mustafa, in addition to having been a founder of DeepMind in the past, now runs AI at Microsoft.

Basically, every single person of note in AI worked at Google, with the 1 exception of Yann LeCun, who worked at Facebook.

Ben Gilbert

Yeah, it's pretty difficult to trace a big AI lab now and not find Google in its origin story. The analogy here is that it's almost as if, at the dawn of the computer era itself, a single company like IBM had hired every single person who knew how to code. It would be like, if anybody else wanted to write a computer program, they would hear, “Sorry, you can't do that. Anybody who knows how to program works at IBM.”

This is how it was with AI and Google in the mid-2010s. But learning how to program a computer wasn't so hard that people out there couldn't learn how to do it. Learning how to be an AI researcher was significantly more difficult.

David Rosenthal

Right. It was the stuff of very specific Ph.D. programs with a very limited set of advisers and a lot of infighting in the field over where the direction of the field was going, what was legitimate versus what was crazy, heretical, religious stuff.

Ben Gilbert

Yeah. So then, yes, the question is: How do we get to this point?

David Rosenthal

Well, it goes back to the start of the company. Larry Page always thought of Google as an artificial intelligence company. In fact, Larry Page's dad was a computer science professor and had done his Ph.D. at the University of Michigan in machine learning and artificial intelligence, which was not a popular field in computer science back then.

Ben Gilbert

Yeah. In fact, a lot of people thought specializing in AI was a waste of time because so many of the big theories from 30 years prior had been kind of disproven at that point, or at least people thought they were disproven. So it was frankly contrarian for Larry's dad to spend his life and career and research work in AI.

David Rosenthal

And that rubbed off on Larry. If you squint, PageRank—the PageRank algorithm that Google was founded upon—is a statistical method. You could classify it as part of AI within computer science.

Larry, of course, was always dreaming much, much bigger. There's the quote we've said before on this show, from the year 2000, 2 years after Google's founding, when Larry says, “Artificial intelligence would be the ultimate version of Google. If we had the ultimate search engine, it would understand everything on the web. It would understand exactly what you wanted, and it would give you the right thing. That's obviously artificial intelligence. We're nowhere near doing that now. However, we can get incrementally closer, and that is basically what we work on here.”

It's always been an AI company.

Ben Gilbert

Yep. And that was in 2000. Well, one day in either late 2000 or early 2001—the timelines are a bit hazy here—a Google engineer named Georges Harik was talking over lunch with Ben Gomes, a famous Google engineer who I think would go on to lead Search, and a relatively new engineering hire named Noam Shazeer.

Georges was one of Google's first 10 employees, an incredible engineer. And just like Larry Page's dad, he had a Ph.D. in machine learning from the University of Michigan. Even when Georges went there, it was still a relatively rare, contrarian subfield within computer science.

So, the three of them are having lunch, and Georges Harik says offhandedly to the group that he has a theory from his time as a PhD student that compressing data is technically equivalent to understanding it. The thought process is that if you can take a given piece of information, make it smaller, store it away, and then later reinstantiate it in its original form, the only way you could possibly do that is if whatever force is acting on the data actually understands what it means. You’re losing information going down to something smaller and then recreating the original thing.

It’s like you’re a kid in school. You learn something in school, you read a long textbook, and you store the information in your memory. Then you take a test to see if you really understood the material. If you can recreate the concepts, then you really understand it.

David Rosenthal

Which foreshadows big LLMs today: compressing the entire world’s knowledge into some number of terabytes that’s just this smashed-down little vector set—little, at least, compared to all the information in the world. But it’s kind of that idea, right? You can store all the world’s information in an AI model in something that’s incomprehensible and hard to understand. But then, if you uncompress it, you can bring knowledge back to its original form.

Ben Gilbert

Yep. And these models demonstrate understanding, right?

David Rosenthal

Do they? That’s the question. That’s the question. They certainly mimic understanding.

Ben Gilbert

So, this conversation is happening 25 years ago. Noam, the new hire, the young buck, sort of stops in his tracks, and he’s like, “Wow, if that’s true, that’s really profound.”

David Rosenthal

Is this in one of Google’s microkitchens?

Ben Gilbert

This is in one of Google’s microkitchens. They’re having lunch.

David Rosenthal

Where did you find this, by the way?

Ben Gilbert

This is in In the Plex. This is a small little passage in Steven Levy’s great book that’s been a source for all of our Google episodes, In the Plex. There’s a small little throwaway passage in here about this because the book came out before ChatGPT and AI and all that.

So Noam kind of latches on to Gor and keeps vibing over this idea. Over the next couple of months, the two of them decide, in the most googly fashion possible, that they’re just going to stop working on everything else and go work on this idea: language models and compressing data, and whether they can generate machine understanding with data. If they can do that, that would be good for Google.

I think this coincides with that period in 2001 when Larry Page fired all the managers in the engineering organization, so everybody was just doing whatever they wanted to do.

David Rosenthal

Funny.

Ben Gilbert

So, there’s this great quote from Gor in the book:

“A large number of people thought it was a really bad thing for Noam and me to spend our talents on, but Sanjay Ghemawat—Sanjay, of course, being Jeff Dean’s famous, prolific coding partner—thought it was cool.”

So Gor would posit the following argument to any doubters they came across: “Sanjay thinks it’s a good idea, and no one in the world is as smart as Sanjay. So why should Noam and I accept your view that it’s a bad idea?”

David Rosenthal

It’s like, if you beat the best team in football, are you the new best team in football no matter what?

Ben Gilbert

Yeah. So all of this ends up taking Noam and Gor deep down the rabbit hole of probabilistic models for natural language. For any given sequence of words that appears on the internet, what is the probability for another specific sequence of words to follow? This should sound pretty familiar to anybody who knows how LLMs work today.

David Rosenthal

Oh, kind of like a next-word predictor.

Ben Gilbert

Yeah. Or a next-token predictor, if you generalized it.

David Rosenthal

Yep.

Ben Gilbert

So, the first thing that they do with this work is create the “Did you mean?” spelling correction in Google Search.

David Rosenthal

Oh, that came out of this. That came out of this. Noam created this.

Ben Gilbert

So this is huge for Google because it’s obviously a bad user experience when you mistype a query and then need to type another one. But it’s a tax on Google’s infrastructure, because every time these mistyped queries come in, Google’s infrastructure serves results for that query that are useless and immediately overwritten with the new one.

David Rosenthal

Right? And it’s a really tightly scoped problem where you can see, “Oh, wow, 80% of the time that someone types in ‘god groomer,’ they actually mean ‘dog groomer,’ and they retype it.” If it’s really high confidence, then you actually just correct it without even asking them, and then ask them if they want to opt out instead of opting in. It’s a great feature, and it’s a great first use case for this in a very narrowly scoped domain.

Ben Gilbert

Totally. So they get this win, and they keep working on it—Noam and Gor—and they end up creating a fairly large—I’m using “large” in quotes here—language model that they affectionately call Phil: the Probabilistic Hierarchical Inferential Learner.

David Rosenthal

These AI researchers love creating their backronyms.

Ben Gilbert

They love their word puns.

David Rosenthal

Yeah.

Ben Gilbert

Yep. So, fast-forward to 2003, and Susan Wojcicki and Jeff Dean are getting ready to launch AdSense. They need a way to understand the content of these third-party web pages—the publishers—in order to run the Google ad corpus against them. Well, Phil is the tool that they use to do it.

David Rosenthal

Huh. I had no idea that language models were involved in this.

Ben Gilbert

Yeah. So Jeff Dean borrows Phil and famously uses it to code up his implementation of AdSense in a week, because he’s Jeff Dean. And boom: AdSense. This is billions of dollars of new revenue to Google overnight because it’s the same corpus of ads—AdWords, the search ads—that they’re now serving on third-party pages. They just massively expanded the inventory for the ads that they already have in the system, thanks to Phil. Thanks to Phil.

All right, this is a moment where we’ve got to stop and just give some Jeff Dean facts. Jeff Dean is going to be the throughline of this episode of, “Wait, how did Google pull that off? How did Jeff Dean just go home and, over the weekend, rewrite some entire giant distributed system and figure out all of Google’s problems?”

Back when Chuck Norris facts were big, Jeff Dean facts became a thing internally at Google. I just want to give you some of my favorites. The speed of light in a vacuum used to be about 35 miles per hour. Then Jeff Dean spent a weekend optimizing physics.

David Rosenthal

So good.

Ben Gilbert

Jeff Dean’s PIN is the last 4 digits of pi.

David Rosenthal

Only Googlers would come up with these.

Ben Gilbert

Yes. To Jeff Dean, NP means “no problemmo.”

David Rosenthal

Oh yeah, I’ve seen that one before. I think that one’s my favorite.

Ben Gilbert

Yes.

David Rosenthal

Oh, man. So, so good. Also, a wonderful human being, who we spoke to in Research and was very, very helpful. Thank you, Jeff.

Ben Gilbert

Yes. So, language models definitely work, definitely going to drive a lot of value for Google, and they also fit pretty beautifully into Google’s mission to organize the world’s information and make it universally accessible and useful. If you can understand the world’s information and compress it and then recreate it, yeah, that fits the mission. I think that checks the box.

Absolutely. So Phil gets so big that apparently, by the mid-2000s, Phil is using 15% of Google’s entire data center infrastructure. I assume a lot of that is AdSense ad serving, but also “Did you mean?” and all the other stuff that they start using it for within Google.

David Rosenthal

So, early natural-language systems: computationally expensive.

Ben Gilbert

Yes. So, okay, now mid-2000s. Fast-forward to 2007, which is a very, very big year for the purposes of our story. Google had just recently launched the Google Translate product. This is the era of all the great products coming out of Google that we’ve talked about: Maps and Gmail and Docs and all the wonderful things. Chrome and Android are going to come later.

They had a 10-year run where they basically launched everything you know of at Google except for Search, truly in a 10-year run. And then there were about 10 years after that, from 2013 onward, where they basically didn’t launch any new products that you’ve heard about until we get to Gemini, which is this fascinating thing. But this 2003-to-2013 era was just so rich, with hit after hit after hit.

David Rosenthal

Magical.

Ben Gilbert

And so one of those products was Google Translate. Not the same level of user base or perhaps impact on the world as Gmail or Maps or whatnot, but still a magical, magical product. The chief architect for Google Translate was another incredible machine-learning PhD named Franz Och. Franz had a background in natural-language processing and machine learning, and that was his PhD. He was German. He got his PhD in Germany.

At the time, DARPA—

David Rosenthal

The Defense Advanced Research Projects Agency, a division of the government—

Ben Gilbert

—had one of its famous challenges going for machine translation. So Google and Franz, of course, enter this, and Franz builds an even larger language model that blows away the competition in that year’s version of the DARPA challenge. This is either 2006 or 2007, and he gets an astronomically high BLEU score for the time.

It’s called the Bilingual Evaluation Understudy. It’s the algorithmic benchmark for judging the quality of translations at the time, and Franz’s score is higher than anything else possible. So Jeff Dean hears about this and the work that Franz and the Translate team have done, and he’s like, “This is great. This is amazing. When are you guys going to ship this in production?”

David Rosenthal

Oh, I heard this story.

Ben Gilbert

So Jeff and Noam talk about this on the Dwarkesh podcast.

David Rosenthal

Yes, that episode is so, so good.

Ben Gilbert

And Franz is like, “No, no, no, no, Jeff, you don’t understand. This is research.”

David Rosenthal

This isn't for the product. We can't ship this model that we built. This is an N-gram language model. N-grams are like the number of words in a cluster. We've trained it on a corpus of 2 trillion words from the Google search index.

This thing is so large that it takes 12 hours to translate a sentence. So the way the DARPA challenge worked in this case was, you got a set of sentences on Monday, and then you had to submit your machine translation of that set of sentences by Friday.

Ben Gilbert

Plenty of time for the servers to run.

David Rosenthal

Yeah. They were like, “Okay, so we have whatever number of hours it is from Monday to Friday. Let's use as much compute as we can to translate these couple of sentences. Hey, learn the rules of the game and use them to your advantage.”

Ben Gilbert

Exactly. So Jeff Dean, being the engineering equivalent of Chuck Norris, is like, “Let me see your code.” Jeff goes and parachutes in and works with the Translate team for a few months. He rearchitects the algorithm to run on the words and the sentences in parallel instead of sequentially.

When you're translating a set of sentences or a set of words in a sentence, you don't necessarily need to do it in order. You can break up the problem into different pieces, work on them independently. You can parallelize it.

David Rosenthal

And you won't get a perfect translation, but imagine you just translate every single word. You can at least translate those all at the same time in parallel, reassemble the sentence, and mostly understand what the initial meaning was.

Ben Gilbert

Yeah. And as Jeff knows very well, because he and Sanjay basically built it, Google's infrastructure is extremely parallelizable and distributed. You can break up workloads into little chunks, send them all over the various data centers that Google has, reassemble the results, and return that to the user.

David Rosenthal

They are the single best company in the world at parallelizing workloads across CPUs, across multiple data centers.

Ben Gilbert

CPUs. We're still talking CPUs here.

David Rosenthal

Yep. And Jeff's work with the team gets that average sentence translation time down from 12 hours to 100 milliseconds. So then they ship it in Google Translate, and it's amazing.

Ben Gilbert

This sounds like a Jeff Dean fact. It used to take 12 hours, and then Jeff took a few months with it. Now it's 100 milliseconds.

David Rosenthal

Right. So this is the first large—I’m using “large” in quotes here—language model used in production in a product at Google. They see how well this works and think, “Maybe we could use this for other things, like predicting search queries as you type. That might be interesting.”

Of course, the crown jewel of Google's business might also be an interesting application for this. The ad quality score for AdWords is literally the predicted click-through rate on a given set of ad copy. You can see how an LLM that's really good at ingesting information, understanding it, and predicting things based on that might be really useful for calculating ad quality for Google.

Ben Gilbert

Yep. Which is a direct translation to Google's bottom line.

David Rosenthal

Indeed. So, obviously, all of that is great on the language model front. I said 2007 was a big year. Also in 2007 begins the momentous intersection of several computer science professors on the Google campus.

In April of 2007, Larry Page hires Sebastian Thrun from Stanford to come to Google and work first part-time and then full-time on machine learning applications. Sebastian was the head of SAIL at Stanford, the Stanford Artificial Intelligence Laboratory—a legendary AI laboratory that was big in the first wave of AI back in the 1960s and 1970s, when Larry's dad was active in the field. It then actually shut down for a while before being restarted and re-energized in the early 2000s.

Ben Gilbert

Funny story about Sebastian: the way that he actually comes to Google—Sebastian was kind enough to speak with us to prep for this episode. I didn't realize it was basically an acquihire. He and some—I think it was grad students—were in the process of starting a company and had term sheets from Benchmark and Sequoia.

David Rosenthal

Yes.

Ben Gilbert

And Larry came over and said, “What if we just acquire your company before it's even started, in the form of signing bonuses?”

David Rosenthal

Yes. Probably a very good decision on their part. So SAIL, this group within the CS department at Stanford, not only had some of the most incredible, most accomplished professors and PhD AI researchers in the world, but they also had this stream of Stanford undergrads who would come through and work there as researchers while they were working on their CS degrees or symbolic systems degrees or whatever it was that they were doing as Stanford undergrads.

Ben Gilbert

One of those people was Chris Cox, who's the chief product officer at Meta. Yeah, that was kind of how he got his start in—

David Rosenthal

—all of this and AI. Obviously, Facebook and Meta are going to come back to the story here in a little bit.

Ben Gilbert

Wow.

David Rosenthal

You really can't make this up. Another undergrad who passed through SAIL while Sebastian was there was a young freshman and sophomore who would later drop out of Stanford to start a company that went through Y Combinator's very first batch in summer 2005.

Ben Gilbert

I'm on the edge of my seat. Who is this?

David Rosenthal

Any guesses?

Ben Gilbert

Dropbox, Reddit. I'm trying to think who else was in the first batch.

David Rosenthal

Oh, no. No. But way more on the nose for this episode. The company was a failed local mobile social network.

Ben Gilbert

Oh, Sam Altman. Loopt.

David Rosenthal

Sam Altman.

Ben Gilbert

That's amazing. He was at SAIL at the same time.

David Rosenthal

He was at SAIL, yep, as an undergrad researcher.

Ben Gilbert

Wow.

David Rosenthal

Wild, right? We told you that it's a very small set of people that are all doing all of this.

Ben Gilbert

Man, I miss those days. Sam presenting at WWDC with Steve Jobs on stage with the double-popped collar, right?

David Rosenthal

Different time in tech.

Ben Gilbert

Yeah, the double-popped collar. That was amazing. That was a vibe. That was a moment. Oh, man.

David Rosenthal

All right. So, April 2007, Sebastian comes over from SAIL into Google—Sebastian Thrun. One of the first things he does over the next set of months is a project called Ground Truth for Google Maps.

Ben Gilbert

Which is essentially Google Maps.

David Rosenthal

It is essentially Google Maps. Before Ground Truth, Google Maps existed as a product, but they had to get all the mapping data from a company called Tele Atlas.

Ben Gilbert

And I think there were two. They were sort of a duopoly. Navteq was the other one.

David Rosenthal

Yeah, Navteq and Tele Atlas.

Ben Gilbert

But it was this kind of crappy source-of-truth map data that everyone used, and you really couldn't do any better than anyone else because you all just used the same data.

David Rosenthal

Yep. It was not that good, and it cost a lot of money. Tele Atlas and Navteq were multibillion-dollar companies. I think maybe one or both of them were public at some point, then got acquired, but a lot of money, a lot of revenue.

Ben Gilbert

Yep. And Sebastian's first thing was Street View, right? So he already had the experience of orchestrating this fleet of all these cars to drive around and take pictures.

David Rosenthal

Yes. So then, coming into Google, Ground Truth is this sort of moonshot-type project to recreate all the Tele Atlas data—

Ben Gilbert

—mostly from their own photographs of streets from Street View. They incorporated some other data. There was census data they used. I think it was 40-something data sources to bring it all together. But Ground Truth was this very ambitious effort to create new maps from whole cloth.

David Rosenthal

Yep. And just like all of the AI and AI-enabled projects within Google that we're talking about here, it works very, very well. Huge win.

Ben Gilbert

Well, especially when you hire 1,000 people in India to help you sift through all the discrepancies in the data and actually hand-draw all the maps. Yes, we are not yet in an era of a whole lot of AI automation.

So, on the back of this win with Ground Truth, Sebastian starts lobbying Larry and Sergey: “Hey, we should do this a lot. We should bring in AI professors and academics—I know all these people—into Google part-time. They don't have to be full-time employees. Let them keep their posts in academia, but come here and work with us on projects for our products. They'll love it. They get to see their work used by millions and millions of people. We'll pay them. They'll make a lot of money. They'll get Google stock, and they get to stay professors at their academic institutions.”

David Rosenthal

Win-win-win.

Ben Gilbert

Win-win-win. So, as you would expect, Larry and Sergey are like, “Yeah, yeah, yeah, that's a good idea. Let's do that. More of that.”

So, in December of 2007, Sebastian brings in a relatively little-known machine learning professor from the University of Toronto named Geoffrey Hinton to the Google campus to come and give a tech talk—not yet hiring him, but to come give a tech talk to all the folks at Google—and talk about some of the new work that Geoffrey and his PhD and postdoc students there at the University of Toronto are doing on blazing new paths with neural networks.

David Rosenthal

And Geoffrey Hinton, for anybody who doesn't know the name, is now very much known as the godfather of neural networks and really the godfather of the whole direction that AI went in—

Ben Gilbert

Modern AI.

David Rosenthal

He was kind of a fringe academic—

Ben Gilbert

At this point in history. I mean, neural networks were not a respected subfield of AI.

David Rosenthal

No, totally not.

Ben Gilbert

And part of the reason is there had been a lot of hype 30 or 40 years before around neural networks that just didn't pan out. So it was effectively something everyone thought was disproven, and certainly a backwater.

David Rosenthal

Yep. Then do you remember from our Nvidia episodes my favorite piece of trivia about Geoffrey Hinton?

Ben Gilbert

Oh, yes. That his great-great-grandfather was George Boole.

Yep. He is the great-great-grandson of George and Mary Boole, who invented Boolean algebra and Boolean logic.

David Rosenthal

Which is hilarious now that I know more about this, because that's the basic building block of symbolic logic, of defined, deterministic computer science logic. And the hilarious thing about neural nets is that it's not symbolic AI. It's not, “I feed you these specific instructions and you follow a big if-then tree.” It is nondeterministic. It is the opposite of that field.

Ben Gilbert

Which actually just underscores again how sort of heretical this branch of machine learning and computer science was.

David Rosenthal

Right. So, Ben, as you were saying earlier, neural networks were not a new idea and had all of this great promise in theory, but in practice, they just took too much computation to do multiple layers. You could really only have a single, or maybe a small single-digit number of layers, in a computer neural network up until this time. But Geoff and his former postdoc, a guy named Yann LeCun, started evangelizing within the community: “Hey, if we can find a way to have multilayered, deep-layered neural networks—something we call deep learning—we could actually realize the promise here.”

It's not that the idea is bad. It's that the implementation would take a ton of compute to actually do all the math, to do all the multiplication required to propagate through layer after layer after layer of neural networks, to sort of detect, understand, and store patterns. If we could actually do that, a big, multilayered neural network would be very valuable and possibly could work.

Ben Gilbert

Yes. Here we are now in 2007, the mid-2000s. Moore's law has increased enough that you could actually start to try to test some of these theories. Jeff comes and gives this talk at Google. It's on YouTube. You can go watch it. We'll link to it in the show notes. This is incredible. This is an artifact of history sitting there on YouTube.

And people at Google—Sebastian, Jeff Dean, and all the other folks who are talking about it—get very, very, very excited because they've already been doing stuff like this with Translate and the language models that they're working with. That's not using the deep neural networks that Jeff's working on. So here's this whole new architectural approach that, if they could get it to work, would enable these models that they're building to work way better, recognize more sophisticated patterns, and understand the data better. Very, very promising.

David Rosenthal

Again, kind of all in theory at this point.

Ben Gilbert

Yep. So Sebastian Thrun brings Geoffrey Hinton into the Google fold after this tech talk. I think first as a consultant over the next couple of years, and then—this is amazing—later, Geoffrey Hinton technically becomes an intern at Google. Like, that's how they get around the—

David Rosenthal

That's correct.

Ben Gilbert

part-time, full-time policies here.

David Rosenthal

Yep. He was a summer intern somewhere around 2011 or 2012. And, mind you, at this point, he's like 60 years old.

Ben Gilbert

Yes. So, in the next couple of years after 2007 here, Sebastian's concept of bringing these computer science and machine learning academics into Google as contractors, part-time workers, or interns—basically letting them keep their academic posts and work on big projects for Google's products internally—goes so well that by late 2009, Sebastian, Larry, and Sergey decide, “Hey, we should just start a whole new division within Google.” And it becomes Google X, the moonshot factory. The first project within Google X, which Sebastian leads himself—

David Rosenthal

I won't say the name. We will come back to it later, but for our purposes for now, the second project would be critically important not only for our story but for the whole world—everything in AI, changing the entire world. And that second project is called Google Brain.

Ben Gilbert

All right, David. So, Google Brain.

David Rosenthal

So, when Sebastian left Stanford full-time and joined Google full-time, of course, somebody else had to take over SAIL. And the person who did is another computer science professor, a brilliant guy named Andrew Ng.

Ben Gilbert

This is like all the hits.

David Rosenthal

All the hits. This is all the AI hits on this episode. So, what does Sebastian do? He recruits Andrew to come part-time and start spending a day a week on the Google campus. And this coincides right with the start of X and Sebastian formalizing this division. So, one day in the 2010–2011 time frame, Andrew's spending his day a week on the Google campus and he bumps into who else? Jeff Dean.

And Jeff Dean is telling Andrew about what he and Franz Och have done with language models and what Geoffrey Hinton is doing in deep learning. Of course, Andrew knows all this. And Andrew is talking about what he and SAIL are doing at Stanford, and they decide, you know, the time might finally be right to try and take a really big swing on this within Google and build a massive, really large deep learning model in the vein of what Geoffrey Hinton has been talking about, on highly parallelizable Google infrastructure.

Ben Gilbert

And when you say the time might be right, Google had tried twice before and neither project really worked. They tried this thing called Brain on Borg. Borg is sort of an internal system that they use to run all of their infrastructure. They tried the Cortex project, and neither of these really worked. So there's a little bit of scar tissue in the research group at Google: Are large-scale neural networks actually going to work for us on Google infrastructure?

David Rosenthal

Yes. So Jeff Dean is working on this system, on the infrastructure, and he decides to name the infrastructure DistBelief, which, of course, is a pun both on the distributed nature of the system and also on the word “disbelief” because—

Ben Gilbert

no one thought it was going to work.

David Rosenthal

Most people in the field thought this was not going to work, and most people in Google thought this was not going to work.

Ben Gilbert

And here's a little bit on why, and it's a little technical, but follow me for a second. All the research from that period of time pointed to the idea that you needed to be synchronous. All the compute needed to be really dense, happening on a single machine with really high parallelism, kind of like what GPUs do. You really would want it all happening in one place so it was easy to look up and see, “Hey, what are the computed values for everything else in the system before I take my next move?”

What Jeff Dean wrote with DistBelief was the opposite. It was distributed across a whole bunch of CPU cores and potentially all over a data center or maybe even in different data centers. So, in theory, this is really bad because it means you would need to be constantly waiting around on any given machine for the other machines to sync their updated parameters before you could proceed. But instead, the system actually worked asynchronously without bothering to go and get the latest parameters from other cores.

So you were sort of updating parameters on stale data. You would think that wouldn’t work. The crazy thing is, it did.

David Rosenthal

Yes. Okay, so you’ve got DistBelief. What do they do with it now? They want to do some research, so they try out, “Can we do cool neural network stuff?” And what they do in a paper that they submitted in 2011, right at the end of the year, is—I’ll give you the name of the paper first—“Building High-Level Features Using Large-Scale Unsupervised Learning.” But everyone just calls it the cat paper.

Ben Gilbert

The cat paper.

David Rosenthal

You talk to anyone at Google, you talk to anyone in AI, they’re like, “Oh yeah, the cat paper.” What they did was train a large, 9-layer neural network to recognize cats from unlabeled frames of YouTube videos, using 16,000 CPU cores on 1,000 different machines.

Ben Gilbert

And listeners, just to underscore how seminal this is, we actually talked with Sundar in prep for the episode. He cited seeing the cat paper come across his desk as one of the key moments that sticks in his brain in Google’s story.

David Rosenthal

Yeah. A little later on, they would do a TGIF where they would present the results of the cat paper. You talk to people at Google and they’re like, “That TGIF—oh my God, that’s when it all changed.”

Ben Gilbert

Yeah. It proved that large neural networks could actually learn meaningful patterns without supervision and without labeled data. And not only that, it could run on a distributed system that Google built to actually make it work on their infrastructure. And that is a huge unlock of the whole thing. Google’s got this big infrastructure asset. Can we take this theoretical computer science idea that the researchers have come up with and use DistBelief to actually run it on our system? Yep, that is the amazing technical achievement here. That is almost secondary to the business impact of the cat paper. I think it’s not that much of a leap to say that the cat paper led to probably hundreds of billions of dollars of revenue generated by Google, Facebook, and ByteDance over the next decade.

David Rosenthal

Definitely—pattern recognizers in data. So YouTube had a big problem at this time, which was that people would upload these videos, and there were tons of videos being uploaded to YouTube, but people were really bad at describing what was in the videos they uploaded. YouTube was trying to become more of a destination site, trying to get people to watch more videos, trying to build a feed, increase dwell time, et cetera. The problem was, the recommender was trying to figure out what to feed people, and it was only working off titles and descriptions that people were writing about their own videos.

Ben Gilbert

Right? And whether you’re searching for a video or they’re trying to figure out what video to recommend next, they need to know what the video is about.

David Rosenthal

Yep. So the cat paper proves that you can use this technology—a deep neural network running on DistBelief—to go inside the videos in the YouTube library, understand what they were about, and use that data to then figure out what videos to serve to people.

Ben Gilbert

If you can answer the question “cat or not a cat,” you can answer a whole lot more questions, too.

David Rosenthal

Here’s a quote from Jeff Dean about this:

“We built a system that enabled us to train pretty large neural nets through both model and data parallelism. We had a system for unsupervised learning on 10 million randomly selected YouTube frames. As you were saying, Ben, it would build up unsupervised representations based on trying to reconstruct the frame from the higher-level representations. We got that working and training on 2,000 computers using 16,000 cores. After a little while, that model was actually able to build a representation at the highest neural net level where one neuron would get excited by images of cats. It had never been told what a cat was, but it had seen enough examples of them in the training data—head-on facial views of cats—that that neuron would then turn on for cats and not much else.”

Ben Gilbert

It’s so crazy. I mean, this is the craziest thing about unlabeled data and unsupervised learning: a system can learn what a cat is without ever being explicitly told what a cat is, and that there’s a cat neuron.

David Rosenthal

Yeah. And so then there’s an iPhone neuron and a San Francisco Giants neuron and all the things that YouTube recommends.

Ben Gilbert

Not to mention porn filtering and explicit-content filtering.

David Rosenthal

Not to mention copyright identification and enabling revenue share with copyright holders. Yeah, this leads to everything in YouTube. Basically, it puts YouTube on the path to becoming, today, the single biggest property on the internet and the single biggest media company on the planet. This kicks off a 10-year period from 2012, when this happens, until ChatGPT on November 30, when AI is already shaping human existence for all of us and driving hundreds of billions of dollars of revenue. It’s just in the YouTube feed, and then Facebook borrows it and hires Yann LeCun and starts Facebook AI Research. Then they bring it into Instagram, and then TikTok and ByteDance take it, and then it goes back to Facebook and YouTube with Reels and Shorts. This is the primary way that humans on the planet spend their leisure time for the next 10 years.

Ben Gilbert

This is my favorite David Rosenthalism. Everyone talks about 2022 onward as the AI era. I love this point from you that, actually, for anyone who could make good use of a recommender system and a classifier system—basically any company with a social feed—the AI era started in 2012.

David Rosenthal

Yes, the AI era started in 2012, and part of it was the cat paper. The other part of it was what Jensen at NVIDIA always calls the big-bang moment for AI, which was AlexNet.

Ben Gilbert

Yes. So, we talked about Geoffrey Hinton back at the University of Toronto. He’s got 2 grad students who he’s working with in this era: Alex Krizhevsky and Ilya Sutskever—

David Rosenthal

Of course.

Ben Gilbert

—future co-founder and chief scientist of OpenAI. And the 3 of them are working with Geoffrey’s deep neural network ideas and algorithms to create an entry for the famous ImageNet competition in computer science.

David Rosenthal

This is Fei-Fei Li’s thing from Stanford.

Ben Gilbert

It is an annual machine-vision algorithm competition. Fei-Fei had assembled a database of 14 million images that were hand-labeled. Famously, she used Amazon Mechanical Turk, I think, to get them all hand-labeled.

David Rosenthal

Yes. And I think that’s right. So then the competition was: what team can write the algorithm that, without looking at the labels—just seeing the images—could correctly identify the largest percentage? The best algorithms that would win the competition year over year were still getting more than a quarter of the images wrong. So, about a 75% success rate. Great, but way worse than a human.

Ben Gilbert

You can’t use it for much in a production setting when a quarter of the time you’re wrong.

David Rosenthal

So then the 2012 competition comes along: AlexNet. Its error rate was 15%. Still high, but a 10% leap from the previous best, going from a 25% error rate all the way down to 15% in 1 year. A leap like that had never happened before.

Ben Gilbert

It’s 40% better than the next best.

David Rosenthal

Yes.

Ben Gilbert

On a relative basis.

David Rosenthal

Yes.

Ben Gilbert

And why is it so much better, David? What did they figure out that would create a $4 trillion company in the future?

David Rosenthal

So, what Geoff, Alex, and Ilya did is—they knew, like we’ve been talking about all episode, that deep neural networks had all this potential, and Moore’s law had advanced enough that you could use CPUs to create a few layers. They had the aha moment of: what if we rearchitected this stuff not to run on CPUs, but to run on a whole different class of computer chips that were, by their very nature, highly parallelizable—video-game graphics cards made by the leading company in the space at the time, NVIDIA? It was not obvious at the time, and especially not obvious that this highly advanced, cutting-edge academic computer science research—

Ben Gilbert

—that was being done on supercomputers, usually—

David Rosenthal

—that was being done on supercomputers with incredible CPUs—would use these toy video-game cards—

Ben Gilbert

—that retail for $1,000.

David Rosenthal

Yeah, less at that point in time—a couple hundred bucks. So the team in Toronto goes out to the local Best Buy or something. They buy 2 NVIDIA GeForce GTX 580s, which were NVIDIA’s top-of-the-line gaming cards at the time. The Toronto team rewrites their neural-network algorithms in CUDA, NVIDIA’s programming language. They train it on these 2 off-the-shelf GTX 580s, and this is how they achieve their deep neural network and do 40% better than any other entry in the ImageNet competition.

So when Jensen says that this was the big-bang moment of artificial intelligence, A, he’s right. This shows everybody that, holy crap, if you can do this with 2 off-the-shelf GTX 580s, imagine what you could do with more of them or with specialized chips. And B, this event is what sets NVIDIA on the path from a somewhat struggling PC-gaming accessory maker to the leader of the AI wave and the most valuable company in the world today.

This is how AI research tends to work: there’s some breakthrough that gets you this big step-change function, and then there’s actually a multiyear process of optimizing from there, where you get these diminishing-returns curves on breakthroughs. The first half of the advancement happens all at once, and then the second half takes many years after that to figure out. It’s rare and amazing, and it must be so cool when you have an idea, you do it, and then you realize, “Oh my God, I just found the next giant leap in the field.”

Ben Gilbert

It’s like, “I unlocked the next level,” to use the video-game analogy.

David Rosenthal

Yes.

Ben Gilbert

I leveled up. So after AlexNet, the whole computer-science world is abuzz.

David Rosenthal

People are starting to stop doubting neural networks at this point.

Ben Gilbert

Yes. So after AlexNet, the 3 of them from Toronto—Geoffrey Hinton, Alex Krizhevsky, and Ilya Sutskever—do the natural thing: they start a company called DNNResearch, deep neural network research.

This company does not have any products. This company has AI researchers who just won a big competition. Predictably, as you might imagine, it gets acquired by Google almost immediately.

David Rosenthal

Are you intentionally shortening this?

Ben Gilbert

That’s what I thought the story was. Oh, it is not immediately.

David Rosenthal

Oh, okay.

Ben Gilbert

There’s a whole crazy thing that happens where the first bid is actually from Baidu.

David Rosenthal

I did not know that.

Ben Gilbert

Baidu offers $12 million. Geoffrey Hinton doesn’t really know how to value the company and doesn’t know if that’s fair. So he does what any academic would do to best determine the market value of the company.

He says, “Thank you so much. I’m gonna run an auction now, and I’m going to run it in a highly structured manner where every time anybody wants to bid, the clock resets and there’s another hour where anybody else can submit another bid.”

David Rosenthal

No way.

Ben Gilbert

He gets in touch with everyone he knows from the research community who is now working at a big company, who he thinks, “Hey, this would be a good place for us to do our research.” That includes Baidu, Google, Microsoft, and there’s one other.

David Rosenthal

Facebook. Of course.

Ben Gilbert

It’s a two-year-old startup.

David Rosenthal

Oh, wait. So it does not include Facebook.

Ben Gilbert

It does not include Facebook. Think about the year. This is 2012. Facebook’s not really in the AI game yet. They’re still trying to build their own AI lab.

David Rosenthal

Yeah, because Yann LeCun and FAIR start in 2013. Is it Instagram?

Ben Gilbert

Nope. It is the most important part of the end of this episode.

David Rosenthal

Wait. Well, it can’t be Tesla because Tesla is older than that.

Ben Gilbert

Nope.

David Rosenthal

Well, OpenAI wouldn’t get founded for years.

Ben Gilbert

What company slightly predated OpenAI doing effectively the same mission?

David Rosenthal

Oh, of course. Of course. Hiding in plain sight. DeepMind. Wow, DeepMind, baby.

Ben Gilbert

They are the fourth bidder in a four-way auction for DNNResearch. Now, of course, right after the bidding starts, DeepMind has to drop out. They’re a startup. They don’t actually have the cash to be able to buy it.

David Rosenthal

Yeah. Didn’t even cross my mind, because my first question was, “Where the hell would they get the money?” They had no money.

Ben Gilbert

But Geoffrey Hinton already knows and respects Demis—

David Rosenthal

Ah.

Ben Gilbert

—even though, at the time, he’s just doing this at a startup called DeepMind.

David Rosenthal

That’s amazing. Wait, how is DeepMind in the auction, but Facebook is not? Isn’t that wild?

Ben Gilbert

That’s wild.

The timing of this is concurrent with what was then called NIPS, now called the NeurIPS conference. So Geoffrey Hinton actually runs the auction from his hotel room at Harrah’s Casino in Lake Tahoe.

David Rosenthal

Oh my God, amazing.

Ben Gilbert

The bids all come in, and we’ve got to thank Cade Metz, the author of Genius Makers, a great book on the whole history of AI that we’re actually going to reference a lot in this episode. The bidding goes up and up and up. At some point, Microsoft drops out. They come back in. I told you, DeepMind drops out.

So it’s Baidu and Google really going at it at the end. Finally, at some point, the researchers look at each other and say, “Where do we actually want to land? We want to land at Google.” And so they stop the bidding at $44 million and just say, “Google, this is more than enough money. We’re going with you.”

David Rosenthal

Wow. I knew it was about $40 million. I did not know that whole story. It’s almost like Google itself and the Dutch auction IPO process, right? How fitting.

Ben Gilbert

That’s kind of a perfect DNA. Yes.

David Rosenthal

Wow.

Ben Gilbert

The three of them were supposed to split it 33% each, and Alex and Ilya go to Jeff and say, “I really think you should have a bigger percentage. I think you should have 40%, and we should each have 30%.” And that’s how it ends up breaking down.

David Rosenthal

Ah, wow. What a team.

Ben Gilbert

That leads to the three of them joining Google Brain directly and turbocharging everything going on there. Spoiler alert: a couple of years later, Astro Teller, who would take over running Google X after Sebastian Thrun left, would be quoted in The New York Times in a profile of Google X saying that the gains to Google’s core businesses in search, ads, and YouTube from Google Brain have way more than funded all of the other bets they have made within Google X and throughout the company over the years.

It’s one of these things that if you make something a few percent better that happens to do tens of billions of dollars or hundreds of billions of dollars in revenue, you find quite a bit of loose change in those couch cushions.

David Rosenthal

Yes, quite a bit of loose change.

Ben Gilbert

But that’s not where the AI history ends within Google. There is another very important piece of the Google AI story that is an acquisition from outside of Google. The AI equivalent of Google’s acquisition of YouTube. It’s what we talked about a minute ago: DeepMind.

David Rosenthal

All right, Ben. DeepMind. I kind of like your framing: the YouTube of AI.

Ben Gilbert

The YouTube of AI for Google. They bought this thing for—we’ll talk about the purchase price—but it’s worth, what, $500 billion today? I mean, this is as good as Instagram or YouTube in terms of the greatest acquisitions of all time.

David Rosenthal

100%. I remember when this deal happened, just like I remember when the Instagram deal happened, because the number was big at the time.

Ben Gilbert

It was big, but I remember it for a different reason. It was like when Facebook bought Instagram: “Oh my God, this is—wow, what a tectonic shift in the landscape of tech.” In January 2014, I remember reading on TechCrunch this random news—

David Rosenthal

Right? You’re like, “Deep what?”

Ben Gilbert

—that Google is spending a lot of money to buy something in London that I’ve never heard of that’s working on artificial intelligence.

David Rosenthal

Right. This really illustrates how outside of mainstream tech AI was at the time.

Ben Gilbert

Yeah. And then you dig in a little further and you’re like, this company doesn’t seem to have any products. It also doesn’t really say anything on its website about what DeepMind is. It says it is a “cutting-edge artificial intelligence company.”

David Rosenthal

Wait, did you look this up on the Wayback Machine?

Ben Gilbert

Yeah, I did. I did.

David Rosenthal

Oh, nice.

Ben Gilbert

“To build general-purpose learning algorithms for simulations, e-commerce, and games.” This is 2014. This does not compute. It does not register.

David Rosenthal

Simulations, e-commerce, and games. It’s kind of a random spattering of—

Ben Gilbert

Exactly. It turns out, though, not only was that description of what DeepMind was fairly accurate, this company and this purchase of it by Google was the butterfly-flapping-its-wings equivalent moment that directly leads to OpenAI, ChatGPT, Anthropic, and basically everything—

David Rosenthal

Certainly Gemini—

Ben Gilbert

—that we know. Yeah, Gemini directly in the world of AI today—

David Rosenthal

—and probably xAI, given Elon’s involvement.

Ben Gilbert

Yeah, of course, xAI.

David Rosenthal

In a weird way, it sort of leads to Tesla self-driving too. Andrej Karpathy.

Ben Gilbert

Yeah, definitely. Okay, so what is the story here? DeepMind was founded in 2010 by a neuroscience PhD named Demis Hassabis, who previously started a video game company.

David Rosenthal

Oh yeah.

Ben Gilbert

And a postdoc named Shane Legg at University College London, and a third co-founder who was one of Demis’s friends from growing up, Mustafa Suleyman.

This was unlikely, to say the least.

David Rosenthal

This would go on to produce a knight and Nobel Prize winner.

Ben Gilbert

Yes. So Demis, the CEO, was a childhood chess prodigy turned video game developer. When he was 17 in 1994, he had gotten accepted to the University of Cambridge, but he was too young, and the university told him, “Hey, take a gap year and come back.” He decided that he was going to work at a video game studio called Bullfrog Productions for the year.

While he was there, he created the game Theme Park, if you remember that. It was like a theme-park version of SimCity. This was a big game.

David Rosenthal

Oh, I played a ton of that. Yeah, it sells 15 million copies in the mid-’90s. Wow, wild.

Ben Gilbert

Then after this, he goes to Cambridge and studies computer science there. After Cambridge, he gets back into gaming, founds another game studio called Elixir that would ultimately fail, and then decides, “You know what? I’m going to go get my PhD in neuroscience.” And that is how Demis ends up at University College London.

There he meets Shane Legg, who’s there as a postdoc. Shane is a self-described, at the time, member of the lunatic fringe in the AI community. This is 2008, 2009, 2010, and he believes that AI is going to get more and more and more powerful every year, and that it will become so powerful that it will become more intelligent than humans. Shane is one of the people who actually popularizes the term artificial general intelligence, AGI.

David Rosenthal

Oh, interesting. Which, of course, lots of people talk about now, and approximately zero people were afraid of that. You had the Nick Bostrom-type folks, but very few people were thinking about superintelligence or the singularity or anything like that.

Ben Gilbert

For what it’s worth, not Elon Musk. He’s not included in that list because Demis would be the one who tells Elon about this. Yes, we’ll get to it.

So Demis and Shane hit it off. They pull in Mustafa, Demis’s childhood friend, who is himself extremely intelligent. He had gone to the University of Oxford and then dropped out, I think, at age 19, to do other startup-y type stuff. So the three of them decide to start a company, DeepMind.

The name, of course, is a reference to deep learning, Geoffrey Hinton’s work, and everything coming out of the University of Toronto, as well as the goal that the three of these guys have of actually creating an intelligent mind with deep learning. Geoffrey, Yann, and Alex aren’t really thinking about this yet. As we said, this is lunatic-fringe-type stuff.

David Rosenthal

Yes, AlexNet, the cat paper, that whole world is about better classifying data. Can we better sort into patterns? It’s a giant leap from there to say, “Oh, we’re going to create intelligence.”

Ben Gilbert

Yes. I think probably some people, almost certainly at Google, were thinking, “Oh, we can create narrow intelligence that’ll be better than humans at certain tasks.”

David Rosenthal

I mean, a calculator is better than humans at certain tasks.

Ben Gilbert

Right, but I don’t think too many people were thinking, “Oh, this is going to be general intelligence, smarter than humans.”

David Rosenthal

So they decide the tagline for the company is going to be “Solve intelligence and use it to solve everything else.”

Ben Gilbert

Ooh, I like it. I like it. Yeah, they’re good marketers, too, these guys.

David Rosenthal

So there’s just one problem. To do what they want to do—

Ben Gilbert

Money. Just saying. Money is the problem.

David Rosenthal

Right, right, right. Money is the problem for lots of reasons. But even more so than any other given startup in the 2010 era, it’s not like they can just go spin up an AWS instance, build an app, and deploy it to the App Store. They want to build really, really, really, really, really big deep-learning neural networks, and that requires Google-sized levels of compute.

Well, it’s interesting. They actually don’t require that much funding yet. The AI of the time was, go grab a few GPUs. We’re not training giant LLMs. That’s the ambition eventually, but right now, what they just need to do is raise a few million bucks. But who’s going to give you a few million bucks when there’s no business plan? When you’re just trying to solve intelligence, you need to find some lunatics.

Ben Gilbert

It’s a tough sell to VCs.

David Rosenthal

Except for exactly the right ones.

Ben Gilbert

As you say, they need to find some lunatics.

David Rosenthal

Oh, I chose my words carefully, didn’t I?

Ben Gilbert

Yeah, we use the term “lunatic” in—

David Rosenthal

It’s endearing.

Ben Gilbert

In the most endearing possible way here, given that they were all basically right.

David Rosenthal

So in June 2010, Demis and Shane managed to get invited to the Singularity Summit in San Francisco, California.

Ben Gilbert

Because they’re not raising money for this in London.

David Rosenthal

Yeah, definitely. I think they tried for a couple of months and learned that was not going to be a viable path.

Ben Gilbert

Yes. This summit, the Singularity Summit, was organized by Ray Kurzweil—a future Google employee, I think, chief futurist, noted futurist.

David Rosenthal

There were also Eliezer Yudkowsky and Peter Thiel.

Ben Gilbert

Yes. So Demis and Shane are excited about getting this invite. This is probably our one chance to get funded.

David Rosenthal

But we probably shouldn’t just walk in guns blazing and say, “Peter, can we pitch you?”

Ben Gilbert

Yeah. So they finagle their way into Demis getting to give a talk on stage at the summit.

David Rosenthal

Always the hack.

Ben Gilbert

They’re like, “This is great. This is going to be the hack. The talk is going to be our pitch to Peter and Founders Fund.”

David Rosenthal

Peter has just started Founders Fund at this point. Obviously, he’s a member of the PayPal Mafia and very wealthy.

Ben Gilbert

I think he had a big Roth IRA at this point, which is the right way to frame it.

David Rosenthal

A big Roth IRA that he had invested in Facebook; he was the first investor in Facebook.

Ben Gilbert

He is the perfect target.

David Rosenthal

They architect the presentation at the summit to be a pitch directly to Peter—essentially, a thinly veiled pitch. Shane has a quote in Parmy Olson’s great book Supremacy, which we used as a source for a lot of this DeepMind story. Shane says, “We needed someone crazy enough to fund an AGI company. Somebody who had the resources not to sweat a few million and liked super ambitious stuff.”

Ben Gilbert

They also had to be massively contrarian, because every professor that he would go talk to would certainly tell him, “Absolutely do not even think about funding this.”

David Rosenthal

That Venn diagram sure sounds a lot like Peter Thiel.

Ben Gilbert

So they show up at the conference. Demis is going to give the talk. He goes out on stage and looks out into the audience. Peter is not there.

David Rosenthal

He’s a busy guy. He’s a co-founder or co-organizer, but he’s a busy guy.

Ben Gilbert

Yes. The guy’s like, “Shoot. Oh, we missed our chance. What are we going to do?” And then fortune turns in their favor. They find out that Peter is hosting an after-party that night at his house in San Francisco.

They get into the party. Demis seeks out Peter, and Demis is very, very smart, as anybody who’s ever listened to him talk would immediately know. Rather than just pitching Peter head-on, he’s going to come about this obliquely. He starts talking to Peter about chess because he knows, as everybody does, that Peter Thiel loves chess. And Demis had been the second-highest-ranked player in the world as a teenager in the under-14 category.

David Rosenthal

Good strategy.

Ben Gilbert

Great strategy.

David Rosenthal

The man knows his chess moves.

Ben Gilbert

So Peter’s like, “Hmm, I like you. You seem smart. What do you do?” And Demis explains that he’s got this AGI startup. They were actually here, and he gave a talk on stage as part of the conference. People are excited about this.

David Rosenthal

“Okay, all right. Come back to Founders Fund tomorrow and give me the pitch.”

Ben Gilbert

So they do. They make the pitch. It goes well. Founders Fund leads DeepMind’s seed round of about $2 million.

David Rosenthal

My, how times have changed for AI company seed rounds these days.

Ben Gilbert

Oh, yes.

David Rosenthal

Imagine leading DeepMind’s seed round with a check of less than $2 million.

Ben Gilbert

And through Peter and Founders Fund, they get introduced—

David Rosenthal

“Hey, Elon, you should meet this guy.”

Ben Gilbert

To another member of the PayPal Mafia, Elon Musk.

David Rosenthal

Yes. So it’s teed up in a pretty low-key way: “Hey, Elon, you should meet this guy. He’s smart. He’s thinking about artificial intelligence.”

Ben Gilbert

So Elon says, “Great. Come over to SpaceX. I’ll give you the tour of the place.”

David Rosenthal

So Demis comes over for lunch and a tour of the factory. Of course, Demis thinks it’s very cool, but really, he’s trying to reorient the conversation over to artificial intelligence.

Ben Gilbert

I’ll read this great excerpt from an article in The Guardian:

“Musk told Hassabis his priority was getting to Mars as a backup planet in case something went wrong here. I don’t think he’d thought much about AI at this point. Hassabis pointed out a flaw in his plan. I said, ‘What if AI was the thing that went wrong here? Then being on Mars wouldn’t help you because if we got there, then it would obviously be easy for an AI to get there through our communication systems or whatever it was.’

“He hadn’t thought about that. So he sat there for a minute without saying anything, just thinking, ‘Hmm, that’s probably true.’ Shortly after, Musk, too, became an investor in DeepMind.”

David Rosenthal

Yes.

Ben Gilbert

Yes. Yes.

David Rosenthal

I think it’s crazy that Demis is sort of the one who woke Elon up to this idea of, “We might not be safe from the AI on Mars either.”

Ben Gilbert

Right. I hadn’t considered that.

David Rosenthal

So this is the first time the bit flips for Elon: We really need to figure out a safe, secure AI for the good of the people. That’s sort of a seed being planted in his head.

Ben Gilbert

Yep.

David Rosenthal

Which, of course, is what DeepMind’s ambition is. We are here doing research for the good of humanity, like scientists, in a peer-reviewed way.

Ben Gilbert

Yep.

Ben Gilbert

I think all that is true. Also, in the intervening months to a year after this meeting between Demis and Elon—and Elon investing in DeepMind—Elon also starts to get really excited and convinced about the capabilities of AI in the near term, and specifically the capabilities of AI for Tesla.

David Rosenthal

Yes. Like with everything else in Elon’s world, once the bit flips and he becomes interested, he completely changes the way he views the world. He completely sheds all the old ways and actions that he was taking. And it’s all about, what do I do to embrace this new worldview that I have?

Ben Gilbert

And other people have been working on it for a while already by this point: AI driving cars.

David Rosenthal

Yep. That sounds like it would be a pretty good idea for Tesla, doesn’t it?

Ben Gilbert

So Elon starts trying to recruit as many AI researchers as he possibly can, along with machine-vision and machine-learning experts, into Tesla. And then AlexNet happens, and AlexNet is really good at identifying and classifying images, including cat videos on YouTube and the YouTube recommender feed. Well, is that really that different from a live feed of video from a car that’s being driven, and understanding what’s going on there?

David Rosenthal

Can we process it in real time and look at differences between frames?

Ben Gilbert

Perhaps controlling the car is not all that different. So Elon’s excitement, channeled initially through DeepMind and Demis, about AI and AI for Tesla starts ratcheting up big time.

David Rosenthal

Yep. Meanwhile, back in London, DeepMind is getting to work. They’re hiring researchers, getting to work on models, and making some vague noises about products to their investors. Maybe they could do something in shopping, maybe something in gaming, like the description on the website at the time of acquisition said. But mostly, what they really want to do is just build these models and work on intelligence.

Then one day in late 2013, they get a call from Mark Zuckerberg. He wants to buy the company. Mark has woken up to everything that’s going on at Google after AlexNet, what AI is doing for social-media feed recommendations at YouTube, and the possibility of what it can do at Facebook and for Instagram. He’s gone out and recruited Yann LeCun, Geoffrey Hinton’s old protégé, who, together with Geoff, is one of the godfathers of AI and deep learning and really popularized the idea of convolutional neural networks, the next hot thing in the field of AI at this point in time.

With Yann, they have created FAIR—Facebook AI Research—which is a Google Brain rival within Facebook. Remember who the first investor in Facebook was, who’s still on the board and is also the lead investor in DeepMind?

Ben Gilbert

Peter Thiel.

David Rosenthal

Where do you think Mark learned about DeepMind? Peter Thiel, was it? Do you know for sure that it was from Peter?

Ben Gilbert

No, I don’t know for sure, but how else could Mark have learned about this startup in London?

David Rosenthal

I’ve got a great story of how Larry Page found out about it.

Ben Gilbert

Oh, okay. Well, we’ll get to that in 1 second.

David Rosenthal

So Mark calls and offers to buy the company. There are various rumors about how much Mark offered, but according to Parmy Olson in her book Supremacy, the reports are that it was up to $800 million. That’s for a company with no products and a long way from AGI.

Ben Gilbert

That squares with what Cade Metz has in his book: the founders would have made about twice as much money from taking Facebook’s offer versus taking Google’s offer.

David Rosenthal

Yep. So Demis, of course, takes this news to the investor group, which, by the way, is kind of against everything the company was founded on. The whole aim of the company, and what he’s promised the team, is that DeepMind is going to stay independent, do research, and publish in the scientific community. We’re not going to be captured and told what to do by the whims of a capitalist institution.

Ben Gilbert

Yep. So definitely some deal-point negotiating has to happen with Mark and Facebook if this offer is going to come through.

David Rosenthal

But Mark is so desperate at this point. He is open to these very large deal-point negotiations, such as Yann LeCun getting to stay in New York and continuing to operate his lab at NYU. Yann is a professor, so Mark is flexible on some things. It turns out Mark is not flexible on letting Demis keep control of DeepMind if he buys it. Demis argues that they need to stay separate and carved out, and that they need this independent oversight board with his ability to intervene if the mission of DeepMind is no longer being followed. Mark is like, “No, you’ll be a part of Facebook.”

Ben Gilbert

Yeah. And you’ll make a lot of money.

David Rosenthal

So, as this negotiation is going on, of course, the investors in DeepMind get wind of this. Elon finds out what’s going on. He immediately calls up Demis and says, “I will buy the company right now with Tesla stock.” This is late 2013, or early 2014. Tesla’s market cap is about $20 billion, so Tesla stock from then to today is about a 70x run-up.

Demis, Shane, and Mustafa are like, “Wow. Okay, there’s a lot going on right now.” But to your point, they have the same issues with Elon and Tesla that they had with Mark. Elon wants them to come in and work on autonomous driving for Tesla. They don’t want to work on autonomous driving—

Ben Gilbert

Right? Or at least exclusively.

David Rosenthal

At least exclusively. Yep. So then Demis gets a third call from Larry Page.

Ben Gilbert

Do you want my story of how Larry knows about the company?

David Rosenthal

I absolutely want your story of how Larry knows about the company.

Ben Gilbert

All right, so this is still early in DeepMind's life. We haven't progressed all the way to this acquisition point yet. Apparently, Elon Musk is on a private jet with Luke Nosek, who's another member of the PayPal Mafia and an angel investor in DeepMind, and they're reading an email from Demis with an update about a breakthrough they had, where DeepMind AI figured out a clever way to win at the Atari game Breakout.

David Rosenthal

Yes. And the strategy it figured out, with no human training, was that you could bounce the ball up around the edges of the bricks. Then, without needing to intervene, it could bounce around along the top and win the game faster, without needing a whole bunch of interactions with the paddle down at the bottom.

Ben Gilbert

They're watching this video of how clever it is, and flying with them on the same private plane is Larry Page. Of course, because Elon and Larry used to be very good friends.

David Rosenthal

Yes. And Larry's like, “Wait, what are you watching? What company is this?” And that's how he finds out.

Ben Gilbert

Wow.

David Rosenthal

Yes.

Ben Gilbert

Elon must have been so angry about all this.

David Rosenthal

And the crazy thing is, this kinship between Larry and Demis is, I think, the reason why the deal gets done at Google. Once the two of them get together, they are like peas in a pod. Larry has always viewed Google as an AI company.

Ben Gilbert

Yeah.

David Rosenthal

Demis, of course, views DeepMind so much as an AI company that he doesn't even want to make any products until they can get to AGI.

Ben Gilbert

And Demis, in fact—we should share with listeners—told us this when we were talking to him to prepare for this episode. He just felt like Larry got it. Larry was completely on board with the mission of everything that DeepMind was doing.

There's something else very convenient about Google: they already have Brain. So Larry doesn't need Demis, Shane, Mustafa, and DeepMind to come work on products within Google.

David Rosenthal

Right?

Ben Gilbert

Brain is already working on products within Google. Demis can really believe Larry when Larry says, “Nah, stay in London. Keep working on intelligence. Do what you're doing. I don't need you to come work on products within Google.”

Brain is actively going and engaging with the product groups, trying to figure out, “Hey, how can we deploy neural nets into your product to make it better?” That's their reason for being. So they're happy to agree to this.

David Rosenthal

And it's working. Brain and neural nets are getting integrated into Search, into ads, into Gmail, into everything. It is the perfect home for DeepMind—a home away from home, shall we say?

Ben Gilbert

Yes. And there's a third reason why Google is the perfect fit for DeepMind: infrastructure. Google has all the compute infrastructure you could ever want right there on tap.

David Rosenthal

Yes. At least with CPUs so far.

Ben Gilbert

Yes.

David Rosenthal

So, how does the deal actually happen?

Ben Gilbert

Well, after buying DNNResearch, Alan Eustace—who, David, you spoke with, right?

David Rosenthal

Yep.

Ben Gilbert

Alan was Google's head of engineering at the time. He makes up his mind that he wants to hire all the best deep-learning research talent that he possibly can, and he has a clear path to do so.

A few months earlier, Larry Page held a strategy meeting on an island in the South Pacific. In Cade Metz's book, it's an undisclosed island.

David Rosenthal

Of course he did.

Ben Gilbert

Larry thought that deep learning was going to completely change the whole industry. And so he tells his team—and this is a quote—“Let's really go big,” which effectively gave Alan a blank check to go secure all the best researchers that he possibly could.

So, in 2013, he decides, “I'm going to get on a plane in December, before the holidays, and go meet DeepMind.” There's a crazy story about this. Geoffrey Hinton, who's at Google at the time, had a problem with his back where he couldn't sit down. He either had to stand or lie down, and so a long flight across the ocean was not doable.

But he needs to be there as part of the diligence process. You have Geoffrey Hinton; you need to use him to figure out if you're going to buy a deep-learning company. And so Alan Eustace decides he's going to charter a private jet, and he's going to build this crazy custom harness rig so that Geoffrey Hinton won't be sliding around when he's lying on the floor during takeoff and landing.

David Rosenthal

Wow. I was thinking, the first part of this, I'm pretty sure Google has planes. They could just get into a Google plane.

Ben Gilbert

For whatever reason, this was a separate charter.

David Rosenthal

But it’s not solvable just with a private plane. You need a harness, right? And Alan is the guy who set the record for the highest free-fall jump that anyone has ever done. Was it from a balloon? I actually don’t know. It was even higher than that Red Bull stunt a few years before.

So he’s very used to designing these custom rigs for airplanes. He’s like, “Oh, no problem. You just need a bed and some straps. I jumped out of the atmosphere in a scuba suit. I think we’ll be fine.”

Ben Gilbert

That is amazing.

David Rosenthal

So they fly to London. They do the diligence. They make the deal. Demis has true kinship with Larry, and it’s done: $550 million US. There’s an independent oversight board that is set up to make sure that the mission and goals of DeepMind are actually being followed, and this is an asset that Google owns today that, again, I think is worth half a trillion dollars if it’s independent.

Ben Gilbert

Do you know what other member of the PayPal Mafia gets put on the ethics board after the acquisition?

David Rosenthal

Reid Hoffman.

Ben Gilbert

Reid Hoffman has to be given the OpenAI tie later. We are going to come back to Reid in just a little bit here.

David Rosenthal

Yes. So after the acquisition, it goes very well, very quickly. Famously, the data center cooling thing happens, where DeepMind carved off some part of the team to go and be an emissary to Google and look for ways to use DeepMind. One of them is around data center cooling.

Very quickly, in July of 2016, Google announces a 40% reduction in the energy required to cool data centers. I mean, Google’s got a lot of data centers—a 40% energy reduction. I actually talked with Jim Gao, who’s a friend of the show and actually led a big part of this project. It was just the most obvious application of neural networks inside of Google right away. Pays for itself.

Ben Gilbert

Yeah. Imagine that paid for the acquisition pretty quickly there.

David Rosenthal

Yes. Ben, should we talk about AlphaGo on this episode?

Ben Gilbert

Yeah. Yeah. Yeah.

David Rosenthal

I watched the whole documentary that Google produced about it. It’s awesome. This is actually something that you would enjoy watching. Even if you’re not researching a podcast episode and you’re just looking to pull something up and spend an hour or 2, I highly recommend it.

It’s on YouTube. It’s the story of how DeepMind, post-acquisition from Google, trained a model to beat the world Go champion at Go. And I mean, everyone in the whole Go community coming in thought there was no chance. This guy, Lee Sedol, is so good that there’s no way that an AI could possibly beat him.

It’s a 5-game thing. It just won the first 3 games straight. I mean, completely cleaned up, and with inventive, new, creative moves that no human has played before. That’s sort of the big crazy takeaway.

Ben Gilbert

There’s a moment in one of the games, right, where it makes a move and people are like, “Is that a mistake? That must have just been an error.”

David Rosenthal

Yeah. Move 37.

Ben Gilbert

Yeah. And then 100 moves later, it plays out and—

David Rosenthal

—that it was completely genius. And humans are now learning from DeepMind’s strategy of playing the game and discovering new strategies.

A fun thing for Acquired listeners who are like, “Why Go?” Go is so complicated compared to chess. Chess has 20 moves that you can make at the beginning of the game in any given turn, and then midgame there are about 30 to 40 moves that you could make. Go, on any given turn, has about 200.

And so if you think combinatorially, the number of possible configurations of the board is more than the number of atoms in the universe.

Ben Gilbert

That’s a great Demis quote, by the way.

David Rosenthal

Yeah.

Ben Gilbert

And so he says, even if you took all the computers in the world and ran them for a million years, as of 2017, that wouldn’t be enough compute power to calculate all the possible variations. So it’s cool because it’s a problem that you can’t brute-force. You have to do something like neural networks.

There is this whitespace to be creative and explore. And so it served as this amazing breeding ground for watching a neural network be creative against a human.

David Rosenthal

Yeah. And of course, it’s totally in with Demis’ background and the DNA of the company of playing games. Demis was a chess champion. And then after Go, they play StarCraft, right?

Ben Gilbert

Oh, really? I actually didn’t know that.

David Rosenthal

Yeah. The next game that they tackled was StarCraft, a real-time strategy game against an opponent.

Ben Gilbert

Yes, David. So what are the second-order effects of Google buying DeepMind? Well, there’s one person who is really, really, really upset about this, and maybe 2 people if you include Mark Zuckerberg, but Mark tends to play his cards a little closer to the vest.

Of course, Elon Musk is very upset about this acquisition. When Google buys DeepMind out from under him, Elon goes ballistic. As we said, Elon and Larry had always been very close. And now here’s Google, who Elon has already started to sour on a little bit as he’s now trying to hire AI researchers.

You’ve got Alan Eustace flying around the world, sucking up all of the AI researchers into Google. Elon had invested in DeepMind, wanted to bring DeepMind into his own AI team at Tesla, and it got taken out from under him.

So this leads to one of the most fateful dinners in Silicon Valley’s history, organized in the summer of 2015 at the Rosewood Hotel on Sand Hill Road. Of course, where else would you do a dinner in Silicon Valley but the Rosewood? It was organized by 2 of the most leading figures in the Valley at the time, Elon Musk and Sam Altman. Sam, of course, being president of Y Combinator at the time.

So what is the purpose of this dinner? They are there to make a pitch to all of the AI researchers that Google and, to a certain extent, Facebook have sucked up and basically created a duopoly around.

David Rosenthal

Again, Google’s business model and Facebook’s business model—feed recommenders or these classifiers—turn out to be unbelievably valuable. So they can, it’s funny in hindsight saying this, pay tons of money to these people—

Ben Gilbert

Tons of money, like millions of dollars—

David Rosenthal

—take them out of academia and put them into their dirty capitalist research labs inside the companies—

Ben Gilbert

—selling advertising.

David Rosenthal

Yes.

Ben Gilbert

How dirty could you be? And the question and the pitch that Elon and Sam have for these researchers gathered at this dinner is, what would it take to get you out of Google? For you to leave?

The answer they get, going around the table from almost everybody, is nothing. You can’t. Why would we leave? We’re getting paid way more money than we ever imagined. Many of us get to keep our academic positions and affiliations, and we get to hang out here at Google—

David Rosenthal

—with each other.

Ben Gilbert

With each other.

David Rosenthal

Iron sharpens iron. These are some of the best minds in the world getting to do cutting-edge research with an enormous amount of resources and hardware at their disposal.

Ben Gilbert

It’s amazing.

David Rosenthal

It’s the best infrastructure in the world. We’ve got Jeff Dean here. There is nothing you could tell us that would cause us to leave Google.

Ben Gilbert

Except there’s one person who is intrigued. To quote from an amazing Wired article at the time by Cade Metz, who would later write Genius Makers, right?

David Rosenthal

Yep. Exactly. The quote is: “The trouble was so many of the people most qualified to solve these problems were already working for Google. And no one at the dinner was quite sure that these thinkers could be lured into a new startup even if Musk and Altman were behind it. But one key player was at least open to the idea of jumping ship.”

And then there’s a quote from that key player: “I felt like there were risks involved, but I also felt like it would be a very interesting thing to try.”

Ben Gilbert

It’s the most Ilya quote of all time, because that person was Ilya Sutskever, of course, of AlexNet, DNNResearch, and Google, and about to become founding chief scientist of OpenAI.

So, the pitch that Elon and Sam are making to these researchers is: let’s start a new nonprofit AI research lab where we can do all this work out in the open. You can publish, free of the forces of Facebook and Google and independent of their control.

David Rosenthal

Yes. You don’t have to work on products. You can only work on research. You can publish your work. It will be open. It will be for the good of humanity. All of these incredible advances, this intelligence that we believe is to come, will be for the good of everyone, not just for Google and Facebook.

Ben Gilbert

And for one of the researchers, it seemed too good to be true. So they weren’t doing it because they didn’t think anyone else would do it. It’s an activation energy problem: once Ilya said, “Okay, I’m in,” Google came back with a big counter, something like double the offer. I think it was delivered personally by Jeff Dean, and Ilya said, “Nope, I’m doing this.” That was massive for getting the rest of the top researchers to go with him.

David Rosenthal

And it was nowhere near all of the top researchers who left Google to do this, but it was enough. It was a group of 7 or so researchers who left Google and joined Elon and Sam and Greg Brockman from Stripe, who came over to create OpenAI, because that was the pitch: we’re all going to do this in the open.

Ben Gilbert

And that’s totally what it was.

David Rosenthal

It totally is what it was. And the stated mission of OpenAI was to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return,” which is fine as long as the thing that you need to fulfill your mission doesn’t take tens of billions of dollars.

Ben Gilbert

Yes.

David Rosenthal

So here’s how they would fund it. Originally, there was $1 billion pledged.

Ben Gilbert

Yes.

David Rosenthal

And that came from, famously, Elon Musk, Sam Altman, Reid Hoffman, Jessica Livingston, who I think most people don’t realize was part of that initial tranche, and Peter Thiel.

Ben Gilbert

Yep.

David Rosenthal

Founders Fund, of course, would go on to put massive amounts of money into OpenAI itself later as well.

Ben Gilbert

The funny thing is, it was later reported that $1 billion was not actually collected. Only about $130 million of it was actually collected to fund this nonprofit. For the first few years, that was plenty for the type of research they were doing and the type of compute they needed.

David Rosenthal

Most of that money was going to paying salaries to the researchers. Not as much as they could make at Google and Facebook, but still $1 million or $2 million for these folks.

Ben Gilbert

Right? And, yeah, so that really worked until it really didn’t.

Yeah. So, David, what were they doing in the early days?

David Rosenthal

Well, in the first days, it was all hands on deck, recruiting and hiring researchers. There was the initial crew that came over, and then pretty quickly after that, in early 2016, they got a big win when Dario Amodei left Google, came over, and joined Ilya and the crew at OpenAI. Dream team, assembling here. And was he on Google Brain before this?

Ben Gilbert

He was on Google Brain. Yep. And he, along with Ilya, would run large parts of OpenAI for the next couple of years before, of course, leaving to start Anthropic.

But we’re still a couple of years away from Anthropic, Claude, ChatGPT, Gemini—everything today. For at least the first year or two, basically the plan at OpenAI was: let’s look at what’s happening at DeepMind and show the research community that we can, as a new lab, do the same incredible things that they’re doing, and maybe even do them better.

David Rosenthal

Is that why it looks so game-like and game-focused?

Ben Gilbert

Yes. Yes. So they started building models to play games. Famously, the big one that they did was Dota 2, Defense of the Ancients 2, the massively multiplayer online battle arena video game. They were like, “All right, DeepMind, you’re playing StarCraft. Well, we’ll go play Dota 2. That’s even more complex, more real-time.”

David Rosenthal

And similar to the emergent properties of Go, the game would devise unique strategies that you wouldn’t see humans trying. So it clearly wasn’t humans coding their favorite strategies and rules in; it was emergent.

Ben Gilbert

Yeah.

David Rosenthal

They did other things. They had a product called Universe, which was around training computers to play thousands of games, from Atari games to open-world games like Grand Theft Auto. They had something where they were teaching a model how to do a Rubik’s Cube. So it was a diverse set of projects that didn’t seem to coalesce around one of these being the big thing.

Ben Gilbert

Yeah. It was research stuff.

David Rosenthal

Yeah. It was like university research. It was like DeepMind.

Ben Gilbert

And if you think back to Elon being an investor in DeepMind, being really upset about Google acquiring it out from under him, it makes sense.

I think Elon deserves a lot of credit for having his name and his time attached to OpenAI at the beginning. A lot of the big heavy-hitter recruiting was Elon throwing his weight behind this: I’m willing to take a chance.

David Rosenthal

Absolutely.

Ben Gilbert

Okay. So that’s what’s going on over at OpenAI, doing a lot of DeepMind-like stuff—a bunch of projects, not one single obvious big thing they’re coalescing around. It’s not ChatGPT time, let’s put it that way.

Let’s go back to Google, because last we sort of checked in on them, they’d bought DeepMind, but they’d had their talent raided. And I don’t want you to get the wrong impression about where Google is sitting just because some people left to go to OpenAI.

David Rosenthal

So, back in 2013, when Alex Krizhevsky arrives at Google with Geoffrey Hinton and Ilya Sutskever, he was shocked to discover that all their existing machine-learning models were running on CPUs. People had asked in the past for GPUs, since machine-learning workloads were well suited to run in parallel, but Google’s infrastructure team had pushed back and said the added complexity of expanding and diversifying the fleet wasn’t worth it. Let’s keep things simple. That doesn’t seem important for us.

Ben Gilbert

We’re a CPU shop here.

David Rosenthal

Yes. And so, to quote from Genius Makers, in his first days at the company, he went out and bought a GPU machine—this is Alex—from a local electronics store, stuck it in the closet down the hall from his desk, plugged it into the network, and started training his neural networks on this lone piece of hardware, just like he did in academia, except this time Google was paying for the electricity.

Obviously, 1 GPU was not sufficient, especially as more Googlers wanted to start using it, too. Jeff Dean and Alan Eustace had also come to the conclusion that DistBelief, while amazing, had to be rearchitected to run on GPUs and not CPUs.

Ben Gilbert

Who we haven’t talked about this episode.

David Rosenthal

Yeah, John Giannandrea.

Ben Gilbert

Yes, you might be wondering, “Wait, isn’t that the Apple guy?” Yes. He went on to be Apple’s head of AI, but at this point in time he was at Google and oversaw Google Brain in 2014.

David Rosenthal

They sat down to make a plan for how to actually formally put GPUs into the fleet of Google’s data centers, which is a big deal. It’s a big change, but they were seeing enough reactions to neural networks that they knew to do this.

Ben Gilbert

Yeah. After AlexNet, it’s just a matter of time.

David Rosenthal

Yeah. So they settled on a plan to order 40,000 GPUs from Nvidia.

Ben Gilbert

Yeah, of course. Who else are you going to order them from?

David Rosenthal

For a cost of $130 million. That’s a big enough price tag that the request got elevated to Larry Page, who personally approved it, even though finance wanted to kill it, because he said, “Look, the future of Google is deep learning.”

As an aside, let’s look at Nvidia at the time. This is a giant, giant order. Their total revenue was $4 billion. This is one order for $130 million.

Ben Gilbert

I mean, Nvidia was primarily a consumer graphics-card company at this point.

David Rosenthal

Yes, and their market cap was $10 billion. It’s almost like Google gave Nvidia a secret: not only does this work in research, like the ImageNet competition, but neural networks are valuable enough to us as a business to make a $100-plus-million investment in right now, no questions asked. We’ve got to ask Jensen about this at some point. This had to be a tell.

Ben Gilbert

This had to really give Nvidia the confidence: “Oh, we should way-forward invest on this being a giant thing in the future.”

David Rosenthal

So all of Google woke up to this idea. They started really putting it into their products. Google Photos happened. Gmail started offering typing suggestions. David, as you pointed out earlier, Google’s giant AdWords business started finding more ways to make more money with deep learning. In particular, when they integrated it, they could start predicting what ads people would click in the future.

And so Google started spending hundreds of millions more on GPUs on top of that $130 million, but very quickly paying it back from their ad system.

Ben Gilbert

So it became more and more of a no-brainer to just buy as many GPUs as they possibly could. But once neural nets started to work, anyone using them, especially at Google scale, had this problem: Now we need to do giant amounts of matrix multiplications anytime anybody wants to use one. Matrix multiplications are effectively how you do that propagation through the layers of the neural network. So you have this problem.

David Rosenthal

Yes, totally. There’s the inefficiency of it, but then there’s also the business problem of, wait a minute, it looks like we’re just going to be shipping hundreds of millions, soon to be billions, of dollars over to NVIDIA every year for the foreseeable future.

Ben Gilbert

Right? So there’s this amazing moment right after Google rolls out speech recognition, their latest use case for neural nets, just on Nexus phones, because again, they don’t have the infrastructure to support it on all Android phones. It becomes a super popular feature, and Jeff Dean does the math and figures out, if people use this for, I don’t know, 3 minutes a day, and we roll it out to all 1 billion Android phones, we’re going to need twice the number of data centers that we currently have across all of Google just to handle it.

David Rosenthal

Just for this feature, yeah.

Ben Gilbert

There’s a great quote where Jeff goes to Urs Hölzle and goes, “We need another Google.” Or, David, as you were hinting at, the other option is we build a new type of chip customized for just our particular use case.

David Rosenthal

Yep. Matrix multiplication, tensor multiplication—a tensor processing unit, you might say.

Ben Gilbert

Ah, yes. Wouldn’t that be nice? So, conveniently, Jonathan Ross, who’s an engineer at Google, has been spending his 20% time at this point in history working on an effort involving FPGAs. These are essentially expensive but programmable chips that yield really fantastic results.

So they decide to create a formal project to take that work, combine it with some other existing work, and build a custom ASIC, or an application-specific integrated circuit. So enter, David, as you said, the tensor processing unit, made just for neural networks, that is far more efficient than GPUs at the time, with the trade-off that you can’t really use it for anything else. It’s not good for graphics processing. It’s not good for lots of other GPU workloads—just matrix multiplication and just neural networks—but it would enable Google to scale their data centers without having to double their entire footprint.

So the big idea behind the TPU, if you’re trying to figure out what the core insight was, is that they used reduced computational precision. So it would take numbers like 4,586.8272 and round them just to 4,586.8, or maybe even just 4,586, with nothing after the decimal point. And this sounds kind of counterintuitive at first. Why would you want less precise, rounded numbers for this complicated math?

The answer is efficiency. If you can do the heavy lifting in your software architecture, or what’s called quantization, to account for it, you can store information as less precise numbers. Then you can use the same amount of power and the same amount of memory and transistors on a chip to do far more calculations per second. So you can either spit out answers faster or use bigger models. The whole thing behind the TPU is quite clever.

David Rosenthal

The other thing that has to happen with the TPU is that it needs to happen now, because it’s very clear speech-to-text is a thing. It’s very clear some of these other use cases at Google—

Ben Gilbert

Yeah. Demand for all of this stuff that’s coming out of Google Brain is through the roof immediately.

David Rosenthal

Right. And we’re not even to LLMs yet. It’s just like everyone sort of expects some of this, whether it’s computer vision in Photos or speech recognition. It’s just becoming a thing that we expect, and it’s going to flip Google’s economics upside down if they don’t have it.

Ben Gilbert

So the TPU was designed, verified, built, and deployed into data centers in 15 months.

David Rosenthal

Wow.

Ben Gilbert

It was not a research project that could just happen over several years. This was a hair-on-fire problem that they launched immediately. One very clever thing that they did was use the FPGAs as a stopgap. So even though they were too expensive on a unit basis, they could get them out as a test fleet and just make sure all the math worked before they actually had the ASIC printed—I don’t know if it was at TSMC—but fabbed and ready.

The other thing they did is they fit the TPU into the form factor of a hard drive, so it could actually slot into the existing server racks. You just pop out a hard drive and pop in a TPU without needing to do any physical rearchitecture.

David Rosenthal

Wow, that’s amazing. That’s the most Google-y infrastructure story—

Ben Gilbert

Since the corkboards.

David Rosenthal

Exactly.

Ben Gilbert

Also, all of this didn’t happen in Mountain View. It was at a Google satellite office in Madison, Wisconsin.

David Rosenthal

Whoa.

Ben Gilbert

Yes.

David Rosenthal

Why Madison, Wisconsin?

Ben Gilbert

There was a particular professor out of the university, and there were a lot of students that they could recruit from.

David Rosenthal

Wow.

Ben Gilbert

Yeah. I mean, it was probably them or Epic. Where are you going to go work?

David Rosenthal

Yeah.

Ben Gilbert

Wow. They also then just kept this a secret.

David Rosenthal

Right? Why would you tell anybody about this?

Ben Gilbert

Because it’s not like they were offering these in Google Cloud, at least at first, and why would you want to tell the rest of the world what you’re doing? So the whole thing was a complete secret for at least a year before they announced it at Google I/O. So, really crazy.

The other thing to know about the TPUs is they were done in time for the AlphaGo match. That match ran on a single machine with 4 TPUs in Google Cloud. And once that worked, obviously that gave Google a little bit of extra confidence to really, really ramp production. So that’s the TPU.

Version 1, by all accounts, was not great. They’re on Version 7 or Version 8 now. It’s gotten much better. TPUs and GPUs look a lot more similar than they used to; they’ve adopted features from each other. But today, Google, it’s estimated, has 2 to 3 million TPUs. For reference, NVIDIA shipped—people don’t know for sure—somewhere around 4 million GPUs last year.

So people talk about AI chips like it’s just a one-horse race with NVIDIA. Google has an almost NVIDIA-scale internal operation making its own chips at this point, both for its own use and for Google Cloud customers. The TPU is a giant deal in AI in a way that I think a lot of people don’t realize.

David Rosenthal

Yep. This is one of the great ironies, and maddening things for OpenAI and Elon Musk, is that OpenAI gets founded in 2015 with the goal of, “Hey, let’s shake all this talent out of Google and level the playing field,” and Google just accelerates.

Ben Gilbert

Right? They also build TensorFlow. That’s the framework that Google Brain built to enable researchers to build, train, and deploy machine learning models. And they built it in such a way that it doesn’t just have to run on TPUs. It’s super portable, without any rewrites, to run on GPUs or even CPUs, too.

So this would replace the old DistBelief system and kind of be their internal and external framework for enabling ML researchers going forward. So, somewhat paradoxically, during these years after the founding of OpenAI, yes, some amazing researchers are getting siphoned off from Google and Google Brain, but Google Brain is also firing on all cylinders during this timeframe.

David Rosenthal

Delivering on the business purposes for Google left and right.

Ben Gilbert

Yes, and pushing the state of the art forward in so many areas. And then, in 2017, a paper gets published from 8 researchers on the Google Brain team, kind of quietly. These 8 folks were obviously very excited about the paper and what it described and the implications of it, and they thought it would be very big. Google itself: “Cool, this is like the next iteration of our language-model work. Great.”

David Rosenthal

Which is important to us. But are we sure this is the next Google? No.

Ben Gilbert

No. There are a whole bunch of other things we’re working on that seem more likely to be the next Google. But this paper and its publication would actually be what gave OpenAI the opportunity—

David Rosenthal

To build the next Google.

Ben Gilbert

To grab the ball and run with it and build the next Google, because this is the Transformer paper.

David Rosenthal

Okay. So where did the Transformer come from? What was the latest thing that language models had been doing at Google?

Ben Gilbert

So, coming out of the success of Franz Och’s work on Google Translate and the improvements that happened there—

David Rosenthal

In, like, the late 2000s—2007.

Ben Gilbert

Yeah, mid-to-late 2000s. They keep iterating on Translate, and then once Geoffrey Hinton comes on board and AlexNet happens, they switch over to a neural-network-based language model for Translate, which was dramatically better and a big, crazy cultural thing.

Because you’ve got these researchers parachuting in, again, led by Jeff Dean, saying, “I’m pretty sure our neural networks can do this way better than the classic methods that we’ve been using for the last 10 years. What if we take the next several months and do a proof of concept?”

They end up throwing away the entire old codebase and just completely, wholesale switching to this neural network. There’s actually this great New York Times Magazine story that ran in 2016 about it. And I remember reading the whole thing with my jaw on the floor. Like, wow. Neural networks are a big effing deal. And this was the year before the Transformer paper would come out.

David Rosenthal

Before the Transformer paper.

Ben Gilbert

Yes. So they do the rewrite of Google Translate, make it based on recurrent neural networks, which were state of the art at that point in time. And it’s a big improvement. But as teams within Google Brain and Google Translate keep working on it, there are some limitations. And in particular, a big problem was that they, quote unquote, “forgot things” too quickly.

David Rosenthal

I don’t know if it’s exactly the right analogy, but you might say, in today’s Transformer-world speak, that their context window was pretty short. As these language models progressed through text, they needed to remember everything they had read so that when they needed to change a word later or come up with the next word, they could have a whole memory of the body of text to do that.

Ben Gilbert

So, one of the ways that Google tries to improve this is to use something called long short-term memory networks, or LSTMs, as the acronym that people use for this. Basically, what LSTMs do is create a persistent, or long short-term, memory. You’ve got to use your brain a little bit here for the model so that it can keep context as it’s going through a whole bunch of steps.

David Rosenthal

And people were pretty excited about LSTMs at first.

Ben Gilbert

People were thinking, “Oh, LSTMs are what’s going to take language models and large language models mainstream,” right?

David Rosenthal

And indeed, in 2016, they incorporated these LSTMs into Google Translate. It reduced the error rate by 60%. Huge jump. Yep.

Ben Gilbert

The problem with LSTMs, though, was that they were effective but very computationally intensive, and they didn’t parallelize that well. All the efforts coming out of AlexNet and then the TPU project were about parallelization. This is the future; this is how we’re going to make AI really work. LSTMs are a bit of a roadblock here.

David Rosenthal

Yes. So, a team within Google Brain starts searching for a better architecture that also has the attractive properties of LSTMs—that it doesn’t forget context too quickly—but can parallelize and scale better.

Ben Gilbert

To take advantage of all these new architectures.

David Rosenthal

Yes. And a researcher named Jakob Uszkoreit had been toying around with the idea of broadening the scope of “attention” in language processing. What if, rather than focusing on the immediate words, you told the model, “Hey, pay attention to the entire corpus of text, not just the next few words. Look at the whole thing.” And then, based on that entire context and giving your attention to the entire context, give me a prediction of what the next translated word should be.

Now, by the way, this is actually how professional human translators translate text. You don’t just go word by word. I actually took a translation class in college, which was really fun. You read the whole original in the original language, get and understand the context of what the original work is, and then go back and start to translate it with the entire context of the passage in mind.

Ben Gilbert

So, it would take a lot of computing power for the model to do this, but it is extremely parallelizable. So Jakob starts collaborating with a few other people on the Brain team. They get excited about this. They decide that they’re going to call this new technique the Transformer because, one, that is literally what it’s doing: it’s taking in a whole chunk of information, processing and understanding it, and then transforming it. And, B, they also loved Transformers as kids. That’s not why they named it the Transformer.

David Rosenthal

And it’s taking in the giant corpus of text and storing it in a compressed format. Right.

Ben Gilbert

Yeah. I bring this up because that is exactly how you pitched the microkitchen conversation with Noam Shazeer in 2000, 17 years earlier, who is a co-author on this paper.

David Rosenthal

Yes. Well, speaking of Noam Shazeer, he learns about this project and decides, “Hey, I’ve got some experience with this. This sounds pretty cool. LSTMs definitely have problems. This could be promising. I’m going to jump in and work on it with these guys.”

And it’s a good thing he did, because before Noam joined the project, they had a working implementation of the Transformer, but it wasn’t actually producing any better results than LSTMs. Noam joins the team, basically pulls a Jeff Dean, rewrites the entire codebase from scratch, and when he’s done, the Transformer now crushes the LSTM-based Google Translate solution.

And it turns out that the bigger they make the model, the better the results get. It seems to scale really, really, really well. Steven Levy wrote a piece in Wired about the history of this. There are all sorts of quotes from the other members of the team just littered all over the piece, with things like, “Noam is a magician. Noam is a wizard.” Noam took the idea and came back and said, “It works now.”

Ben Gilbert

Yeah. And you wonder why Noam and Jeff Dean are the ones working together on the next version of Gemini now.

David Rosenthal

Yes. Noam and Jeff Dean are definitely two peas in a pod here.

Ben Gilbert

Yes. So, we talked to Greg Corrado from Google Brain, one of the founders of Google Brain, and it was a really interesting conversation because he underscored how elegant the Transformer was. He said it was so elegant that people’s response was often, “This can’t work. It’s too simple. Transformers are barely a neural network architecture,” right? It was another big change from the AlexNet–Geoffrey Hinton lineage of neural networks.

David Rosenthal

Yeah, it actually changed the way that I look at the world because he pointed out that in nature—this is Greg—the way things usually work is the most energy-efficient way they could work, almost from an evolution perspective. The simplest, most elegant solutions are the ones that survive because they are the most efficient with their resources.

And you can port this idea over to computer science, too. He said he’s developed a pattern recognition inside the research lab to realize that you’re probably onto the right solution when it’s really simple and really efficient versus a complex idea.

Ben Gilbert

Mhm.

David Rosenthal

It’s very clever. I think it’s very true. You know how, when you sit around and have a thorny problem, you debate and whiteboard, and then you’re like, “Oh my God, oh my God, it’s so simple.” And that ends up being the right answer.

Ben Gilbert

Yeah. There’s an elegance to the Transformer.

David Rosenthal

Yes. And that other thing that you touched on there: this is the beginning of modern AI. Just feed it more data. The famous piece “The Bitter Lesson” by Rich Sutton wouldn’t be published until 2019. For anyone who hasn’t read it, it’s basically this: We always think, as AI researchers, that we’re so smart and that our job is to come up with another great algorithm, but effectively, in every field from language to computer vision to chess, you just figure out a scalable architecture, and then the more data wins. Just these infinitely scaling—

Ben Gilbert

More data, more compute, better results.

David Rosenthal

Yes. And this is really the start of when that begins to be, like, “Oh, we have found the scalable architecture that will go on for, I don’t know, close to a decade of just more data, more energy, more compute, better results.”

Ben Gilbert

So, the team and Noam are like, “Yo, this thing has a lot, a lot of potential.”

David Rosenthal

This is more than better Google Translate. We can really apply this.

Ben Gilbert

Yeah, this is going to be more than better Google Translate. The rest of Google, though, is definitely slower to wake up to the potential.

David Rosenthal

They build some stuff within a year. They build BERT, the large language model.

Ben Gilbert

Yes, absolutely true. It is a false narrative out there that Google did nothing with the Transformer after the paper was published. They actually did a lot.

David Rosenthal

In fact, BERT was one of the first LLMs.

Ben Gilbert

Yes, they did a lot with Transformer-based large language models after the paper came out. What they didn’t do was treat it as a wholesale technology-platform change, right? They were doing things like BERT and MUM, this other model. They could work it into search-results quality. And I think that did meaningfully move the needle, even though Google wasn’t bragging about it and talking about it.

They got better at query comprehension. They were working it into the core business, just like every other time Google Brain came up with something great.

David Rosenthal

Yep. So, in perhaps one of the greatest decisions ever for value to humanity, and maybe one of the worst corporate decisions ever for Google, Google allows this group of 8 researchers to publish the paper under the title “Attention Is All You Need,” obviously a nod to the classic Beatles song about love.

As of today in 2025, this paper has been cited over 173,000 times in other academic papers, making it currently the seventh most-cited paper of the 21st century. And I think all of the other papers above it on the list have been out much longer.

Ben Gilbert

Wow. And also, of course, within a couple of years, all 8 authors of the Transformer paper had left Google to either start or join AI startups, including OpenAI. Brutal.

And, of course, Noam started Character.AI—which, what are we calling it? Acquisition. He would end up back at Google via some strange licensing, IP, and hiring agreement on the few-billion-dollar order. Very, very expensive mistake on Google’s part.

David Rosenthal

It is fair to say that 2017 begins the 5-year period of Google not sufficiently seizing the opportunity that they had created with the Transformer.

Ben Gilbert

Yes. So, speaking of seizing opportunities, what is going on at OpenAI during this time?

David Rosenthal

And does anyone think the Transformer is a big deal over there?

Ben Gilbert

Yes. Yes, they did. But here’s where history gets really, really crazy. Right after Google publishes the Transformer paper in September of 2017, Elon gets really, really fed up with what’s going on at OpenAI.

David Rosenthal

There are, like, 7 different strategies. Are we doing video games? Are we doing competitions? What’s the plan?

Ben Gilbert

What is happening here? As best as I can tell, all you’re doing is just trying to copy DeepMind. Meanwhile, I’m here building SpaceX and Tesla. Self-driving is becoming more and more clear as critical to the future of Tesla. I need AI researchers here, and I need great AI advancements to come out to help what we’re doing at Tesla.

David Rosenthal

OpenAI isn't cutting it. So he makes an ultimatum to Sam Altman and the rest of the OpenAI board. He says, “I'm happy to take full control of OpenAI, and we can merge this into Tesla. I don't even know how that would be possible, to merge a nonprofit into Tesla.”

Ben Gilbert

But in Elon Land, if he takes over as CEO of OpenAI, it almost doesn't matter. We're just treating it as if it's the same company anyway, just like we do with the deals with all of my companies, right?

David Rosenthal

Or he's out completely, along with all of his funding. And Sam and the rest of the board are like, “No.”

Ben Gilbert

And as we know now, they're sort of calling capital into the business. It's not like they actually got all the cash up front, right? So they're only 130 million-ish into the $1 billion commitment.

David Rosenthal

They don't reach a resolution, and by early 2018, Elon is out, along with the main source of OpenAI's funding. So either this is just a really, really, really bad misjudgment by Elon, or the sort of panic that this throws OpenAI into is the catalyst that makes them reach for the Transformer and say, “All right, we've got to figure things out. Necessity is the mother of invention. Let's go for it.”

Ben Gilbert

It's true. I don't know if, during this personal tension between Elon and Sam, they had already decided to go all-in on Transformers or not, because the thing you very quickly get to if you decide, “Transformer language models—we're going all-in on that”—you do quickly realize you need a bunch of data, a bunch of compute, a bunch of energy, and a bunch of capital. And so, if your biggest backer is walking away, the 3D-chess move is, “Oh, we've got to keep him because we're about to pivot the company, and we need his capital for this big pivot we're doing.” The 4D-chess move is, if he walks away, maybe I can turn it into a for-profit company, then raise money into it, and eventually generate enough profits to fund this extremely expensive new direction we're going in. I don't know which of those it was.

David Rosenthal

Yeah, I don't know either. I suspect the truth is some of both.

Ben Gilbert

Yes. But either way, how nuts is it that, A, these things happened at the same time, and B, the company wasn't burning that much cash, and then they decided to go all-in on, “We need to do something so expensive that we need to be a for-profit company in order to actually achieve this mission,” because it's just going to require hundreds of billions of dollars for the foreseeable future.

David Rosenthal

Yep. So in June of 2018, OpenAI releases a paper describing how they have taken the Transformer and developed a new approach of pre-training it on very large amounts of general text on the internet and then fine-tuning that general pre-training to specific use cases. They also announce that they have trained and run the first proof-of-concept model of this approach, which they are calling GPT-1, Generative Pre-trained Transformer 1.

Ben Gilbert

Which we should say is right around the same time as BERT and right around the same time as another large language model based on the Transformer out of here in Seattle, the Allen Institute.

David Rosenthal

Yes, indeed. So it's not as if this is heretical or a secret. Other AI labs, including Google's own, are doing it. But from the very beginning, OpenAI seemed to be taking this more seriously, given that the cost of it would require betting the company if they continued down this path.

Ben Gilbert

Yeah. Or betting the nonprofit, betting the entity.

David Rosenthal

Yes.

Ben Gilbert

We're going to need some new terminology here.

David Rosenthal

Yes.

Ben Gilbert

So Elon's just walked out the door. Where are they going to get the money for this? Sam turns to one of the other board members of OpenAI, Reid Hoffman. Reid, just a year or so earlier, had sold LinkedIn to Microsoft, and Reid is now on the board of Microsoft. So Reid says, “Hey, why don't you come talk to Satya Nadella about this?”

David Rosenthal

Do you know where he actually talks to Satya Nadella?

Ben Gilbert

Oh, I do. Oh, I do. In July of 2018, they set a meeting for Sam Altman and Satya Nadella to sit down while they're both at the Allen & Company Sun Valley Conference in Sun Valley, Idaho.

David Rosenthal

It's perfect.

Ben Gilbert

And while they're there, they hash out a deal for Microsoft to invest $1 billion into OpenAI in a combination of both cash and Azure cloud credits. And in return, Microsoft will get access to OpenAI's technology, get an exclusive license to OpenAI's technology for use in Microsoft's products. And the way that they will do this is OpenAI, the nonprofit, will create a captive for-profit entity called OpenAI LP, controlled by the nonprofit OpenAI Inc., and Microsoft will invest into the captive for-profit entity. Reid Hoffman joins the board of this new structure, along with Sam Altman, Ilya Sutskever, Greg Brockman, Adam D'Angelo, and Tasha McCauley. And thus, the modern OpenAI for-profit/nonprofit question mark is created.

David Rosenthal

The thing that's still being figured out even today, here in 2025, is created. This is like the complete history of AI. This is not just the Google AI episode.

Ben Gilbert

Well, these things are totally inextricable. And I was just going to say, this is the Google Part III episode. Microsoft, they're back. Microsoft is Google's mortal enemy. Yes. In our first episode on the founding of Google and search, and then in the second episode on Alphabet and all the products that they made, the whole strategy at Google was always about Microsoft. They finally beat them on every single front, and here they are—

David Rosenthal

—showing up again, saying, “What was Satya's line? We just want to see them dance.” I think the line that would come a couple of years later is, “We want the world to know that we made Google dance.” Oh, man.

Ben Gilbert

But this is all still pre-ChatGPT. This is just Sam lining up the financing he needs for what appears to be a very expensive scaling exercise they're about to embark on with GPT-2 and onward.

David Rosenthal

Yep. And this is the right time to talk about why, from OpenAI's perspective, Microsoft is the absolute perfect partner. It's not just that they have a lot of money—

Ben Gilbert

—although that helps.

David Rosenthal

I mean, that helps. That helps a lot. But more important than money, they have a really, really great public cloud: Azure.

Ben Gilbert

Yes. OpenAI is not going to go buy a bunch of NVIDIA GPUs and then build their own data center here at this point in 2018. That's not the scale of company that they are. They need a cloud provider in order to actually do all the compute that they want to do. If they were back at Google and these researchers were doing it, great. Then they have all the infrastructure. But OpenAI needs to tie themselves to someone with the infrastructure.

David Rosenthal

And there's basically only 2 non-Google options. They're both in Seattle. And hey, one of them, Microsoft, is really interested and also has a lot of cash. It seems like a great partnership.

Ben Gilbert

That's true. I wonder if they did talk to AWS at all about it, because I think this is a crazy Easter egg. I hesitate to say it out loud, but I think AWS was actually in the very first investment with Elon in OpenAI.

David Rosenthal

Oh, wow. And I don't know if it was in the form of credits or what the deal was, but I'd seen it reported in a couple of places that AWS actually was in that nonprofit round.

Ben Gilbert

Yeah, in the nonprofit funding, the donations to—

David Rosenthal

Yes.

Ben Gilbert

—the early OpenAI.

David Rosenthal

Anyway, Microsoft and OpenAI, they end up tying up—

Ben Gilbert

—a match made in heaven. Satya and Sam are onstage together, talking about how this amazing partnership and marriage has come together, and they're off to model training.

David Rosenthal

Yeah. And this paves the way for the GPT era of OpenAI. But before we tell that story,

Ben Gilbert

All right. So what are we in GPT-2? Is that what's being trained right here?

David Rosenthal

Yes, GPT-2.

Ben Gilbert

This was the first time I heard about it. Data scientists around Seattle were talking about this cool thing.

David Rosenthal

Right? So, after the first Microsoft partnership and the first billion-dollar investment in 2019, OpenAI releases GPT-2, which is still early but very promising and can do a lot of things.

Ben Gilbert

A lot of things, but it required an enormous amount of creativity on your part. You had to be a developer to use it. If you were a consumer, there was a very heavy load put on you. You had to write a few paragraphs and then paste those paragraphs into the language model, and then it would suggest a way to finish what you were writing based on the source paragraphs. But it wasn't interactive.

David Rosenthal

Yes, it was not a chat interface.

Ben Gilbert

Yes.

David Rosenthal

There was no interface, essentially, for it.

Ben Gilbert

It was an API, but it could do things like obviously translate text. I mean, Google had been doing that for a long time, but with GPT-2, you could do stuff like make up a fake news headline and give it to GPT-2, and it would write a whole article. You would read it and you'd be like, “Sounds like it was written by a bot.”

David Rosenthal

Yeah.

Ben Gilbert

But again, there was no front door to it for normal people. You had to be really willing to wait in the muck to use this thing.

David Rosenthal

So then, the next year, in June 2020, GPT-3 comes out. Still no front door or user interface to the model, but it's very good. GPT-2 showed the promise of what was possible. With GPT-3, it's starting to be in the conversation of, can this thing pass the Turing test?

Ben Gilbert

Oh, yeah.

David Rosenthal

You have a hard time distinguishing between articles that GPT wrote and articles that humans wrote. It's very good. There starts to be a lot of hype around this thing. Even though consumers aren't really using it, the broader awareness is that there's something interesting on the horizon. I think the number of AI pitch decks that VCs are seeing is starting to tick up around this time, as is NVIDIA's stock price.

Ben Gilbert

Yes.

David Rosenthal

So then, in the next year, in the summer of 2021, Microsoft releases GitHub Copilot using GPT-3. This is the first not just Microsoft product that comes out with GPT baked into it, but the first—

Ben Gilbert

Productization.

David Rosenthal

—product anywhere. Yeah, first productization of GPT.

Ben Gilbert

Yes, of any OpenAI technology.

David Rosenthal

Yeah, it's big. This starts a massive change in how software gets written in the world.

Ben Gilbert

Slowly, then all at once. It's one of these things where, at first, just a few software engineers were whispering about how cool this was. It made them a little bit more efficient. And now you get all these comments like, “75% of all companies' code is written with AI.”

David Rosenthal

Yep. So after that, Microsoft invests another $2 billion in OpenAI, which seemed like a lot of money at the time. So that takes us to the end of 2021. There's an interesting context shift that happens around here.

Ben Gilbert

Yeah. The bottom falls out on tech stocks, crypto, and the broader markets. Really, everyone suddenly goes from risk-on to risk-off. Part of it was the war in Ukraine, but a lot of it was interest rates going up. Google gets hit really hard. The high-water mark was November 19, 2021. Google was right at $2 trillion of market cap. About a year after that slide began, they were worth $1 trillion. Nearly a 50% drawdown.

David Rosenthal

Wow. So toward the end of 2022, leading up to the launch of ChatGPT—

Ben Gilbert

People, I think, are starting to realize Google is slow. They're slow to react to things. It feels like they're an old, crusty company. Are they like Microsoft in the 2000s, where they haven't had a breakthrough product in a while? People are not bullish on the future of Google, and then ChatGPT comes out.

David Rosenthal

Yeah. Wow. Which means if you were bullish on Google back then and contrarian, you could have invested at a $1 trillion market cap.

Ben Gilbert

Which is interesting. In October 2021, the market was saying that the forthcoming AI wave would not be a strength for Google. Or maybe what it was saying is, we don't even know anything about a forthcoming AI wave, because people are talking about AI, but they've been talking about VR, and they've been talking about crypto, and they've been talking about all this frontier tech, and that's not the future at all. This company just feels slow and unadaptive. Slow and unadaptive at that point in history, I think, would have been a fair characterization. They had an internal chatbot, right?

David Rosenthal

Yes, they did. All right. So before we talk about ChatGPT, Google had a chatbot. Noam Shazeer, an incredible engineer, re-architected the transformer and made it work. He was one of the lead authors of the paper and had a storied career within Google. He had all of this sway—should have had all of this sway—within the company.

After the transformer paper comes out, he and the rest of the team are like, “Guys, we can use this for a lot more than Google Translate.” In fact, the last paragraph of the paper—

Ben Gilbert

Are you about to read the transformer paper?

David Rosenthal

Yes, I am. “We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the transformer to problems involving input and output modalities other than text and to investigate large inputs and outputs, such as images, audio, and video.” This is in the paper.

Ben Gilbert

Wow.

David Rosenthal

Google obviously does not do any of that for quite a while. Noam, though, immediately starts advocating to Google leadership: “Hey, I think this is going to be so big—the transformer—that we should actually consider just throwing out the search index and the 10 blue links model and going all in on transforming all of Google into one giant transformer model.”

And then Noam actually goes ahead and builds a chatbot interface to a large transformer model.

Ben Gilbert

Is this LaMDA?

David Rosenthal

This is before LaMDA. Meena is what he calls it.

Ben Gilbert

And there is a chatbot in the late-teens-to-2020 timeframe that Noam has built within Google that arguably is pretty close to ChatGPT. Now, it doesn't have any of the post-training safety that ChatGPT does, so it would go off the rails.

David Rosenthal

Yeah. Someone told us that you could just ask it who should die, and it would come up with names for you of people who should die. It was not a shippable product. It was a very raw, unsafe, un-post-trained chatbot and model.

Ben Gilbert

Right? But it existed within Google, and they didn't ship it.

David Rosenthal

And technically, not only did it not have post-training, it didn't have RLHF either. This is a very core component of the models today: reinforcement learning with human feedback. ChatGPT—I don't know if it had it in GPT-3, but it did in GPT-3.5, and it did for the launch of ChatGPT. Realistically, it wasn't launchable, even if it was an OpenAI thing, because it was so bad. But a company of Google's stature certainly could not take the risk.

Ben Gilbert

So strategically, they have this working against them. But aside from the strategy thing, there are 2 business-model problems here. One, if you're proposing dropping the 10 blue links and just turning Google.com into a giant AI chatbot, revenue drops when you provide direct answers to questions versus showing advertisers and letting people click through to websites. That upsets the whole apple cart. Obviously, they're thinking about it now, but until 2021, that was an absolute nonstarter to suggest something like that.

Two, there were legal risks of sitting in between publishers and users. Google, at this point, had spent decades fighting the public perception and court rulings that they were disintermediating publishers from readers. So there was a very high bar internally and culturally to clear if you were going to do something like this. Even those info boxes that popped up—it took until the 2010s to make that happen—were mostly on non-monetizable queries anyway. So anytime you were going to say, “Hey, Google's going to provide you an answer instead of 10 blue links,” you had to have a bulletproof case for it.

David Rosenthal

Yeah. And there was also a brand-promise and trust issue, too. Consumers trusted Google so much. For us, even today—you know, when I'm doing research for Acquired, we need to make sure we get something right—I'm going to Google.

Ben Gilbert

I look something up in Claude. Yeah.

David Rosenthal

It gives me an answer. I'm like, “That's a really good answer.” And then I verify by searching Google that I can find those facts, too, if I can't click through the sources on Claude. That's my workflow.

Ben Gilbert

Which sounds funny today, but it's important. If you're going to propose replacing the 10 blue links with a chatbot, you need to be really damn sure that it's going to be accurate.

David Rosenthal

Yes.

Ben Gilbert

And in 2020 and 2021, that was definitely not the case. Arguably, it still isn't the case today. And there also wasn't a compelling reason to do it, because nobody was really asking for this product.

David Rosenthal

Right?

Ben Gilbert

Noam knew, and people in Google knew, that you could make a chatbot interface to a transformer-based LLM, and that was a really compelling product. The general public didn't know. OpenAI didn't even really know. I mean, GPT-3 was out there.

David Rosenthal

Do you know the story of the launch of ChatGPT? Well, I think I do. I have it in my notes here.

Ben Gilbert

All right. So they've got GPT-3.5. It's becoming very, very useful.

David Rosenthal

Yeah, this is late 2022. They've got GPT-3.5, but there's still this problem of how am I supposed to actually use it? How is it productized?

Ben Gilbert

And Sam just says, “We should make a chatbot. That seems like a natural interface for this. Can someone just make a chat?” And within about a week internally, someone makes a chat. They just turn calls to the GPT-3.5 API into a product where you're just chatting with it. Every time you kick off a chat message, it calls GPT-3.5 through the API, and that turns out to be this magic product. I don't think they expected it. I mean, servers are tipping over.

David Rosenthal

They’re working with Microsoft to try to get more compute. They’re cutting deals with Microsoft in real time to try to get more investment, to get more Azure credits, or to get advances on their Azure credits in order to handle the incredible load in November 2022 of people wanting to use this thing.

They also just throw up a paywall randomly because they thought the business was going to be an API business. They thought the projections were all about how much revenue they were going to do through B2B licensing deals, and then they just realized, “Oh, there are all these consumers trying to use this.” Put up a paywall to at least dampen the most expensive use of this thing so we can offset the cost or slow the rollout, right?

This isn’t Google Search, you know, 89% gross-margin stuff here, right? So they end up having an incredibly fast revenue takeoff just from the quick Stripe paywall that they threw up over a weekend to handle all the demand. To say that OpenAI had any idea what was coming would also be completely false. They did not get that this would be the next big consumer product when they launched it.

Ben Gilbert

Ben Thompson loves to call OpenAI the accidental consumer tech company, right?

David Rosenthal

Yes, it was definitely accidental. Now, there is actually another, slightly different version of the motivation for launching the chat.

Ben Gilbert

Is this the Dario interface?

David Rosenthal

Yeah, the Dario and Anthropic version. Anthropic was working on what would become Claude, and rumors were out there. People at OpenAI got wind of it: “Oh, hey, Anthropic and Dario are working on a chat interface. We should probably do one, too. And if we’re going to do one, we should probably launch it before they launch theirs.”

So I think that had something to do with the timing, but again, I don’t think anybody, including OpenAI, realized what was going to happen. Ben, you alluded to it, but to give the actual numbers, on November 30, 2022—

Ben Gilbert

Basically Thanksgiving.

David Rosenthal

OpenAI launches a research preview of an interface to the new GPT-3.5 called ChatGPT. That morning, on the 30th, Sam Altman tweets, “Today we launched ChatGPT. Try talking with it here,” and then a link to chat.

Within a week—less than a week, actually—it gets 1 million users. By the end of the year, one month later, on December 31, 2022, it has 30 million users. By the end of the next month, by the end of January 2023, so two months after launch, it crosses 100 million registered users: the fastest product in history to hit that milestone. Completely insane.

Before we talk about what that unleashes within Google, which is the famous code red, let’s rewind a little bit back to Noam and the chatbot within Google, Meena. Google does keep working on Meena. They develop it into something called LaMDA, which is also a chatbot, also internal.

Ben Gilbert

I think it was a language model. At this point in time, they still differentiated between the underlying model brand name and the application name.

David Rosenthal

Yes, LaMDA was the model, and then there was also a chat interface to LaMDA that was internal for Google use only. Noam is still advocating to leadership, “We’ve got to release this thing.” He leaves in 2021 and founds a chatbot company, Character.AI, that still exists to this day. They raise a lot of money, as you would expect.

Then Google ultimately, in 2024, after ChatGPT launches, pays $2.7 billion, I think, to do a licensing deal with Character.AI, the net of which is that Noam comes back to Google. I think Larry and Sergey were like, “If we’re going to compete seriously, we kind of need Noam back,” and gave them a blank check to go get him.

Ben Gilbert

Yeah. So throughout 2021 and 2022, Google’s working on the LaMDA model and then the chat interface to it. In May 2022, they do release something that is available to the public called AI Test Kitchen, which is an AI product test area where people can play around with Google’s internal AI products, including the LaMDA chat interface.

David Rosenthal

Yep. And in all fairness, it predates ChatGPT.

Ben Gilbert

Do you know what they do to nerf the chat so that it doesn’t go too far off the rails? This is amazing.

David Rosenthal

No. For the version of the LaMDA chat that is in AI Test Kitchen, they stop all conversations after 5 turns. You can only have 5 turns of conversation with the chatbot, and then it’s just, “We’re done for today. Thank you. Goodbye.”

Ben Gilbert

Oh, wow.

David Rosenthal

And the reason they did that was for safety. The more turns you had with it, the more likely it would start to go off the rails.

Ben Gilbert

And honestly, it was a fair concern. This thing was not for public consumption. If you remember, back a few years before, Microsoft released Tay, which was this crazy racist chatbot.

David Rosenthal

Yeah. They launched it as a Twitter bot, right? It was going off the rails on Twitter. This was in 2016, I think.

Ben Gilbert

Right. Maximal impact of badness.

David Rosenthal

Yeah. And so, despite the fact that all the way back in 2017 Sundar declared, “We are an AI-first company,” Google is understandably very cautious in real public AI launches, especially on consumer-facing things.

Yep. And as far as anyone else is concerned, before ChatGPT, they are an AI-first company and they’re launching all this amazing AI stuff. It’s just within the vector of their existing products, right?

So ChatGPT comes out, becomes the fastest product in history to reach 100 million users, and it is immediately obvious to Sundar, Larry, Sergey, and all of Google leadership that this is an existential threat to Google. ChatGPT is a better user experience for doing the same job function that Google Search does.

And to underscore this, if you didn’t know it in November of 2022, you sure knew it by February of 2023 because good old Microsoft, our biggest, scariest enemy—

Ben Gilbert

Oh, yeah.

David Rosenthal

Microsoft announces a new Bing powered by OpenAI. And Satya has a quote: “It’s a new day for search. The race starts today.” There’s an announcement of a new AI-powered search page. He says, “We want to rethink what search was meant to be in the first place. In fact, Google’s success in the initial days came by reimagining what could be done in search. And I think the AI era we’re entering gets us to think about it.”

This is the worst possible thing that could happen to Google: that now Microsoft can actually challenge Google on its own turf, the internet, with a legitimately different, better, differentiated product vector. Not what Bing was trying to do—copycat. This is the full leapfrog, and they have the technology partnership to do it.

Ben Gilbert

Or so everybody thinks at the moment.

David Rosenthal

Oh my God, terrifying.

This is when Satya says the quote in an interview around this launch with Bing: “I want people to know that we made Google dance.”

Ben Gilbert

Oh boy. Well, hey, if you come at the king, you’d best not miss, right?

David Rosenthal

And this big launch kind of misses.

Ben Gilbert

Yes. So what happens in Google in December 2022, even before the big launch but after the ChatGPT moment? Sundar issues a code red within the company.

David Rosenthal

And what does that mean?

Up until this point, Google and Sundar and Larry and everyone had been thinking about AI as a sustaining innovation, in Clayton Christensen’s terms. This is great for Google. This is great for our products. Look at all these amazing things that we’re doing.

Ben Gilbert

It further entrenches incumbents.

David Rosenthal

It further entrenches our lead in all of our already-leading products. We can deploy more capital in a predictable way to either drive down costs or make our product experiences that much better than any startup could make. Get them monetized that much better. All the things.

Once ChatGPT comes out, on a dime, overnight, AI shifts from being a sustaining innovation to a disruptive innovation. It is now an existential threat. And many of Google’s strengths from the last 10, 15, 20 years of all the AI work that’s happened in the company are now liabilities. They have a lot of existing castles to protect.

Ben Gilbert

That’s right. They have to run everything through a lot of filters before they can decide if it’s a good idea to go try to out-OpenAI OpenAI.

David Rosenthal

Yep. So this code red that Sundar issues to the company is actually a huge moment because what it means, and what he says, is, “We need to build and ship real native AI products ASAP.”

This is actually what you need to do in the textbook response to a disruptive innovation as the incumbent. You need to not bury your head in the sand, and you need to say, “Okay, we need to actually go build and ship products that are comparable to these disruptive innovators.”

You need to be laser-focused operationally in all the details to try to figure out where the new product is actually cannibalizing your old product and where the new product can be complementary, and just lean into all the ways in which you can be complementary in all the different little scenarios.

Really, what they’ve been trying to do—this ballet from 2022 onward—is protect the growth of Search while also creating the best AI experiences they can. It’s very clever, the way that they do AI Overviews for some but not all queries. They have AI Mode for some but not all users. And then they have Gemini, the full AI app, but they’re not redirecting Google.com to Gemini.

It’s this very delicate dance of protecting the existing franchise while also building, hopefully, a new franchise that is as non-cannibalizing as possible.

Ben Gilbert

Yep. And you see them really going hard and, I think, building leading products in non-cannibalizing categories like video.

David Rosenthal

Right? Veo 3 or Nano Banana. These are things that don’t in any way cannibalize the existing franchise. They, in fact, use some of Google’s strengths, all the YouTube training data and stuff like that.

Ben Gilbert

Yeah. So what happens next?

David Rosenthal

As you might expect, it gets worse before it gets better. Code red goes out in December 2022.

Bard, baby. Launch Bard.

Ben Gilbert

Oh boy. Well, even before that—in January 2023—OpenAI hits 100 million registered users for ChatGPT. Microsoft announces that they are investing another $10 billion in OpenAI and says that they now own 49% of the for-profit entity. Incredible in and of itself.

But think about this from Google's lens: Microsoft, our enemy, now arguably owns—obviously, in retrospect, they don't own OpenAI, but at the time it seemed like, oh my God, Microsoft might now own OpenAI—which is our first true existential threat in our history as a company.

David Rosenthal

Not great, Bob.

Ben Gilbert

So then, in February 2023, the Bing integration launches. Satya Nadella has the quote about wanting to make Google dance. Meanwhile, Google is scrambling internally to launch AI products as fast as possible.

So the first thing they do is take the LaMDA model and the chatbot interface to it. They rebrand it as Bard.

David Rosenthal

They ship that publicly.

Ben Gilbert

And they release it immediately. February 2023: ship it publicly. It's available GA to anyone.

David Rosenthal

Which maybe was the right move, but God, it was a bad product.

Ben Gilbert

It was really bad.

David Rosenthal

I didn't know the term at the time—RLHF—but it was clear it was missing a component of some magic that ChatGPT had. This reinforcement learning with human feedback, where you could really tune the appropriateness, the tone, the voice, and the correctness of the responses, just wasn't there.

Ben Gilbert

Yep. So, to make matters worse, in the launch video for Bard—a choreographed, prerecorded video where they're showing conversations with Bard—Bard gives an inaccurate factual response to one of the queries that they include in the video.

David Rosenthal

This is one of the worst keynotes in history.

Ben Gilbert

After the Bard launch and this keynote, Google's stock drops 8% that day. And then, like we were saying, once the actual product comes out, it becomes clear it's just not good.

David Rosenthal

Yep.

Ben Gilbert

And it pretty quickly becomes clear it's not just that the chatbot isn't good; the model isn't good. So in May, they replace LaMDA with a new model from the Brain team called PaLM. It's a little bit better, but it's still clearly behind not only GPT-3.5, but also GPT-4, which OpenAI comes out with in March 2023.

David Rosenthal

You can access that now through ChatGPT. And here is where Sundar makes 2 really, really big decisions. Number 1, he says, “We cannot have 2 AI teams within Google anymore. We're merging Brain and DeepMind into one entity called Google DeepMind.”

Ben Gilbert

Which is a giant deal. This is in full violation of the original deal terms of bringing DeepMind in.

David Rosenthal

Yep. And the way he makes it work is he says, “Demis, you are now CEO of the AI division of Google, Google DeepMind. This is all hands on deck, and you and DeepMind are going to lead the charge. You're going to integrate with Google Brain, and we need to change all of the past 10 years of culture around building and shipping AI products within Google.”

Ben Gilbert

To further illustrate this, when Alphabet became Alphabet, they had all these separate companies, but things that were really core to Google, like YouTube, actually stayed a part of Google. DeepMind was its own company. That's how separate this was. They're working on their own models. In fact, those models are predicated on reinforcement learning—that was the big thing that DeepMind had been working on the whole time.

David Rosenthal

Yep. There's a little bit of interesting backstory to this too. So Mustafa Suleyman, the third co-founder of DeepMind, at some point before this—

Ben Gilbert

He became the head of Google AI policy or something.

David Rosenthal

He had already shifted over to Brain and to Google.

Ben Gilbert

He stayed there for a little while, and then he ended up getting close with who else? Reid Hoffman. Remember, Reid is on the ethics board for DeepMind. Mustafa and Reid leave and go found Inflection AI, which—fast-forward now into 2024—comes after the absolute insanity that goes down at OpenAI over Thanksgiving 2023, when Sam Altman gets fired over the weekend during Thanksgiving and then brought back by Monday, when all the team threatened to quit and go to Microsoft.

David Rosenthal

They love Thanksgiving. Can't wait for this year.

Ben Gilbert

They love Thanksgiving. Yeah. Gosh. After all that, which certainly strains the Microsoft relationship—remember again, Reid is on the board of Microsoft—Microsoft does one of these acquisition-type deals with Inflection AI and brings Mustafa in as the head of AI for Microsoft.

David Rosenthal

Crazy.

Ben Gilbert

Wild, right? Just wild.

David Rosenthal

Crazy turn of events.

Ben Gilbert

Okay, so that first big decision that Sundar makes is unifying DeepMind and Brain. That was huge. Equally big, he says, “I want you guys to go make a new model, and we're just going to have 1 model that's going to be the model for all of Google internally, for all of our AI products externally. It's going to be called Gemini. No more different models, no more different teams—just 1 model for everything.”

David Rosenthal

It's a giant deal, and it's twofold. It's push and it's pull. It's saying, “Hey, if anyone's got a need for an AI model, you've got to start using Gemini.” But 2, it's actually kind of the Google+ thing, where they go to every team and start saying, “Gemini is our future. You need to start looking for ways to integrate Gemini into your product.”

Ben Gilbert

Yes, I'm so glad you brought up Google+. This came up with a few folks I spoke to in the research. Obviously, this is all playing out in real time, but the point a lot of people at Google made is that the Gemini situation is very different from the Google+ situation. This is a technical thing, A, which has always been Google's wheelhouse, but B, even more importantly, this is the rational business thing to do in the age of these huge models.

David Rosenthal

The more data you put in, the better it's going to get, and the better all the outputs are going to be.

Ben Gilbert

And because of scaling laws, you need your models to be as big as possible in order to have the best performance possible. If you're trying to maintain multiple models within a company, you're repeating multiple huge costs to maintain huge models. You definitely don't want to do that. You need to centralize on just 1 model.

David Rosenthal

Yeah, it's interesting. There's also something to read into where, at first, it was the Gemini model underneath the Bard product. Bard was still the consumer name. Then at some point they said, “No, we're just calling it all Gemini,” and Gemini became the user-facing name. Also, this pulls in my quintessence from the Alphabet episode. I know it's a little woo-woo, but with Google saying, “We're actually going to name the consumer service the name of the AI model,” they're sort of admitting to themselves: this product is nothing but technology. There isn't productiness to do on top of it. It's just like Gmail. Gmail was technology: it was fast search, lots of storage, use it on the web. The productiness wasn't particularly the way that, like, Instagram was all about the product. Gemini the model, Gemini the chatbot says, “We're just exposing our amazing breakthrough technology to you all and you get to interface directly with it.” Anthropologically, looking from afar, it kind of feels like it's that principle at work.

Ben Gilbert

I totally agree. I think it's actually a really important branding point and sort of rallying point for Google and Google culture to do this.

David Rosenthal

Right. All right, so this is all the stuff going on in Google, 2023-ish, in AI. Before we catch up to the present, I have a whole other branch of Alphabet that's been a real bright spot for AI. Can I go there? Can I take this offramp, if you will?

Ben Gilbert

Can you take the wheel, so to speak?

David Rosenthal

May I take the wheel? May I investigate another bet?

Ben Gilbert

Yeah, please tell us the Waymo story.

David Rosenthal

Awesome. So we've got to rewind all the way back to 2004, the DARPA Grand Challenge, which was created as a way to spur research into autonomous ground robots for military use. And actually, what it did for our purposes here today is create the seed talent for the entire self-driving car revolution 20 years later.

The competition itself is really cool. There is a 132-mile race course. Now, mind you, this is 2004 in the Mojave Desert that the cars have to race on. It is a dirt road. No humans are allowed to be in or interact with the cars. They are monitored 100% remotely, and the winner gets $1 million.

Ben Gilbert

$1 million.

David Rosenthal

Which was a break from policy. Normally, these are grants, not prize money, so this needs to be authorized by an act of Congress. The $1 million eventually felt comical, so the second year they raised the pot to $2 million. It's crazy thinking about what these researchers are worth today—that that was the prize for the whole thing.

The first year, in 2004, went fine. There were some amazing tech demonstrations on these really tight budgets, but ultimately 0 of the 100 registered teams finished the race.

Ben Gilbert

But the next year, in 2005, was the real special year. The progress that the entire industry made in those first 12 months from what they learned is totally insane. Of the 23 finalists entering the competition, 22 of them made it past the spot where the furthest team the year before had made it. The amount that the field advanced in that 1 year is insane.

Not only that, 5 of those teams actually finished all 132 miles. Two of them were from Carnegie Mellon, and 1 was from Stanford, led by a name that all of you will now recognize: Sebastian Thrun.

David Rosenthal

Indeed.

Ben Gilbert

This is Sebastian’s origin story before Google. Now, as we said, Sebastian was kind enough to help us with prep for this episode, but I actually learned most of this from watching a 20-year-old NOVA documentary that is available on Amazon Prime Video. Thanks to Brett Taylor for giving us the tip on where to find this documentary. Yes, the hot research tip.

So, what was special about this Stanford team? Well, one, there’s a huge problem with noisy data that comes out of all of these sensors. It’s in a car in the desert getting rocked around. It’s in the heat. It’s in the sun.

So common wisdom, and what Carnegie Mellon did, was to do as much as you possibly can on the hardware to mitigate that. So, things like custom rigging and gimbals and giant springs to stabilize the sensors. Carnegie Mellon would essentially buy a Hummer and rip it apart and rebuild it from the wheels up. We’re talking welding and real construction on a car.

The Stanford team did the exact opposite. They viewed any new piece of hardware as something that could fail. So, in order to mitigate risks on race day, they used all commodity cameras and sensors that they just mounted on a nearly unmodified Volkswagen. They only innovated in software, and they figured they would come up with clever algorithms to help them clean up the messy data later. Very Google-y, right?

David Rosenthal

Very Google-y.

Ben Gilbert

The second thing they did was an early use of machine learning to combine multiple sensors. They mounted lidar hardware on the roof, just like what other teams were doing. This is the way that you can measure texture and depth of what is right in front of you. The data is super precise, but you can’t drive very fast because you don’t really know much about what’s far away, since it’s this fixed field of view. It’s very narrow.

Essentially, you can’t answer the question, “How fast can I drive?” or “Is there a turn coming up?” So, on top of that, the way they solved it was they also mounted a regular video camera. That camera can see a pretty wide field of view, just like the human eye, and it can see all the way to the horizon, just like the human eye. Crucially, it could see color.

David Rosenthal

So you could figure out your safe path through the desert.

Ben Gilbert

That’s awesome.

David Rosenthal

It’s so awesome.

Ben Gilbert

I’m imagining a Dell PC sitting in the middle of this car in 2005.

It’s not far off. In the email that we send out, we’ll share some photos of it. It could then drive faster with more confidence, and it knew when turns were coming up. Again, this is real time onboard the car. 2005 is wild on that tech.

So ultimately, both of these bets worked, and the Stanford team won in super dramatic fashion. They actually passed one of the Carnegie Mellon teams autonomously through the desert. It’s this big dramatic moment in the documentary.

So you would kind of think, “So then Sebastian goes to Google and builds Waymo.” No. As we talked about earlier, he does join Google through that crazy, “Please don’t raise money from Benchmark and Sequoia, and we’ll just hire you instead.” But he goes and works on Street View and Project Ground Truth and co-founds Google X.

David, as you were alluding to earlier, Project Chauffeur, which would become Waymo, is the first project inside Google X. I think the story is that Larry came to Sebastian and was like, “Yeah, that self-driving car stuff—do it.”

Sebastian was like, “No, come on. That was a DARPA challenge.” And Larry’s like, “No, no, you should do it.” He’s like, “No, no, that won’t be safe. There are people running around cities. I’m not just going to put multiton killer robots on roads and potentially harm people.”

Larry finally comes to him and says, “Why? What is the technical reason that this is impossible?” Sebastian goes home, sleeps on it, and comes in the next morning and says, “I realized what it was. I’m just afraid.”

David Rosenthal

Such a good moment.

Ben Gilbert

So they start. He’s like, “There’s not a technical reason. As long as we can take all the right precautions and hold a very high bar on safety, let’s get to work.”

Larry then goes, “Great. I’ll give you a benchmark, so that way you know if you’re succeeding.” He comes up with these 10 stretches of road in California that he thinks will be very difficult to drive. It’s about 1,000 miles, and the team starts calling it the Larry 1000. It includes driving to Tahoe, Lombard Street in San Francisco, Highway 1 to Los Angeles, and the Bay Bridge. This is the bogey.

David Rosenthal

Yep. If you can autonomously drive these stretches of road, that’s a pretty good indication that you can probably do anything.

Ben Gilbert

Yep. So they start the project in 2009. Within 18 months, this tiny team—I think they hired, I don’t know, a dozen people or something—had driven thousands of miles autonomously, and they managed to succeed on the full Larry 1000 within 18 months.

David Rosenthal

Totally unreal how fast they did it. And then also totally unreal how long it takes after that to productize and create the Waymo that we know today.

Ben Gilbert

Right. It’s like the first 99% and then the second 99% that takes 10 years.

Yeah. Self-driving is one of these really tricky types of problems where it’s surprisingly easy to get started, even though it seems like it would be an impossible thing. But then there are edge cases everywhere: weather, road conditions, other drivers, novel road layouts, night driving. So it takes this massive amount of work for a production system to actually happen.

So then the question is, what business do we build? What is the product here? There was what Sebastian wanted, which was highway assist—sort of the lowest-stakes, most realistic option. Let’s make a better cruise control.

There was what Eric Schmidt wanted, which is crazy. He proposed, “Oh, let’s just go buy Tesla, and that’ll be our starting place. Then we’ll just put all of our self-driving equipment on all the cars.” David, do you know what it would have cost to buy Tesla at the time?

David Rosenthal

I think at the time that negotiations were taking place between Elon and Larry and Google, this was in the depths of the Model S production-scaling woes. I think Google could have bought the company for $5 billion. That’s what I remember.

Ben Gilbert

It was $3 billion.

David Rosenthal

$3 billion. Oh my goodness.

Ben Gilbert

Obviously, that didn’t happen, but what a crazy alternative history that could have been, right?

David Rosenthal

Right? I mean, I think if that had happened, DeepMind would not have gone down in the same way, and probably OpenAI would not have gotten founded.

Ben Gilbert

That’s probably right.

David Rosenthal

I think that is obviously unprovable, right? The counterfactuals that we always come up with on this show—you can’t know.

Ben Gilbert

Yeah. It seems more likely than not to me that, at a minimum, OpenAI would not exist, right?

David Rosenthal

Right? So then there was what Larry wanted to do: option three, build robotaxis. Yeah.

Ben Gilbert

And ultimately, that is at least right now what they would end up doing. We could do a whole episode about this journey, but we’ll just hit some of the major points for the sake of time.

David Rosenthal

The big thing to keep in mind here is that neither Google nor the public really knew if self-driving was something that could happen in the next 2 years from any given point or take another 10. And just to illustrate it, for the first 5 years of Project Chauffeur, it did not use deep learning at all. They did the Larry 1000 without any deep learning and then went another 3 and a half years.

Ben Gilbert

Wow, that’s crazy.

David Rosenthal

Yeah. And yet it totally illustrates that you never know how far away the end goal is.

Ben Gilbert

And this is a field where the only way progress happens is through this series of breakthroughs. You don’t know, A, how far the next breakthrough is, because at any given time there are lots of promising things in the field, most of which don’t work out; and B, when there is a breakthrough, how much lift that will give you over existing methods.

So anytime people are forecasting, “Oh, in AI we’re going to be able to do X, Y, Z in X years,” it’s a complete fool’s errand. Even the experts don’t know.

David Rosenthal

Here are the big milestones. In 2013, they started using convolutional neural networks. They could identify objects, and they got much better perception capabilities. This 2013–2014 period is when Google found religion around deep learning. This was right after the 40,000 GPUs rolled out, so they actually had some hardware to start doing this on.

Ben Gilbert

In 2016, they had seen enough proof of the technology that they thought, “Let’s commercialize this. We can actually spin this out into a company.” So Waymo became its own subsidiary inside Alphabet. It was no longer part of Google X.

David Rosenthal

In 2017, obviously, the Transformer came out. They incorporated some learnings from the Transformer, especially around prediction and planning.

Ben Gilbert

In March of 2020, they raised $3.2 billion from folks like Silver Lake, the Canada Pension Plan Investment Board, Mubadala, Andreessen Horowitz, and, of course, the biggest check, I think, from Alphabet. I think Alphabet is always the biggest check because it’s still the majority owner, even after a bunch more fundraises.

David Rosenthal

In October of 2020, they launched the first public commercial, no-human-behind-the-driver’s-seat service in Phoenix. It was the first in the world. This was 11 years after succeeding in the DARPA Grand Challenge. And this is nuts.

Ben Gilbert

I had given up at this point. I was like, “That’s cute that Waymo and all these other companies are trying to do self-driving. Seems like it’s never going to happen.” And then they were actually doing a large volume of rides safely with consumers and charging money for it in Phoenix.

David Rosenthal

Then they bring it to San Francisco, where, for me and lots of people in San Francisco, it is a huge part of life in the city here now. It’s amazing. Every time I’m down there, I love taking them. They’re launching in Seattle soon. I’m pumped.

Interestingly, they don’t make the hardware. They use a Jaguar I-PACE. From what I can tell, that vehicle is only in Waymos. I don’t know if anybody else drives that Jaguar or if you can buy it, but they’re working on a sort of van next. They have some next-generation hardware.

For anyone who hasn’t taken it, it’s an Uber but with no driver. That launched in June of 2024. Along the way there, they raised their quote-unquote Series B, another $2.5 billion. Then, after the San Francisco rollout, they raised their quote-unquote Series C, $5.6 billion. This year in January, they were reportedly doing more in gross bookings than Lyft in San Francisco. Wow. I totally believe it. I mean, it is the number-one option in San Francisco that I and everybody I know always go to for ride-hailing. It’s like, try to get a Waymo. If there’s not a Waymo available anytime soon, then go down the stack.

Ben Gilbert

Like we’re living in the future, and how quickly we fail to appreciate it.

David Rosenthal

Yeah. And what’s cool, I think, for people who it hasn’t come to their city and is not part of their lives yet, it’s not just that it’s a cool experience to not have a driver behind the wheel. Pretty quickly, that just fades. It’s actually a different experience.

If I need to go somewhere with my older daughter, I don’t mind hailing a Waymo, bringing the car seat, installing the car seat in the Waymo, and driving with my daughter. She loves it. We call it a robot car, and she’s like, “A robot car? I’m so excited.”

Ben Gilbert

Huh.

David Rosenthal

I would never do that with an Uber.

Ben Gilbert

That’s interesting.

David Rosenthal

With my dog, whenever I need to go with my dog, it’s super awkward to hail an Uber and be like, “Hey, I’ve got my dog. Can the dog come in it?” Not a big deal with a Waymo.

Ben Gilbert

Yeah, we can actually have sensitive conversations in the car.

David Rosenthal

You can have phone calls. It really is a different experience.

Ben Gilbert

Yeah, that’s so true. So, may as well catch up to today. They’re operating in 5 cities: Phoenix, San Francisco, LA, Austin, and Atlanta. They have hundreds of thousands of paid rides every week. They’ve now driven over 100 million miles with no human behind the wheel, growing at 2 million every week. There are over 10 million paid rides across 2,000 vehicles in the fleet.

They’re going to be opening a bunch more cities in the U.S. next year. They’re launching in Tokyo, their first international city—slowly and then all at once. I mean, that’s kind of the lesson here.

The technology—they really continued with that multisensor approach all the way from the DARPA Grand Challenge. Cameras, lidar; they added radar, and they actually use audio sensing as well. Their approach is basically that any data we can gather is better because that makes it safer. So they have 13 cameras, 4 lidar sensors, 6 radar sensors, and an array of external microphones.

This is obviously a way more expensive solution than what Tesla is doing with just cameras. But Waymo’s party line is that they believe it is the only path to full autonomy, to hit the safety bar and regulatory bar that they’re aiming for.

David Rosenthal

Yeah.

Ben Gilbert

It seems like a really big line in the sand for them anytime you talk to somebody in that organization.

David Rosenthal

Yeah. And look, as a regular user of both products—you know, happy owner and driver of a Model Y, in addition to being a regular Waymo user—at least with the current instantiation of Full Self-Driving on my Tesla, they’re vastly different products.

Full Self-Driving on my Model Y is great. I use it all the time on the freeway, but I would never not pay attention. Whereas every time I get in a Waymo, it’s almost like Google Search, right? It’s like, I just trust that, oh, this is going to be completely and totally safe, and I’m sitting in the back seat and I can totally tune out.

Ben Gilbert

I think I trust my Model Y FSD more than you do. But I get what you’re saying, and frankly, regulatorily, you are required to still pay attention in a Tesla and not in a Waymo.

The safety thing is super real, though. I mean, if you look at the numbers, over a million motor vehicle crash fatalities every year, or there are over a million fatalities in the U.S. alone. Over 40,000 deaths occur per year. So if you break that down, that’s 120 every day. That’s like a giant cause of death.

David Rosenthal

Yes.

Ben Gilbert

The study that Waymo just released last month showed that they have 91% fewer crashes involving serious injuries or worse compared to the average human driver, even controlled for the fact that Waymos right now are only driving on city surface streets. So they controlled it apples-to-apples with human driving data, and it’s a 91% reduction in either fatalities or serious injuries. Why aren’t we all talking about this all the time, every day? This is going to completely change the world and a giant cause of death.

David Rosenthal

Yeah.

Ben Gilbert

So, while we’re in Waymo land, what do you think about doing some quick analysis?

David Rosenthal

Great.

Ben Gilbert

Because I’ve been scratching my head about what this business is. Then I promise we’ll go back to the rest of Google AI and catch up to today.

It is super expensive to operate, especially at early scale. The training is high, the inference is high, the hardware is high, et cetera, et cetera, et cetera.

David Rosenthal

Also, the operations are expensive.

Ben Gilbert

Yes. And in fact, they’re experimenting. In some cities, they actually outsource the operations. The fleet is managed by a rental car company in Texas, or they’ve partnered, I believe, with Lyft and with Uber and different companies. So they’re trying all sorts of owned-and-operated versus partnership models to operate it.

David Rosenthal

Yeah. And the operations are—these are electric cars. They need to be charged. They need to be cleaned. They need to be returned to depots. They need to be checked out. They need to have sensors replaced.

Ben Gilbert

So the question is, what is the potential market opportunity? How big could this business be? And there are a few different ways you could try to quantify it.

One total market-size thing you could do is try to sum the entire automaker market cap today, and that would be $2.5 trillion globally if you include Tesla, or $1.3 trillion without Tesla. But Waymo is not really making cars, so that’s probably the wrong way to slice it.

You could look at all the ridesharing companies today, which might be a better comp because that’s the business that Waymo is actually in today. That’s on the order of $300 billion, most of which is Uber.

David Rosenthal

Yep. So that’s addressable market cap today with ridesharing. Waymo’s ambitions, though, are bigger than that. They want to be in the cars that you own. They want to be in long-haul trucking.

So they believe they can grow the share of transportation because there are blind people who could own a car. There are elderly people who could get where they need to go on their own without having a driver, that sort of thing.

So the squishiest but, I think, most interesting way to look at it is: What is the value from all of the reduction in accidents? Because that’s really what they’re doing. It’s a product to replace accidents with non-accidents.

Ben Gilbert

I think that’s viable, but again, I would say, as a regular user of the product, it is a different and expanding product to human rideshare. So your argument is, whatever number I come up with for reducing accidents, it’s still a bigger market than that because there’s additional value created in the product experience itself.

David Rosenthal

Yeah. Scoping just to rideshare, now that we have Waymo in San Francisco, I use Waymo in scenarios where I would never use an Uber or a Lyft.

Ben Gilbert

Yeah, makes sense. So here’s the data we have. The CDC released a report saying deaths from crashes in 2022 in the U.S. resulted in $470 billion in total costs, including medical costs and the cost estimates for lives lost, which is crazy—that the CDC has some way of putting a cost on human life, but they do.

So, if you reduce crashes 10x, which is what Waymo seems to be saying in their data, at least for the serious crashes, that’s over $420 billion a year in total costs that we would save as a nation. Now, it’s not totally apples-to-apples. I recognize this, but that cost savings is more than Google does today in revenue in their entire business.

You could see a path to a Google-sized opportunity for Waymo as a standalone company just through this analysis, as long as they figure out a way to get costs down to the point where they can run this as a large and profitable business. Yeah, it is an incredible 20-plus-year success story within Google.

David Rosenthal

The way I want to close it is, the investment so far actually hasn’t been that large. When you consider this opportunity, they have burned somewhere in the neighborhood of $10 billion to $15 billion. That’s sort of why I was listing all the investments to get to this point.

Ben Gilbert

Chump change compared to foundation models.

David Rosenthal

Dude, also, let’s just keep it scoped in this sector. That’s one year of Uber’s profits.

Ben Gilbert

Wow. Seems like a good bet.

David Rosenthal

I used to think this was some wild goose chase. It now looks really, really smart.

Ben Gilbert

Yep. Totally agree.

David Rosenthal

Also, that $10 billion to $15 billion cost is the profits that Google made last month.

Ben Gilbert

Google. Well, speaking of Google, should we catch ourselves up to today with Google AI?

David Rosenthal

Yes. So, I think where you were is the Gemini launch.

Ben Gilbert

So, Sundar makes these 2 decrees in mid-2023. One: We’re merging Brain and DeepMind into one team for AI within Google.

And two, we're going to standardize on 1 model, the future Gemini. And Google DeepMind team, you go build it, and then everybody in Google, you're going to use it.

David Rosenthal

Not to mention, apparently Sergey Brin is now back as an employee working on Gemini.

Ben Gilbert

Yes. Employee number—

David Rosenthal

Got his badge back.

Ben Gilbert

Yeah. Got his badge back.

David Rosenthal

So once Sundar makes these decisions, Jeff Dean and Oriol Vinyals from Brain go over and team up with the DeepMind team, and they start working on Gemini.

Ben Gilbert

I'm a believer now. By the way, you got Jeff Dean working on it, I'm in.

David Rosenthal

If you got Jeff Dean on it, it's probably going to work. If you weren't a believer yet, wait until I tell you the next thing. Once they get Noam back, when they do the deal with Character AI, they bring him back into the fold. Noam joins the Gemini team, and Jeff and Noam are the 2 co-technical leads for Gemini now.

Ben Gilbert

Let's go.

David Rosenthal

Let's go. So, they actually announced this very quickly at the Google I/O keynote in May 2023. They announced Gemini. They announced the plans. They also launched AI Overviews in Search, first as a Labs product, and then later that became standard for everybody using Google Search, which is crazy, by the way.

The number of Google searches that happen is unfathomably large. I'm sure there's a number for it, but just think about it: that's about the highest level of computing scale that exists, other than high-bandwidth things like streaming. Just think about all the instances of Google searches that happen. They are running an LLM inference on all of those, or at least as many as they're willing to show AI Overviews on, which I'm sure is not every query, but many—a subset.

David Rosenthal

Yeah.

Ben Gilbert

But still, a large, large number of Google searches. I mean, I see them all the time.

David Rosenthal

Yep.

Ben Gilbert

This is really Google immediately deciding to operate at AI speed. ChatGPT happened on November 30, 2022. We're now in May 2023. All of these decisions have been made, all of these changes have happened, and they're announcing things at I/O—

David Rosenthal

And they're really flexing the infrastructure that they've got. I mean, the fact that they can go, “Oh, yeah, sure, let's do inference on every query. We're Google. We can handle it.”

Ben Gilbert

So a key part of this new Gemini model that they announced in May 2023 is that it's going to be multimodal. Again, this is 1 model for everything: text, images, video, audio—1 model.

They released it for early public access in December 2023. So, also crazy: 6 months. They build it, they train it, they release it.

David Rosenthal

That is amazing.

Ben Gilbert

By February 2024, they launched Gemini 1.5 with a 1 million-token context window, a much, much larger context window than any other model on the market—

David Rosenthal

Which enables all sorts of new use cases. There are all these people who were like, “Oh, I tried to use AI before, but it couldn't handle my XYZ use case.” Now they can.

Ben Gilbert

Yep.

David Rosenthal

The next year, in February 2025, they released Gemini 2.0. In March 2025, 1 month later, they launched Gemini 2.5 Pro in experimental mode. Then that went GA in June.

Ben Gilbert

This is like NVIDIA pace, how often they're shipping.

David Rosenthal

Yeah, seriously. Also in March 2025, they launched AI Mode, so you can now switch over on Google.com to chatbot mode—

Ben Gilbert

And they're split-testing, auto-opting some people into AI Mode to see what the response is. This is the golden goose.

David Rosenthal

Yeah, the elephant is tap-dancing here.

Ben Gilbert

Yep.

David Rosenthal

Then there's all the other AI products that they launched during this period. NotebookLM comes out, with AI-generated podcasts—

Ben Gilbert

Which, does that sound like us to you? It feels a little trained.

David Rosenthal

The number of texts that we got when that came out—“This must be trained on Acquired.”

Ben Gilbert

I do know that a bunch of folks on the NotebookLM team are Acquired fans. I don't know if they trained on us.

David Rosenthal

And then there's the video and image stuff: Veo 3, Nano Banana, Genie 3. Genie—this is insane. This is a world-builder based on prompts and videos.

Ben Gilbert

Yeah. You haven't actually used it yet, right? You watched that hype video.

David Rosenthal

Yeah, I watched the video. I haven't actually used it.

Ben Gilbert

Yeah. I mean, if it does that, that's unbelievable. It's a real-time generative—

David Rosenthal

World-builder.

Ben Gilbert

World-builder. Yeah. You look right, and it invents stuff to your right. I mean, you combine that with Vision Pro hardware, and you're just living in a fantasy land.

So, they announced there are now 450 million monthly users of Gemini. Now, that includes everybody who's accessing Nano Banana.

David Rosenthal

Yeah, I can't believe this stat. This is insane. Even with recently being number 1 in the App Store, it still feels hard to believe. Google's saying it, so it must be true. But I just wonder: what are they counting as use cases of the Gemini app?

Ben Gilbert

Right? Certainly everybody who's using Nano Banana is using Gemini.

David Rosenthal

But is it counting AI Overviews? Is it counting AI Mode? Or is it counting something where I'm accidentally—like Meta said that crazy high number of people using Meta AI—

Ben Gilbert

Right. Right. Right.

David Rosenthal

That was complete garbage. That was people searching Instagram who accidentally hit a Llama model that made some things happen, and they were like, “Go away. I'm actually just looking for a user.” Is it really 450 million, or is it 450 million?

Ben Gilbert

Yeah, good question. Either way, going from zero is crazy impressive in the amount of time that they have done it, especially given that revenue is at an all-time high. They seem to, so far, at least in this squishy early phase, be able to figure out how to keep the core business going while doing well as a competitor at the cutting edge of AI.

David Rosenthal

Yeah. And to foreshadow a little bit, we're going to do a bull and bear here in a minute. As we talked about in our Alphabet episode, Google does have a history of navigating platform shifts incredibly well, in the transition to mobile.

Ben Gilbert

It's true.

David Rosenthal

Definitely a rockier start here in the AI platform shift. Much rockier. But, hey, look, if you were to lay out a recipe for how to respond, given the rocky start, it'd be hard to come up with a much better slate of things than what they've done over the last 2 years.

Ben Gilbert

Yeah. All right. Should I give us the snapshot of the business today?

David Rosenthal

Give us the snapshot of the business today. Oh, yeah, also, by the way, the federal government decided they were a monopoly and then decided not to do anything about it because of AI.

Ben Gilbert

Yeah. So, between the time when we shipped our Alphabet episode and here with our Google AI episode—or our Part II and Part III, for those who prefer simpler naming schemes—there was a U.S. v. Google antitrust case. The judge first ruled that Google was a monopoly in internet search and then did not come up with any material remedies. I mean, there are some, but I would call them immaterial.

They did not need to spin off Chrome, and they did not need to stop sending tens of billions of dollars to Apple and others. In other words, yes, Google's a monopoly, and the cost of doing anything about that would have too many downstream consequences for the ecosystem. So, we're just going to let them keep doing what they're doing.

One of the reasons that the judge cited for why they weren't going to really take these actions is the race in AI. Because tens of billions of dollars of funding have gone into companies like OpenAI, Anthropic, and Perplexity, Google essentially has this new war to fight, and we're going to leave it to the free market to do its thing, where it creates viable competition on its own. We're not going to hamstring Google.

Personally, I think this argument is a little bit silly. None of these AI companies are generating net income, and just because they've raised a huge amount of money, it doesn't mean that will last forever. They'll all burn through their existing cash in a pretty short period of time. If the spigots ever dry up, Google doesn't have any self-sustaining competition right now, whether in its old search business or in AI. It is all dependent on people believing that the opportunity is so large that they keep pouring tens of billions of dollars into these competitors.

Yeah, plenty of other folks have made the glib comment, but there's merit to it: “Hey, as flat-footed as Google was when ChatGPT happened, if the outcome of this is they avoid a Microsoft-level distraction and damage to their business from a U.S. federal court monopoly judgment, worth it.”

David Rosenthal

Well, there's a funny meme here that you could draw. You know that meme of someone pushing the domino and it knocking over some big wall later?

Ben Gilbert

Yeah.

David Rosenthal

There's the domino of Ilya Sutskever leaving Google to start OpenAI, and the downstream effect is Google is not broken up.

Ben Gilbert

Yeah, right, exactly.

David Rosenthal

It actually saves Google.

Ben Gilbert

It's totally wild.

David Rosenthal

Totally wild.

Ben Gilbert

All right. So, here's the business today. Over the last 12 months, Google has generated $370 billion in revenue. On the earnings side, they've generated $140 billion over the last 12 months, which is more profit than any other tech company. The only company in the world with more earnings is Saudi Aramco.

Let's not forget: Google is the best business ever. We also made the point at the end of the Alphabet episode that, even in the midst of all of this AI era and everything that's happened over the last 10 years, the last 5 years, Google's core business has continued to grow 5× since the end of our Alphabet episode in 2015–2016.

David Rosenthal

Yeah. Market cap: Google surged past its old peak of $2 trillion and just hit that $3 trillion mark earlier this month. They're the fourth-most-valuable company in the world, behind NVIDIA, Microsoft, and Apple. It's just crazy.

On their balance sheet, I actually think this is pretty interesting. I normally don't look at the balance sheet as a part of this exercise, but it's useful.

Ben Gilbert

And here's why. In this case, they have $95 billion in cash and marketable securities. I was about to stop there and make the point: Wow, look how much cash and resources they have.

David Rosenthal

I'm actually surprised it's not more. It used to be $140 billion in 2021, and over the last 4 years, they've massively shifted from this mode of accumulating cash to deploying cash. A huge part of that has been the capex of the AI data center buildout, so they're very much playing offense in the way that Meta, Microsoft, and Amazon are in deploying that capex.

Ben Gilbert

But the thing that I can't quite figure out is that the largest part of that was actually buybacks, and they started paying a dividend. If you're not a finance person, the way to read into that is: Yes, we still need a lot of cash for investing in the future of AI and data centers, but we still had way more cash than we needed, and we decided to distribute that to shareholders.

David Rosenthal

Yeah, that's crazy.

Ben Gilbert

Best business of all time, right? That illustrates what a crazy business their core search ads business is. They're saying, "The most capital-intensive race in business history is happening right now. We intend to win it."

David Rosenthal

Yeah.

Ben Gilbert

And we have tons of extra cash lying around on top of what we think we need, plus a safety cushion for investing in that capex race.

David Rosenthal

Yeah. Yes.

Ben Gilbert

Wow. So there are 2 businesses that are worth looking at here. One is Gemini, to try to figure out what's happening there, and two is a brief history of Google Cloud. I want to tell you the cloud numbers today, but it's probably worth actually understanding how we got here on cloud.

David Rosenthal

Yep.

Ben Gilbert

First, on Gemini, because this is Google, and they have, I think, the most obfuscated financials of any of the companies we've studied. They anger me the most in being able to hide the ball in their financial statements. Of course, we don't know Gemini-specific revenue. What we do know is that there are over 150 million paying subscribers to the Google One bundle.

Most of that is on a very low tier. It's on the $5-a-month or $10-a-month tier. The AI stuff kicks in on the $20-a-month tier, where you get the premium AI features, but I think that's a very small fraction of the 150 million today.

David Rosenthal

Yeah, I think that's what I'm on.

Ben Gilbert

But 2 things to note. One, it's growing quickly. That 150 million is growing almost 50% year-over-year. But two, Google has a subscription bundle that 150 million people are subscribed to. I've had it in my head that AI doesn't have a future as a business model that people pay money for—that it has to be ad-supported like search.

David Rosenthal

But hey, that's not nothing. That's almost half of America.

Ben Gilbert

I mean, how many subscribers does Netflix have?

David Rosenthal

Netflix is in the hundreds of millions.

Ben Gilbert

Yeah, there are really scaled consumer subscription services. I owe this insight to Shishir Mehrotra. We chatted actually last night because I name-dropped him on the last episode, and then he heard it, so we reached out and talked. That's made me do a 180. I used to think that if you're going to charge for something, your total addressable market shrank by 90% to 99%.

But he has this point that if you build a really compelling bundle—and Google has the digital assets to build a compelling bundle—

David Rosenthal

Oh, my goodness. YouTube Premium, NFL Sunday Ticket.

Ben Gilbert

Yes. Stuff in the Play Store, YouTube Music, all the Google One storage stuff. They could put AI in that bundle and figure out, through clever bundle economics, a way to make a paid AI product that actually reaches a huge number of paying subscribers. Totally.

David Rosenthal

So we really can't figure out how much money Gemini makes right now. It's probably not profitable anyway. So what's the point of even analyzing it?

Ben Gilbert

Yeah. But, okay, tell us the cloud story.

David Rosenthal

So we intentionally did not include cloud in our Alphabet episode.

Ben Gilbert

Google Part II, effectively.

David Rosenthal

Google Part II. Yes, because it is a new product, and now a very successful one within Google that was started during the same time period as all the other ones that we talked about during Google Part II. But it's so strategic for AI. It is a lot more strategic now, in hindsight, than it looked when they launched it.

So, just quick background on it: It started as Google App Engine. It was a way, in 2008, for people to quickly spin up a backend for a web or, soon after, a mobile app. It was a Platform as a Service, so you had to do things in this very narrow, Google-y way. It was very opinionated. You had to use this SDK, you had to write it in Python or Java, and you had to deploy exactly the way they wanted you to deploy.

It was not a thing where they would say, "Hey, developer, you can do anything you want. Just use our infrastructure." It was opinionated—super different from what AWS was doing at the time and what they're still doing today, which the whole world eventually realized was right: cloud should be Infrastructure as a Service.

Even Microsoft pivoted Azure to this reasonably quickly, where it was like, "You want some storage? We got storage for you. You want a VM? We got a VM for you. You want some compute? You want a database?"

Ben Gilbert

We got you. Fundamental building blocks.

David Rosenthal

So eventually, Google launched its own Infrastructure as a Service in 2012. It took 4 years. They launched Google Compute Engine, which they would later rebrand as Google Cloud Platform. That's the name of the business today.

The knock on Google is that they could never figure out how to interface with the enterprise. Their core business was making really great products for people to use that they loved polishing. They made them all as self-serve as possible, and then the way they made money was from advertisers. And let's be honest, there's no other choice but to use Google Search, right?

It didn't necessarily need to have a great enterprise experience for its advertising customers because they were going to come anyway, right? And so they've got this self-serve experience. Meanwhile, the cloud is a knife fight. These are commodities.

Ben Gilbert

It's all about the enterprise.

David Rosenthal

It's the lowest possible price, and it's all about enterprise relationships, clever ways to bundle, and being able to deliver a full solution.

Ben Gilbert

You say "solution," I hear gross margin.

David Rosenthal

Yes. But yes, Google was out of its natural habitat in this domain. Early on, they didn't want to give away any crown jewels. They viewed their infrastructure as, "This is our secret thing. We don't want to let anybody else use it." And the best software tools that they had written for themselves, like Bigtable or Borg—how they ran Google—or MapReduce, these were not services that they were making available on Google Cloud.

Ben Gilbert

Yeah. These are competitive advantages.

David Rosenthal

Yes. And then they hired the former president of Oracle, Thomas Kurian.

Ben Gilbert

Yes, and everything kind of changed. In 2017, 2 years before he came in, they had $4 billion in revenue—10 years into running this business. In 2018, they made their first very clever strategic decision: They launched Kubernetes.

The big insight here is, if we make it more portable for developers to move their applications to other clouds, the world is kind of wanting multicloud here, right? We're the third-place player. We don't have anything to lose.

So we can offer this tool as a kind of counter-position against AWS and Azure. We shift the developer paradigm to use these containers. They orchestrate on our platform, and then we have a great service to manage it for you.

It was very smart. This kind of becomes one of the pillars of their strategy: You want multicloud? We're going to make that easy, and you can still choose AWS or Azure, too. It's going to be great.

So, David, as you said, the former president of Oracle, Thomas Kurian, was hired in late 2018. You couldn't ask for a better person who understands the needs of the enterprise than the former president of Oracle. This shows up in revenue growth right away.

In 2020, they crossed $13 billion in revenue, which was nearly tripling in 3 years. They hired something like 10,000 people into the go-to-market organization. I'm not exaggerating that. And that's on a base of 150 people when he came in, most of whom were seated in California, not regionally distributed throughout the world.

The funniest thing is that Google was kind of a cloud company all along. They had the best engineers building this amazing infrastructure, right? They had the products, they had the infrastructure, they just didn't have the go-to-market organization.

And the productization was all Google-y. It was for us, for engineers. They didn't really build things that let enterprises build the way that they wanted to build. This all changes. In 2022, they hit $26 billion in revenue. In 2023, they're a really viable third cloud.

They also flipped to profitability in 2023. Today, they're over a $50 billion annual revenue run rate. It's growing 30% year-over-year. They're the fastest-growing of the major cloud providers—5x in 5 years.

And it's really 3 things. It's finding religion on how to actually serve the enterprise. It's leaning into this multicloud strategy and actually giving enterprise developers what they want. And three, AI has been such a good tailwind for all hyperscalers because these workloads all need to run in the cloud: giant amounts of data, giant amounts of compute, and giant amounts of energy.

But in Google Cloud, you can use TPUs, which they make a ton of, while everyone else is desperately begging NVIDIA for allocations of GPUs. So, if you're willing to not use CUDA and build on the Google stack, they have an abundant amount of TPUs for you.

This is why we saved cloud for this episode. There are 2 aspects of Google Cloud that I don't think they foresaw back when they started the business with App Engine but are hugely strategically important to Google today.

One is simply that cloud is the distribution mechanism for AI. So if you want to play in AI today, you either need to have a great application, a great model, a great chip, or a great cloud. Google is trying to have all 4 of those.

David Rosenthal

Yes.

Ben Gilbert

There is no other company that has, I think, more than 1.

David Rosenthal

I think that's the right call. Think about the big AI players. NVIDIA—

Ben Gilbert

Chips.

David Rosenthal

NVIDIA kind of has a cloud, but not really. They just have chips, and they're the best chips and the chips everyone wants—but chips.

Ben Gilbert

Mainly, it's cloud.

David Rosenthal

Yes, cloud and cloud leader. Microsoft—

Ben Gilbert

Cloud.

David Rosenthal

It's just cloud, right? They make some models, but—

Ben Gilbert

I mean, they've got applications, but yeah, cloud.

David Rosenthal

Cloud. Apple—

Ben Gilbert

Nothing. Nothing.

David Rosenthal

AMD just chips.

Ben Gilbert

Yep. OpenAI, model.

David Rosenthal

Anthropic, model.

Ben Gilbert

Yep.

David Rosenthal

These companies don't have their own data centers. They're making noise about making their own chips, but not really, and certainly not at scale.

Ben Gilbert

Google has scale: data centers, scale chips, scale usage of models. I mean, even just from Google.com queries now on AI Overviews—

David Rosenthal

And scale applications.

Ben Gilbert

Yes. Yeah, they have all of the pillars of AI, and I don't think any other company has more than 1—

David Rosenthal

And they have the very most net-income dollars to lose.

Ben Gilbert

Right? So then there's the chip side specifically of this. If Google didn't have a cloud, it wouldn't have a chip business. It would only have an internal chip business. The only way that external companies, users, developers, and model researchers could use TPUs would be if Google had a cloud to deliver them, because there's no way in hell that Amazon or Microsoft are going to put TPUs from Google in their clouds.

David Rosenthal

We'll see.

Ben Gilbert

We'll see, I guess.

David Rosenthal

I think within a year it might happen. There are rumors already that some neoclouds in the coming months are going to have TPUs.

Ben Gilbert

Hmm, interesting. Nothing announced, but TPUs are likely going to be available in neoclouds soon, which is an interesting thing. Why would Google do that? Are they trying to build an NVIDIA-type business where they make money selling chips? I don't think so. I think it's more that they're trying to build an ecosystem around their chips the way that CUDA does. And you're only going to credibly be able to do that if your chips are accessible anywhere that someone's running their existing workloads.

David Rosenthal

Yep. It'd be very interesting if it happens. And, you know, look, you may be right. Maybe there will be TPUs in AWS or Azure someday, but I don't think they would have been able to start there. If Google didn't have a cloud and there weren't any way for developers to use TPUs and start wanting TPUs, would Amazon or Microsoft be like, "Ah, you know, all right, Google, we'll take some of your TPUs even though no developer out there uses them"? Right?

Ben Gilbert

All right. Well, with that, let's move into analysis. I think we need to do bull and bear on this one.

David Rosenthal

You have to this time.

Ben Gilbert

Got to bring that back.

David Rosenthal

For these episodes in the present, it seems like we need to paint the possible futures.

Ben Gilbert

Yes. Bringing back bull and bear. I love it. Then we'll do playbook, powers, quintessence. Bring it home.

David Rosenthal

Perfect. All right. So, here's my set of bull cases. Google has distribution to basically all humans as the front door to the internet. They can funnel that however they want. You've seen it with AI Overviews. You've seen it with AI Mode. Even though lots of people use ChatGPT for lots of things, Google's traffic, I assume, is still essentially at an all-time high, and it's a default behavior.

Ben Gilbert

Yep. Powerful. So that is a bet on implementation—that Google figures out how to execute and build a great business out of AI—but it is still theirs to lose.

David Rosenthal

Yeah. And they've got a viable product. It's not clear to me that Gemini is any worse than OpenAI's or Anthropic's products.

Ben Gilbert

No, I completely agree. This is a value-creation, value-capture thing. The value creation is there in spades. The value-capture mechanism is still TBD.

David Rosenthal

Yeah. Google's old value-capture mechanism is one of the best in history. So that's the issue at hand. Let's not get confused that it's not like a good experience—it's a great experience.

Ben Gilbert

Yeah. Yeah. Yeah. Okay. So we've talked about the fact that Google has all the capabilities to win in AI, and it's not even close: foundational model, chips, hyperscaler, all this with self-sustaining funding. I mean, that's the other crazy thing. You look at the clouds—they have self-sustaining funding. NVIDIA has self-sustaining funding. None of the model makers have self-sustaining funding, so they're all dependent on external capital.

David Rosenthal

Yeah. Google is the only model maker who has self-sustaining funding.

Ben Gilbert

Yes. Isn't that crazy?

David Rosenthal

Yeah.

Ben Gilbert

Basically, all the other large-scale-usage foundational-model companies are effectively startups.

David Rosenthal

Yes.

Ben Gilbert

And Google's is funded by a money funnel so large that they're giving extra dollars back to shareholders for fun.

David Rosenthal

Yeah.

Ben Gilbert

Again, we're in the bull case.

David Rosenthal

Well, when you put it that way. Yeah, a thing we didn't mention: Google has incredibly fat pipes connecting all of their data centers. After the dot-com crash in 2000, Google bought all that dark fiber for pennies on the dollar, and they've been activating it over the last decade. They now have their own private backbone network between data centers. No one has infrastructure like this.

Ben Gilbert

Yep.

David Rosenthal

Not to mention that it serves YouTube.

Ben Gilbert

Which in and of itself is its own bull case for Google in the future.

David Rosenthal

That's a great point.

Ben Gilbert

Yeah, Ben Thompson had a big article about this yesterday, at the time of recording.

David Rosenthal

Yeah, that was like a mega bull case that Ben Thompson published this week. It was an interesting point. A text-based internet is kind of the old internet. It's the first instantiation of the internet because we didn't have much bandwidth. The user experience that is actually compelling is—

Ben Gilbert

Video.

David Rosenthal

High-resolution video everywhere, all the time.

Ben Gilbert

We already live in the YouTube internet.

David Rosenthal

Right? And not only can they train models on really the only scaled source of UGC media across long-form and short-form, but they also have that as the number 2 search engine, this massive destination site. So they previewed things like you'll be able to buy AI-labeled or AI-determined things that show up in videos. And if they wanted to, they could just go label every single product in every single video and make it all instantly shoppable. It doesn't require any human work to do it. They could just do it and then run their standard ads model on it. That was a mind-expanding piece that Ben published yesterday—or, I guess, if you're listening to this a few weeks ago—about that. And then there's also all the video AI applications that they've been building, like Flow and Veo. What is that going to do for generating videos for YouTube that will increase engagement and add dollars for YouTube?

Yep.

Ben Gilbert

Going to work real well.

David Rosenthal

Yep. They still have an insane talent bench. Even though they've bled talent here and there and lost people, they have also shown they're willing to spend billions for the right people and retain them.

Ben Gilbert

Unit economics. Let's talk about unit economics of chips. Everyone is paying NVIDIA 75–80% gross margins, implying something like a 4x or 5x markup on what it costs to make the chips. A lot of people refer to this as the Jensen tax or the NVIDIA tax. You can call it that, you can call it good business, you can call it pricing power, you could call it scarcity of supply—whatever you want. But that is true. Anyone who doesn't make their own chips is paying a giant, giant premium to NVIDIA.

Google has to still pay some margin to their chip hardware partner, Broadcom, which handles a lot of the work to actually make the chip interface with TSMC. I have heard that Broadcom has something like a 50% margin when working with Google on the TPU versus NVIDIA's 80%. But that's still a huge difference to play with. A 50% gross margin from your supplier or an 80% gross margin from your supplier is the difference between a 2x markup and a 5x markup.

David Rosenthal

Yeah, I guess that's right.

Ben Gilbert

When you frame it that way, it's actually a giant difference in the impact to your cost. So you might wonder, appropriately, well, are chips actually the big part of the cost of the total cost of ownership of running one of these data centers or training one of these models? Chips are the main driver of the cost. They depreciate very quickly. I mean, this is at best a 5-year depreciation because of how fast we are pushing the limits of what we can do with chips, the needs of next-generation models, and how fast TSMC is able to produce.

David Rosenthal

Yeah. I mean, even that is ambitious, right? If you think you're going to get 5 years of depreciation on AI chips, 5 years ago we were still 2 years away from ChatGPT, right? Or think about what Jensen said when we were at GTC this year. He was talking about Blackwell, and he said something about Hopper, and he was like, "Eh, you don't want Hopper. My sales guys are going to hate me, but you really don't want Hopper at this point." I mean, these were the H100s. This was the hot chip just when we were doing our most recent NVIDIA episode.

Ben Gilbert

Yes. Things move quickly.

David Rosenthal

Yes. So I've seen estimates that over half the cost of running an AI data center is the chips and the associated depreciation.

The human cost of R&D is actually a pretty high amount because hiring these AI researchers and all the software engineering is meaningful. Call it 25% to 33%. The power is actually a very small part. It's like 2% to 6%.

So when you're thinking about the economics of doing what Google's doing, it's actually incredibly sensitive to how much margin you're paying your supplier in the chips, because it's the biggest cost driver of the whole thing.

Ben Gilbert

Mhm.

David Rosenthal

So I was sanity-checking some of this with Gavin Baker, who's a partner at Atreides Management, to prep for this episode. He's a great public equities investor who's studied the space for a long time. We actually interviewed him at the NVIDIA GTC pregame show, and he pointed out that normally, in historical technology eras, it hasn't been that important to be the low-cost producer.

Google didn't win because they were the lowest-cost search engine. Apple didn't win because they were the lowest-cost. That's not what makes people win. But this era might actually be different, because these AI companies don't have 80% margins the way that we're used to in the technology business, or at least in the software business. At best, these AI companies look like 50% gross margins.

So Google being definitively the low-cost provider of tokens because they operate all their own infrastructure and because they have access to low-markup hardware, it actually makes a giant difference and might mean that they are the winner in producing tokens for the world.

Ben Gilbert

Very compelling bull case there.

David Rosenthal

That's a weirdly winding analytical bull case, but it's kind of the—if you want to really get down to it, they produce tokens.

Ben Gilbert

Yep. I've got one more bullet point to add to the bull case for Google here. Everything that we talked about in Part II, the Alphabet episode—all of the other products within Google: Gmail, Maps, Docs, Chrome, Android—that is all personalized data about you that Google owns, that they can use to create personalized AI products for you that nobody else has.

David Rosenthal

Another great point. So really, the question to close out the bull case is: Is AI a good business to be in compared to search? Search is a great business to be in. So far, AI is not. But in the abstract, again, we're in the bull case, so I'll give you this: It should be.

With traditional web search, you type in 2 to 3 words. That's the average query length. And I was talking to Bill Gross, and he pointed out that in AI chat, you're often typing 20-plus words. So there should be an ad model that emerges, and ad rates should actually be dramatically higher because you have perfect precision—

Ben Gilbert

Right? You have even more intent.

David Rosenthal

Yes, you know the crap out of what that user wants. So you can really decide to target them with the ad or not. And AI should be very good at targeting with the ad. So it's all about figuring out the user interface, the mix of paid versus not, exactly what this ad model is. But in theory, even though we don't really know what the product looks like now, it should actually lend itself very well to monetization.

Ben Gilbert

Yep.

David Rosenthal

And since AI is such an amazing, transformative experience, all these interactions that were happening in the real world or weren't happening at all, like answers to questions and time spent, are now happening in these AI chats. So it seems like the pie is actually bigger for digital interactions than it was in the search era. So again, monetization should kind of increase because the pie increases there.

Ben Gilbert

Yep.

David Rosenthal

And then you've got the bull case that Waymo could be its own Google-sized business.

Ben Gilbert

I was just thinking that, yeah. That's scoping all of this to a replacement for the search market. Waymo and potentially other applications of AI beyond the traditional search market could add to that—

David Rosenthal

Right? And then there's the galaxy-brain bull case, which is if Google actually creates AGI, none of this even matters anymore. And, of course, it's the most valuable thing.

Ben Gilbert

That feels out of the scope for an Acquired episode.

David Rosenthal

It's disconnected. Yes, agreed. Bear case.

So far, this is all fun to talk about, but then the product shape of AI has not lent itself well to ads. So despite more value creation, there's way less value capture. Google makes something like $400 per user per year, just based on some napkin math in the U.S. That's a free service that everyone uses, and they make $400 a year. Who's going to pay $400 a year for access to AI? It's a very thin slice of the population.

Ben Gilbert

Some people certainly will, but not every person in America.

David Rosenthal

Some people will pay $10 million, but right. So if you're only looking at the game on the field today, I don't see the immediate path to value capture. And think about when Google launched in 1998: It was only 2 years before they had AdWords. They figured out an amazing value-capture mechanism instantly, very quickly.

Ben Gilbert

Yep. Another bear case: Think back to Google's launch in 1998. It was immediately obvious that it was the superior product.

David Rosenthal

Yes.

Ben Gilbert

Definitely not the case today.

David Rosenthal

No, there's 4 or 5 great products.

Ben Gilbert

Google's dedicated AI offering, its chatbot, was initially the immediately obviously inferior product, and now it's arguably on par with several others, right? They own 90% of the search market. I don't know what they own of the AI market, but it ain't 90%. Is it 25%? I don't know. But at steady state, it probably will be something like 25%, maybe up to 50%. But this is going to be a market with several big players in it. So even if they monetized each user as well as they monetize it in search, they're just going to own way less of them.

David Rosenthal

Yep. Or at least it certainly seems that way right now.

Ben Gilbert

Yes. AI might take away the majority of the use cases of search. And even if it doesn't, I bet it takes away a lot of the highest-value ones.

David Rosenthal

Mhm.

Ben Gilbert

If I'm planning a trip, I'm planning that in AI. I'm no longer searching on Google for things that are going to land Expedia ads in my face.

David Rosenthal

Or health, another huge vertical.

Ben Gilbert

Hey, I think I might have something that reminds me of mesothelioma. Is it that or not—

David Rosenthal

Right?

Ben Gilbert

Oh, where are you going to put the lawyer ads? Maybe you put them there. Maybe it's just an ad product thing, but these are very high-value—

David Rosenthal

Queries—

Ben Gilbert

Former searches that feel like some of the first things that are getting siphoned off to AI.

David Rosenthal

Yep.

Ben Gilbert

Any other bear cases? I think the only other bear case I would add is that they have the added challenge now of being the incumbent this time around, and people and the ecosystem aren't necessarily rooting for them in the way that people were rooting for Google when they were a startup, and in the way that people were still rooting for Google in the mobile transition. I think the startups have more of the hearts and minds these days—

David Rosenthal

Right? So I don't think that's quantifiable, but it's just going to make it all a little harder path to row this time around.

Ben Gilbert

Yep. You're right. They had this incredible PR and public-love tailwind the first time around.

David Rosenthal

Yep. And part of that's systemic, too. All of tech and all of big tech is just generally more out of favor with the country and the world now than it was 10 or 15 years ago.

Ben Gilbert

There's more important. It's just big infrastructure. It's not underdogs anymore.

David Rosenthal

Yep. And that affects OpenAI, Anthropic, and the startups too, but I think to a lesser degree.

Ben Gilbert

Yeah, they had to start behaving like big tech companies really early in their life compared to Google. I mean, Google gave a Playboy interview during the quiet period of their IPO. Times have changed.

David Rosenthal

Well, I mean, given all the drama at OpenAI, I don't know that I characterize them as acting like a mature company.

Ben Gilbert

Fair. Fair—

David Rosenthal

Company, entity, whatever they are.

Ben Gilbert

Yes.

David Rosenthal

Yeah. But point taken.

Well, I worked most of my playbook into the story itself. So, you want to do power?

Ben Gilbert

Yeah. Great. Let's move on and do power.

Hamilton Helmer's Seven Powers analysis of Google here in the AI era. And the Seven Powers are scale economies, network economies, counter-positioning, switching costs, branding, cornered resource, and process power. And the question is: Which of these enables a business to achieve persistent differential returns? What entitles them to make greater profits than their nearest competitor sustainably?

Normally, we would do this on the business all up. I think for this episode we should try to scope it to AI products.

David Rosenthal

Yes, agreed. Usage of Gemini, AI Mode, and AI Overviews versus the competitive set of Anthropic, OpenAI, Perplexity, Grok, Meta AI, et cetera.

Scale economies, for sure. Even more so in AI than traditionally in tech.

Ben Gilbert

Yeah, they're just way better. I mean, look, they're amortizing the cost of model training across every Google search. I'm sure it's some super-distilled-down model that's actually happening for AI Overviews, but think about how many inference tokens are generated for the other model companies and how many inference tokens are generated by Gemini. They just are amortizing that fixed training cost over a giant, giant amount of inference.

I saw some crazy chart. We'll send it out to email subscribers. In April of 2024, Google was processing 10 trillion tokens across all their surfaces. In April of 2025, that was almost 500 trillion.

David Rosenthal

Wow.

Ben Gilbert

That's a 50x increase in 1 year of the number of tokens that they're vending out across Google services through inference. And between April of 2025 and June 2025, it went from a little under 500 trillion to a little under 1 quadrillion tokens. Technically, 980 trillion, but they are now, because it's later in the summer, definitely sending out maybe even multiple quadrillion tokens.

David Rosenthal

Wow.

Ben Gilbert

Wow. So among all the other obvious scale economies—amortizing all the costs of their hardware—they are amortizing the cost of training runs over a massive amount of value creation.

Yeah, scale economies must be the biggest one.

David Rosenthal

I find switching costs to be relatively low. I use Gemini for some stuff, then it’s really easy to switch away. That probably stops being the case when it’s personal AI, to the point that you’re talking about integrating with your calendar and your mail and all that stuff. The switching costs have not really come out yet in AI products, although I expect they will.

Ben Gilbert

Yes, they have within the enterprise for sure.

David Rosenthal

Yep.

Ben Gilbert

Network economies. I don’t think if anyone else is a Gemini user, it makes it better for me because they’re sucking up the whole internet whether anyone’s participating or not.

David Rosenthal

Yep, agree. I’m sure AI companies will develop network economies over time. I can think of ways it could work, but right now, no. And arguably, for the foundational model companies, I can’t think of obvious reasons right now. Where does Hamilton put distribution? Because that’s a thing that they have right now that no one else has, despite ChatGPT having the Kleenex brand. Google’s distribution is still unbelievable. I don’t know. Is that a cornered resource?

Ben Gilbert

Cornered resource, I guess. Yeah.

David Rosenthal

Definitely have that.

Ben Gilbert

Yeah, Google Search is a cornered resource for sure.

David Rosenthal

Certainly don’t have counter-positioning. They’re getting counter-positioned.

Ben Gilbert

Yeah.

David Rosenthal

I don’t think they have process power unless they were coming up with the next Transformer reliably, but I don’t think we’re necessarily seeing that. There’s great research being done at a bunch of different labs. Branding they have—

Ben Gilbert

Yeah, branding is a funny one, right? Well, I was going to say it’s a little bit to my bear-case point about them being the incumbent.

David Rosenthal

It cuts both ways, but I think it’s net positive.

Ben Gilbert

Yeah, probably. For most people, they trust Google. Yeah, they probably don’t trust these who-knows-what AI companies, but I trust Google. I bet that’s actually stronger than any downsides as long as they’re willing to still release stuff on the cutting edge.

David Rosenthal

Yep.

Ben Gilbert

So, to sum it up, scale economies is the biggest one. It’s branding, and it’s a cornered resource—

David Rosenthal

—and potential for switching costs in the future. Yep. Sounds right to me.

Ben Gilbert

But it’s telling that it’s not all of them. In search, it was very obviously all of them or most of them.

David Rosenthal

Yep. Quite telling.

Well, I’ll tell you, after hours and hours spent over multiple months learning about this company, my quintessence when I boil it all down is just that this is the most fascinating example of the innovator’s dilemma ever. I mean, Larry and Sergey control the company. They have been quoted repeatedly saying that they would rather go bankrupt than lose at AI. Will they really?

If AI isn’t as good a business as search—and it kind of feels like, of course it will be. Of course it has to be. It’s just because of the sheer amount of value creation. But if it’s not, and they’re choosing between 2 outcomes, one is fulfilling our mission of organizing the world’s information and making it universally accessible and useful and having the most profitable tech company in the world. Which one wins?

Because if it’s just the mission, they should be way more aggressive on AI Mode than they are right now, and fully flip over to Gemini. It’s a really hard needle to thread. I’m actually very impressed at how they’re managing to currently protect the core franchise, but it might be one of these things where it’s being eroded away at the foundation in a way that just somehow isn’t showing up in the financials yet. I don’t know.

Ben Gilbert

Yep. I totally agree. And in fact, perhaps influenced by you, I think my quintessence is a version of that, too. I think if you look at all the big tech companies, Google, as unlikely as it seems, given how things started, is probably doing the best job of trying to thread the needle with AI right now. And that is incredibly commendable to Sundar and their leadership. They are making hard decisions, like, “We’re unifying DeepMind and Google Brain. We’re consolidating and standardizing on 1 model, and we’re going to ship this stuff real fast,” while at the same time not making rash decisions.

David Rosenthal

It’s hard. Rapid but not rash, you know.

Ben Gilbert

Yes. And obviously, we’re still in the early innings of all this going on, and we’ll see in 10 years where it all ends up. Yeah. Being tasked with being the steward of a mission and the steward of a franchise with public company shareholders is a hard dual mission, and Sundar and the company are handling it remarkably well, especially given where they were 5 years ago.

David Rosenthal

Yep. And I think this will be one of the most fascinating examples in history to watch it play out.

Ben Gilbert

Totally agree. Well, thus concludes our Google series for now.

David Rosenthal

Yes. All right, let’s do some carve-outs.

Ben Gilbert

All right, let’s do some carve-outs. Well, first off, we have a very, very fun announcement to share with you all. The NFL called us.

David Rosenthal

We’re going to the Super Bowl, baby.

Ben Gilbert

Acquired is going to the Super Bowl. This is so cool.

David Rosenthal

It’s the craziest thing ever.

Ben Gilbert

The NFL is hosting an innovation summit the week of the Super Bowl, the Friday before Super Bowl Sunday. The Super Bowl is going to be in San Francisco this year in February. And so it’s only natural, coming back to San Francisco with the Super Bowl, that the NFL should do an innovation summit.

David Rosenthal

Yep.

Ben Gilbert

And we’re going to host it.

David Rosenthal

That’s right. So, the Friday before, there’s going to be some great onstage interviews and programming. Most of you know, we can’t fit millions of people in a tiny auditorium in San Francisco the week of the Super Bowl, when every other venue has tons of stuff, too. So there will be an opportunity to watch that streaming online. And as we get closer to that date in February, we will make sure that you all know a way that you can tune in and watch the MCing, interviewing, and festivities at hand. Super Bowl week.

Ben Gilbert

It’s going to be an incredible, incredible day leading up to an incredible Sunday.

David Rosenthal

Yes. Well, speaking of sport, my carve-out is I finally went and saw F1. It is great. I highly recommend anyone go see it, whether you’re an F1 fan or not. It is just beautiful cinema.

Ben Gilbert

Amazing. Did you see it in the theater?

David Rosenthal

I did see it in the theater. Yeah.

Ben Gilbert

Wow.

David Rosenthal

I unfortunately missed the IMAX window, but it was great. It was my first time being in a movie theater in a while. And whether you watch it at home or whether you watch it in the theater, I recommend the theater. But it’s going to be a great surround-sound experience wherever you are.

Ben Gilbert

I haven’t been to the movie theater since The Eras Tour.

David Rosenthal

Ah.

Ben Gilbert

Which I think is just more about the current state of my family life with 2 young children.

David Rosenthal

Yes. My second one, some of you are going to laugh, is the Travelpro suitcase.

Ben Gilbert

Ah, this is the brand that pilots and flight attendants use, right?

David Rosenthal

Maybe. I think I’ve seen some of them use it. Usually they use something higher-end like a Briggs & Riley or a Tumi or, you know, Travelpro is not the most high-end suitcase, but I bought 2 really big ones for some international travel that we were doing with my 2-year-old toddler. And I must say, they’re robust. The wheels glide really well. They’re really smooth. They have all the features you would want. They’re soft-shell, so you can really jam it full of stuff, but it’s also a thick amount of protection. So even if you do jam it full of stuff, it’s probably not going to break.

This is approximately the most budget suitcase you could buy. I mean, I’m looking at the big honking international checked-bag version. It’s $416 on Amazon right now. I’ve seen it cheaper. They have great sales pretty often. Everything about this suitcase checked lots of boxes for me, and I completely thought I would be the person buying the Rimowa suitcase or something very high-end, and this is just perfect. So, I think I may be investing in more Travelpro suitcases.

Ben Gilbert

More Travelpro. Nice. Nice. Well, I mean, hey, look, for family travel, you don’t want nice stuff.

David Rosenthal

Yeah. I mean, I bought it thinking, like, I’ll just get something crappy for this trip, but it’s been great. I don’t understand why I wouldn’t have a full lineup of Travelpro gear. So—

Ben Gilbert

Amazing.

David Rosenthal

This is my budget pick gone right that I highly recommend for all of you.

Ben Gilbert

I love how Acquired is turning into the Wirecutter here.

David Rosenthal

That’s it for me today.

Ben Gilbert

Great. All right. I have 2 carve-outs. I have 1 carve-out, and then I have an update in my ongoing Google carve-out saga. But first, my actual carve-out: it is the Glue Guys podcast.

David Rosenthal

Oh, it’s great. Those guys are awesome. So great. Our buddy Robbie Gupta, partner at Sequoia, and his buddies Shane Battier, the former basketball player, and Alex Smith, the former quarterback for the 49ers, the Kansas City Chiefs, and the Redskins. Their dynamic is so great. They have so much fun. Half of their episodes, like us, are just them, and then half of their episodes are with guests. Ben and I went on it a couple of weeks ago. That was really fun.

When we were on it, we were talking about this dynamic of some episodes doing better than others and pressure for episodes and whatnot. And the guys brought up this interview they did with a guy named Wright Thompson. And they said, like, “Look, this is an episode. It’s got like 5,000 listens. Nobody’s listened to it. It’s so good.” And the mentality that we have about it is not that we’re embarrassed that nobody listened to it. It’s that we feel sorry for the people who have not yet listened to it because it’s so good.

Ben Gilbert

I was like, that is the way to think about—

David Rosenthal

That’s great.

Ben Gilbert

—your episodes.

David Rosenthal

So here you are. You’re giving everyone the gift of—

Ben Gilbert

I’m giving everyone the gift because then I was like, all right, well, I got to go listen to this episode.

Wright Thompson—I didn't know anything about him before. I probably read his work in magazines over the years without realizing it.

David Rosenthal

He's the coolest dude.

Ben Gilbert

He has the same accent as Bill Gurley. Listening to him sounds like listening to Bill Gurley if, instead of being a VC, he only wrote about sports and basically dedicated his whole life to understanding the mentality and psychology of athletes and coaches. It's so cool. It's a great episode. Highly, highly, highly recommend.

David Rosenthal

All right. Legitimately, I'm queuing that up right now.

Ben Gilbert

Great. That's my carve-out. And then my ongoing family video-gaming saga from Google Part I: I said I was debating between the Switch 2 and the Steam Deck.

David Rosenthal

That's right. First, you got the Steam Deck because you decided your daughter actually wasn't old enough to play video games with you, so you just got the thing for yourself.

Ben Gilbert

The update was that I went with the Steam Deck for that reason. I thought if it was just for me, it would be more ideal. I have an update.

David Rosenthal

You also got a Switch.

Ben Gilbert

No, not yet.

David Rosenthal

Okay.

Ben Gilbert

But the most incredible thing happened. My daughter noticed this device that had appeared in our house that Dad plays every now and then. We were on vacation, and I was playing the Steam Deck. She was like, “What's that?” Well, let me tell you.

I was playing this really cool indie, old-school-style RPG called Sea of Stars. It's like a Chrono Trigger-style, Super Nintendo-style RPG. I'm playing it, and my daughter comes up and says, “Can I watch you play?” I'm like, “Hell yeah, you can watch me play. I get to play video games, and you sit here and snuggle with me. Amazing.”

David Rosenthal

I get to play video games and call it parenting.

Ben Gilbert

Then it gets even better. Probably like 2 weeks ago, we were playing, and she was like, “Hey, Dad, can I try?” I'm like, “Absolutely, you can try.” I hand her the Steam Deck, and it was one of the most incredible experiences I've had as a parent because she doesn't know how to play video games, and I'm watching her learn how to use a joystick and hit the button.

David Rosenthal

Supervised learning. Yeah, yeah, yeah. Supervised learning.

Ben Gilbert

I'm telling her what to do, and then within 2 or 3 nights, she got it. She doesn't even know how to read yet, but she figured it out. I'm watching it happen in real time.

So now, over the last week, it's turned into mostly her playing, and I'm helping her by asking questions like, “What do you think you should do here? Should you go here? I think this is the goal. I think this is where it's so, so fun.” Her birthday's coming up, so I think I might actually end up getting a Switch so that we can play together on the Switch.

David Rosenthal

But unintentionally, the Steam Deck was the gateway drug for my soon-to-be-four-year-old daughter. That's awesome. There you go. Parent of the year right there. Getting to play video games and—“Oh, honey, I got it. I'll take it.”

Ben Gilbert

Oh, yeah. I got it. I got it.

Google Part III: The AI Company. Google is amazingly well-positioned... will they win in AI? (Audio) | BidClub