[BidClub_]
Latent Space · · 78 min

Snipd: The AI Podcast App for Learning — with CEO Kevin Ben-Smith

swyxKevin Ben-Smith

YouTube
TL;DR
  • Snipd found its direction by disproving its first thesis: users enjoyed a TikTok-like discovery feed, but they kept listening to full episodes and created Snips “like crazy.” The four-person, all-technical team shifted from social clips to a learning system that captures, summarizes, and preserves podcast knowledge; a triple tap on headphones converts the just-heard moment into a note. That behavioral evidence gives the business a precise job: stop listeners forgetting “99%” before reflexively starting another episode.
  • The wedge now resembles a structured knowledge layer over audio, with more than a million podcasts already processed. Episodes can receive transcripts, diarized and named speakers, guest bios, AI chapters, extracted books, author-linked appearances, timestamp-sourced chat, and Snips retaining the original text and audio. Dynamic advertising breaks fixed timestamps, so Snipd built fuzzy audio matching—“basically Shazam for podcasts”—to resynchronize whichever file reaches the listener.
  • Snipd’s model strategy is margin- and latency-aware: self-host some cost-sensitive transcription and diarization models, then route higher-level work among OpenAI, Gemini, and Perplexity according to the minimum intelligence required. Perplexity search was quoted at roughly $5 per 1,000 queries, materially more than an ordinary LLM call, while Claude 3.5 Sonnet is admired for its formulation and personality but its cost leads the team to use other models for many workloads. Predictable preprocessing runs continuously; bursty user requests stay on APIs, which Kevin compares to “the AWS of AI.”
  • The defensibility lies in production reliability rather than the ease of a demo: episode chat can be prototyped in 15 minutes, but reaching “99% of the time” requires prompt formats, frontend rendering, heuristics, and countless regexes. Snipd supplements startup-style “vibe evals” with an LLM-as-judge pattern—generate five candidates cheaply, then let a stronger model choose—used for quotes, books, and speakers. Hosted closed models currently win because iteration velocity matters more than uncertain open-model savings.
  • The consumer opportunity is AI that disappears into the workflow, because “you can already ask the chat” is not equivalent to opening an app and finding the desired personalized summary already rendered. Kevin keeps returning to the job to be done: a simple interaction such as clicking a transcript word and jumping to the audio can matter more than model novelty. His endpoint is that “AI is the electricity of the future”—eventually Snipd should simply be a podcast app whose intelligence is assumed.
  • Voice is Snipd’s proposed learning loop: when an episode ends, a two- or three-minute conversation could force the listener to choose one takeaway and connect it to action. More than half a billion monthly podcast listeners already possess the listening habit, so Snipd need not manufacture a new one with Duolingo-style owls, streaks, and notifications. Kevin believes this might “10x” the value of an episode, while explicitly leaving the detailed interaction open for experimentation.
  • Expansion spans content, discovery, video, and creators, but distribution incentives will determine how much of the stack Snipd can own. Kevin wants native audiobooks, YouTube, AI-generated material, recommendation systems users can talk to, and “backgroundable video,” whose principal advantage is discovery even when roughly 90% of consumption accompanies another activity. swyx’s counterproposal is creator tooling—record, press one button, finish—because YouTube is already the strongest podcast platform; hard migrations and Substack-hosted Apple Watch playback failure expose both the opening and the platform risk.
Digest · the substance, structured for research

1. Demand, not the hackathon trophy, created Snipd

  • Kevin studied at ETH Zurich in mathematics and economics, specializing in quantitative finance, but his retrospective signal of weak fit was revealing: “I never read an academic paper about the subject in my free time.” A friend sent him machine-learning lecture slides; after one weekend, his reaction was, “Freaking hell, like this is it. I’m in love.”

  • He hunted for excuses to apply machine learning at his bank, then concluded that pursuing it required “a real cut.” Kevin quit and spent five years building an AI team at an early-stage Zurich startup, delivering bank-sales models and systems that converted cryptic transaction booking text into readable merchant descriptions.

  • The same friend later joined him for HackZurich, where they built natural-language search within podcasts. Their stage demo searched “smoking weed” inside the 2½-hour Joe Rogan episode and found the exact Elon Musk moment. Winning supplied “activation energy,” but the stronger validation came when other participants immediately asked, “Can I use this?” and described adjacent problems.

  • Snipd remains unusually lean: four technical people, split between two backend/AI engineers and two frontend engineers. That team supports iOS, Android, and an Apple Watch app. The product is freemium, with Kevin offering a link for a free month of premium access.

2. A failed TikTok thesis exposed the real job

  • Snipd’s first release arrived roughly 3½ years earlier, before ChatGPT and Whisper. The initial concept was a social, TikTok-like feed: some users would listen to full episodes, clip the best moments into Snips, and let everyone else consume those clips for discovery or as a means to an end.

  • The founders expected attracting Snip viewers to be easy and persuading people to create them to be hard, so they optimized aggressively for creation. Reality inverted the hypothesis: users liked discovering podcasts through clips but still wanted long-form audio—and “they were creating Snips like crazy.” Snipd consequently doubled down on capture, retention, and learning.

  • swyx had lived the underlying pain through an earlier “personal mixtape podcast”: note a timestamp and URL, later download the MP3, manually cut a five-to-10-minute clip, record commentary, and republish it. Kevin’s reframing was that podcast apps remain “repurposed music players” despite podcasts being “one of the largest sources of knowledge in the world.”

3. Every episode becomes a navigable knowledge object

  • swyx’s baseline requirement was uncompromising: Snipd still had to be a full podcast player, effectively a superset of Overcast or Apple Podcasts. He initially disliked its opt-in queue and download behavior, but found that visible Snip counts and their positions inside an episode became a useful proxy for audience interest.

  • For enabled episodes, Snipd transcribes the audio, diarizes and names speakers, generates a mini biography and image for each guest, creates titled chapters with descriptions, and exposes a clickable transcript. Selecting transcript text jumps directly to its timestamp—a simple interaction Kevin uses to illustrate why consumer value is not synonymous with AI sophistication.

  • Book extraction goes beyond detecting a title. An LLM reads the transcript, while Perplexity and other orchestration retrieve the cover, author, and description; Snipd then shows other episodes featuring that author. Earlier models over-selected frequently mentioned people such as Sam Altman and Elon Musk, so the team tightened the product toward genuine guest appearances.

  • A listener can keyword-search or chat with an episode, ask when a topic appears, request takeaways, and follow timestamp citations back to audio. Snips preserve the summarized insight alongside transcript and sound; runners can create them from the Watch, although Kevin said Substack-hosted podcasts cannot play there—even through Apple Podcasts—and that Substack did not seem to care despite outreach.

4. Dynamic advertising forced Snipd to build “Shazam for podcasts”

  • The only processing-scale number Kevin disclosed was more than one million podcasts. Some shows are handled automatically; others enter the pipeline after a paying user requests them. Transcription and diarization produce speech blocks, after which LLM orchestration identifies guests and assigns names to the appropriate blocks.

  • Dynamic ads make ordinary timestamps unreliable. Each playback request can return a different MP3 with advertising inserted according to factors such as IP address; if Snipd transcribed another version, every word-level timestamp after the insertion shifts, breaking chapters, transcript navigation, chat citations, and Snips.

  • Snipd therefore resynchronizes the listener’s audio against its transcript on the fly. The matching happens near the audio-byte level rather than through partial transcription, and it is fuzzy rather than dependent on exact equality. Kevin’s concise description: “We basically built Shazam for podcasts” as a side project needed to make the main product work.

5. The stack buys only the intelligence each job requires

  • Roughly 90% of the backend is Python on Google Cloud Platform. The mobile application uses Flutter and Dart from one codebase, with native work where Flutter does not reach, such as Apple Watch. Asked whether Flutter was a good decision, Kevin’s bounded answer was “up until now, yes”; the team has no plan to leave it.

  • Snipd began before GPT-3.5 Turbo and initially ran and fine-tuned open models itself. For transcription it used wav2vec 2.0, whose combination of transformers, continuous audio, and self-supervised learning convinced Kevin that audio would follow text’s trajectory—even if the desired product capabilities did not exist yet.

  • Today it uses Whisper and still self-hosts some models for cost-sensitive transcription and diarization, while most downstream work uses OpenAI and Gemini APIs. Perplexity handles tasks needing web search; Kevin cited about $5 per 1,000 queries and said Google search grounding was not cheaper enough to justify replacing a system that already worked.

  • The governing rule is not “take the best model,” but identify the intelligence level a task needs and buy the best price at that level. Claude 3.5 Sonnet is Kevin’s favorite for formulation, personality, coding, and brainstorming, yet its cost is still high, so the team opts for different models for many workloads. “Given that we do work with a lot of content, price is actually something that we do look at.”

6. Podcast structure makes diarization tractable, not solved

  • Snipd benefits from a bounded domain: podcasts generally provide high-quality, controlled, studio-recorded audio and consistent background noise. Kevin contrasted that with meeting-oriented voice products that must handle arbitrary rooms and conditions, where usable structural heuristics and in-domain data are far scarcer.

  • One heuristic is airtime: in a one-hour episode, a voice heard for only 30 seconds is probably neither host nor guest, but an inserted advertiser. The underlying system embeds speech segments and clusters them by speaker; Snipd has modified the clustering and applies podcast-specific rules unavailable to a generic diarization service.

  • LLMs combine transcript meaning with acoustic clusters to identify speakers and recalibrate transition points. swyx objected that LLMs can introduce errors and are unsafe as precise authorities; Kevin conceded plainly that Snipd’s diarization “is also not perfect,” though he said recent episodes had improved substantially over those processed a year earlier.

  • swyx nevertheless judged Snipd’s presentation and identification better than Descript’s in a difficult, multi-guest TechMeme Ride Home episode. Both speakers expect today’s specialized pipeline eventually to collapse into a multimodal model ingesting raw audio; Kevin tested Gemini 1.5 Flash, but said its cost remained far above the self-hosted pipeline.

7. Production quality lives in regexes, judges, and “vibe evals”

  • Episode chat demonstrates the distance from prototype to product. Pasting a transcript into ChatGPT and requesting timestamped answers can yield a demo in “15 minutes,” Kevin said; making it work correctly “99% of the time” requires carefully formatted context, prompt engineering, citation behavior, frontend rendering, and recovery from malformed responses.

  • Snipd uses “countless regexes” to repair formatting mistakes before they become ugly UI. A second LLM might sound more AI-native, but chat must stream immediately; the available APIs cannot conveniently stream incomplete output into another live corrective stream. swyx’s verdict was that this supposedly inelegant machinery is ordinary “real-world engineering.”

  • Kevin characterized much of the company’s evaluation culture as “vibe evals.” A startup can tolerate occasional roughness because users receive rapid iteration in exchange, whereas a Spotify-scale organization might subject half the same features to six months of legal and organizational review. The trade is speed, not a claim that failures have disappeared.

  • Snipd Wrapped exposed the tail problem: a model usually selected an interesting representative quote, but occasionally returned something inexplicably dull. The fix was to generate five candidates with a cheaper model, then have a much stronger LLM judge choose one. Snipd applies the same challenger pattern when validating detected books and speakers.

8. Consumer AI wins when the chat box disappears

  • swyx’s formal feature request was customizable summarization through a user-supplied prompt. Kevin noted that long-context personalization remains expensive if every summary must be regenerated for every user, although falling model costs should eventually make “everything” personalized.

  • Kevin’s immediate answer was technically correct: open episode chat and ask for a summary in the desired style. But he then accepted the deeper request—swyx listens while moving and does not want to initiate a chat. The real product should open with the preferred information already generated, structured, and visually consumable.

  • That distinction underpins Kevin’s consumer-AI thesis: builders must “move beyond the chat box.” Intelligence may already be available, but the opportunity lies in identifying the natural interface through which a person engages with it. His recurring discipline for AI-oriented founders is to ask, “What’s actually the job to be done here?”

  • The endpoint is invisible infrastructure. Kevin’s analogy was that “AI is the electricity of the future”: nobody markets a microphone or phone as electricity-enabled, and eventually AI-enabled will sound equally redundant. Snipd uses the phrase now because novelty attracts attention, but its desired destination is simply a better podcast product.

9. Voice can turn an episode ending into a learning trigger

  • Kevin evaluates consumer products by the real-life trigger that causes someone to open them: travel prompts Airbnb, but language study supplies no natural daily moment. Duolingo is the exceptional company that manufactured such a habit through notifications, streaks, leaderboards, and “the owl memes”; Kevin called it “the GOAT” of that game.

  • Snipd already inherits a trigger—being alone, walking, driving, exercising, or otherwise starting a podcast—but lacks one for processing what was learned. Kevin’s own forcing function is to choose exactly one takeaway after every episode, even when ten ideas were valuable, because the prioritization tests relevance and makes action more likely.

  • His proposed voice product starts when playback ends: rather than rolling into another episode, an AI companion holds a two- or three-minute reflection. More than half a billion monthly podcast listeners already have the preceding habit. If users feel that this tiny investment materially improves retention, application, and thinking, Kevin believes Snipd may “10x the value” they receive.

  • The implementation remains deliberately unsettled, and swyx initially resisted the framing as another “chat with your podcast” feature before recognizing the post-listening use case. Kevin also expects voice cloning to normalize: capabilities that provoked grave ethical debate at an earlier NeurIPS workshop now exist in mass-market tools, and an authorized “AI swyx” could eventually discuss Latent Space episodes.

10. YouTube turns discovery into a platform question

  • Kevin’s content roadmap extends beyond RSS podcasts: users can already add YouTube videos and upload audiobooks, while he wants native audiobooks and AI-generated programming that combines Deep Research with NotebookLM-style podcast generation. swyx pushed him to prioritize video podcasts over audiobooks because, in his view, YouTube is already “the best podcasting platform.”

  • Discovery should also move from opaque personalization to an algorithm the user can address. Kevin’s example was telling a feed: “I know I freaking love” cat videos, but for the next two hours show AI material instead. TikTok may lack the incentive to surrender engagement optimization; a learning product aligned with user goals does not.

  • Kevin adopted Spotify CPO Gustav Söderström’s phrase “backgroundable video”: roughly 90% of podcast consumption still accompanies another activity, even on YouTube, but visuals help with clips, slides, demonstrations, and connection to hosts. His largest reason for adding video is discovery—video recommendations are more engaging, while full attention remains optional.

  • swyx identified podcast creators as a third constituency beyond listeners and Snip makers. Riverside comes closest and Descript owns editing, yet neither delivers his desired workflow: “sit down, record, press the button, done.” The opportunity is paired with real switching friction—despite loving Snipd, he waited four to six months to migrate because OPML could not carry half-listened episodes, rankings, and accumulated state.

swyx

Hey, I’m here in New York with Kevin Ben-Smith of Snipd. Welcome.

Kevin Ben-Smith

Hi. Hi. Amazing to be here.

swyx

Yeah. This is our first-ever outdoor podcast recording, I think. It’s quite a location for the first time, I’d say.

Kevin Ben-Smith

I was actually unsure because it’s cold. I checked the temperature; it’s 1°C.

swyx

It’s not that bad with the sun.

Kevin Ben-Smith

No, it’s quite nice.

swyx

Yeah. Yeah. Especially with our beautiful tea.

Kevin Ben-Smith

With the tea.

swyx

Yeah. Perfect. We’re going to talk about Snipd. I’m a Snipd user. Apart from Twitter, it’s the number-one-used app on my phone. When I wake up in the morning, I open Snipd and see what’s new. In terms of time spent or usage on my phone, I think it’s number 1 or number 2.

I had to talk about it because we’re in an AI podcast. We have to talk about AI podcasts. But before we get there, we just finished the AI Engineer Summit, and you came for the 2 days. How was it?

Kevin Ben-Smith

It was quite incredible. For me, the most valuable part was being in the same room with like-minded people who are building the future and seeing the future. Especially when it comes to AI agents, I often have conversations with friends who aren’t in the AI world, and it happens so quickly that it sounds like you’re talking in science fiction. It’s just crazy talk.

It was so refreshing to talk with so many other people who already see these things and be inspired by them, rather than always feeling like, “Okay, I think I’m just crazy, and this will never happen.” It really is happening, and for me it was very valuable.

swyx

So day 2 was more relevant for you than day 1?

Kevin Ben-Smith

Yeah, day 2 was the engineering track. That was definitely the most valuable for me, also as a practitioner myself. There were 1 or 2 talks about voice AI and AI agents with voice, which was quite fascinating. I also spoke with the speakers afterward, and they were very open. There’s this sharing attitude that I think is generally quite prevalent in the AI community. I learned a lot of practical things that I can now take away with me.

swyx

Yeah. On my side, I watched only about half of the talks because I was running around. I think people saw me toward the end; I was kind of collapsing. I was on the floor toward the end because I needed to get some rest. But I’m excited to watch the voice AI talks myself.

Kevin Ben-Smith

Yeah, do that. From my side, thanks a lot for organizing this conference and bringing everyone together. Do you have anything like this in Switzerland?

swyx

The short answer is no. I have to say, the AI community in Zurich, especially where we’re based, is quite good and growing. It’s especially driven by ETH, the technical university there, and all of the big companies that have AI teams there.

Google has its biggest tech hub outside the U.S. in Zurich. Meta is doing a lot with Reality Labs. Apple has a secret AI team. OpenAI is there, and SwiftKey just announced that they’re coming to Zurich. So there’s a lot happening.

Yeah, I think the most recent notable move was that the entire vision team from Google—Lucas Beyer and all the other authors of SigLIP—left Google to join OpenAI. I thought that was a big move, for a whole team to move all at once and at the same time.

I’ve been to Zurich, and it just feels expensive. It’s a great city with a great university, but I don’t see it as a business hub. Is it a business hub? I guess it is, right?

Kevin Ben-Smith

Historically, it’s a finance hub.

swyx

A finance hub?

Kevin Ben-Smith

Yeah. There are some large banks there, especially UBS, the largest wealth manager in the world. But it’s really becoming more of a tech hub now, with all of the big tech companies there.

swyx

And research-wise, is it all ETH, or are there other things?

Kevin Ben-Smith

Yeah, it’s all driven by ETH, and then there’s the university EPFL in Lausanne, which is also doing a lot. But it’s really ETH.

swyx

Otherwise, it’s a beautiful city. I can recommend that anyone come visit Zurich. Let me know; I’d be happy to show you around. Of course, you have nature so close, the mountains so close, and beautiful lakes. I think that’s what makes it such a livable city.

The cost isn’t cheap, but we’re in New York City right now, and I paid $8 for a coffee this morning. The coffee is cheaper in Zurich than in New York City.

Okay, let’s talk about Snipd. What is Snipd? Then we’ll talk about your origin story, but let’s get it crisp. What is Snipd?

Kevin Ben-Smith

I always see 2 definitions of Snipd. I’ll give you 1 really simple, straightforward one and then a second, more nuanced one, which I think will be valuable for the rest of our conversation.

The simplest way to put it is that we’re an AI-powered podcast app. If you listen to podcasts, we’re providing this AI-enhanced experience. But from a more nuanced perspective, we have a big focus on people like your audience who listen to podcasts to learn something new. Your audience wants to learn about AI—what’s happening, what’s the latest research, what’s going on—and we want to provide a spoken-audio platform where you can do that most effectively. AI is basically the way we can achieve that.

swyx

Means to an end.

Kevin Ben-Smith

Exactly.

swyx

When you started, was it always meant to be AI, or was it more about social sharing?

Kevin Ben-Smith

The first version that we ever released was about 3.5 years ago. This was before ChatGPT.

swyx

Before Whisper?

Kevin Ben-Smith

Yeah, before Whisper. A lot of the features that we now have in the app weren’t really possible yet back then. But from the beginning, we always had a focus on knowledge. That’s the reason why our team listens to podcasts.

We did have a different approach. The idea in the very beginning was that the name is Snipd, and you can create what we call Snips, which are basically small snippets or clips from a podcast. We envisioned a social platform, sort of like TikTok, where some people would listen to full episodes, snip certain best parts of them, and post those in a feed. Other users would consume this feed of Snips and use it as a discovery tool or as a means to an end.

You would have both people who create Snips and people who listen to Snips. Our big hypothesis in the beginning was that it would be easy to get people to listen to these Snips but super difficult to get them to create them. So we focused a lot of our effort on making it as seamless and easy as possible to create a Snip.

swyx

It’s similar to TikTok. You need CapCut for there to be videos on TikTok.

Kevin Ben-Smith

Exactly. For Snipd, whenever you hear an amazing insight or a great moment, you just triple-tap your headphones. Our AI then saves the moment that you just listened to and summarizes it to create a note. That’s basically a Snip.

We built all of this, launched it, and found the exact opposite. People used Snips to discover podcasts, but they really loved listening to long-form podcasts. They were creating Snips like crazy. This was definitely one of those aha moments when we realized that we should double down on knowledge and learning—helping you learn most effectively and capture the knowledge that you listen to, so you can actually do something with it.

We live in a world where there’s so much content. We consume and consume and consume, and it’s so easy to finish one podcast and immediately start listening to the next one. Five minutes later, you’ve forgotten 99% of what you actually just learned.

swyx

You don’t notice it, and most people don’t notice it, but this is my fourth podcast. My third podcast was a personal mixtape podcast where I manually snipped sections of podcasts that I liked, added my own commentary on top of them, and published them as small episodes.

They would be 5- to 10-minute snips of something that I thought was a good story or a good insight, and then I added my own commentary and published it as a separate podcast.

Kevin Ben-Smith

It’s cool. Is that still live?

swyx

It’s still live, but it’s not active. You can go back and find it if you’re curious enough. You’ll see it.

Kevin Ben-Smith

Nice, nice. You have to show me later.

swyx

It was very manual. My process would be: I’d hear something interesting, note down the timestamp and the URL of the podcast, and put it in my note-taking app. I used to use Overcast, so it would just link to the Overcast page. Then, whenever I felt like publishing, I would take one of those items, download the MP3, cut out the clip, record my intro and outro, and publish it as a podcast.

But now, with Snipd, I can just double-click or triple-tap.

Kevin Ben-Smith

Those are very similar stories to what we hear from our users. It’s normal that you’re doing something else while listening to your podcast. A lot of our users are driving, working out, or walking their dog. In those moments, when you hear something amazing, it’s difficult to write it down. You have to take out your phone.

Some people take a screenshot, write down the timestamp, and then later have to go back and try to find it again. Of course, you can’t find it anymore because there’s no search—there’s no Command-F. These were all issues that we encountered ourselves as users, and given that our background was in AI, we realized, “Wait, this should not be the case.”

Podcast apps today are basically repurposed music players, but we look at podcasts as one of the largest sources of knowledge in the world. Once you have that different angle, together with everything that AI is now enabling, you realize, “Hey, this is not the way podcast apps should be.”

swyx

Yeah, I agree. You mentioned something there: you said your background’s in AI. First of all, who’s on the team, and what do you mean by your background being in AI? Those are 2 very different questions.

Kevin Ben-Smith

Maybe starting with my backstory: it actually goes back, let’s say, 12 years or something like that. I moved to Zurich to study at ETH, and I studied something completely different. I studied mathematics and economics, basically specializing in quant finance.

swyx

Okay, well, all right.

Kevin Ben-Smith

So, yeah, there were all of these mathematical models for asset pricing, derivative pricing, and quantitative trading. For me, what fascinated me most was mathematical modeling, mathematics, and statistics, but I was never really that passionate about the finance side of things.

swyx

Really? Oh, okay. Yeah, I mean, we’re different there.

Kevin Ben-Smith

One symptom that I notice now, looking back, is that during that time, I think I never read an academic paper about the subject in my free time.

Then, toward the end of my studies, I was already working for a big bank. One of my best friends comes to me and says, “Hey, I just took this course. You have to do this. You have to take this lecture.”

I’m like, “What is it about?” He says, “It’s called machine learning.” And I’m like, “What kind of stupid name is that?” He sent me the slides, and over a weekend I went through all of them. I just knew: “Freaking hell, this is it. I’m in love.”

swyx

Wow. Yeah. Okay.

Kevin Ben-Smith

Over the course of the next 12 months, I really got into it. I started reading all about it, reading blog posts, and building my own models.

swyx

Was this course by a famous person at a famous university? Was it a Coursera thing?

Kevin Ben-Smith

No, this was an ETH course.

swyx

Oh, it was ETH. A professor at ETH? Did he teach in English, by the way?

Kevin Ben-Smith

Yeah, yeah, yeah.

swyx

Okay. So these slides are available somewhere?

Kevin Ben-Smith

Yeah, definitely. Though now they’re quite outdated.

swyx

Yeah, sure, sure.

swyx

Reflecting on the finance thing for a bit, I used to be a trader, on the sell side and buy side. I was an options trader first, and then I was more of a quantitative hedge fund analyst. We never really used machine learning. It was more like a little bit of statistical modeling, where you fit your regression.

swyx

No, I mean, that’s what it is. Or you solve partial differential equations and then use numerical methods to solve them. That’s for your degree. That’s not really what you do at work, right? Unless I don’t know what you do at work.

Kevin Ben-Smith

In my job, no. No, we weren’t solving the partial differential equations.

swyx

You learn all this in school and then you use it.

Kevin Ben-Smith

Let’s put it like this: in some things, yeah, I did code algorithms that would do it, but they were basically the most basic algorithms, and then you just slightly improved them. You tweak them here and there. It wasn’t like starting from scratch with a new partial differential equation.

swyx

No. Yeah, I mean, that’s real life, right? Most of it is kind of boring, or you’re using established things because they’re established because they tackle the most important topics.

swyx

Yeah, portfolio management was more interesting for me. We were sort of the first to combine social data with quantitative trading, and I think now it’s very common.

swyx

Then you went deep on machine learning. What happened next? You quit your job?

Kevin Ben-Smith

Yeah.

swyx

Wow.

Kevin Ben-Smith

I quit my job because I started using it at the bank as well. I desperately tried to find any kind of excuse to use it here or there, but it was clear to me that if I wanted to do this, I had to make a real cut.

So I quit my job and joined an early-stage tech startup in Zurich, where I built up the AI team over 5 years.

swyx

Wow.

Kevin Ben-Smith

We built various machine-learning systems for banks, from models for sales teams to identify which clients would like which product, what to sell to them, and for what reasons, all the way to doing a lot with bank transactions.

One of the most fun projects for me was an NLP model that would take the booking text of a transaction, such as a credit-card transaction, and prettify it. They had all of these numbers, abbreviations, and whatnot in there. Sometimes you’d look at it and think, “What is this?” The model would just change it to something like “CVS.”

swyx

Would you have hallucinations?

Kevin Ben-Smith

No, no, no. The way everything was set up, it wasn’t yet a fully end-to-end innovative neural network like what you would use today.

swyx

Okay, okay, awesome. And then when did you go full-time on Snipd?

Kevin Ben-Smith

That was afterward. The friend who got me into machine learning also got me interested in startups. He’s had a big impact on my life. His background is also in AI and data science.

The 2 of us would just jam on startup ideas every now and then. We had a couple of ideas, but because we were working full-time, we thought, “We could participate in HackZurich. It’s just a weekend. Let’s try out an idea, hack something together, and see how it works.”

The idea was that we’d be able to search through podcast episodes within a podcast. We did that, and long story short, we managed to build something that made us realize, “Hey, this actually works.” You could find things again in podcasts through natural-language search.

We pitched it on stage, and we actually won the hackathon, which was cool. I think we also had a good pitch and a good example. We used the famous Joe Rogan episode with Elon Musk where Elon Musk smokes a joint. It’s a 2½-hour episode, so we were on stage and searched for “smoking weed.” It would find that exact moment and play it, with Elon Musk coming on and smoking.

swyx

Was it video as well?

Kevin Ben-Smith

No, it was completely based on audio, but we did have the video for the presentation, which, of course, had an amazing effect. That gave us a lot of activation energy, but it wasn’t actually about winning the hackathon.

The interesting thing that happened was that after we pitched on stage, several of the other participants came up to us—many of them—and started saying, “Can I use this? I have this issue.” Some also told us about other problems that were very adjacent to this, asking whether they could use it for those as well.

That was the moment when we realized it wasn’t just us having these issues with podcasts and getting the most out of this knowledge.

swyx

Yeah, there are other people.

Kevin Ben-Smith

That was, I guess, 4 years ago or something like that. Then we decided to quit our jobs and start this whole Snipd thing.

swyx

How big is the team now?

Kevin Ben-Smith

We’re just 4 people. We’re all technical: 2 on the backend side, with the AI and all of the other backend things, and 2 on the frontend side, building the app.

swyx

Which is mostly Android and iOS.

Kevin Ben-Smith

Yeah, it’s iOS and Android. We also have a watch app for Apple, but it’s mostly iOS.

swyx

The watch thing is very funny because in the Latent Space community, most of us have been slowly adopting Snipd. You came to me about a year ago and introduced Snipd to me. I was like, “I don’t know. I’m very sticky to Overcast.” Then, slowly, we switched. Why watch?

Kevin Ben-Smith

It goes back to the fact that a lot of our users do something else while listening to a podcast, right? Giving them the ability to capture this knowledge even though they’re doing something else at the same time is one of the killer features.

Maybe at some point I should give a bit more of an overview of all the features that we have.

swyx

Sure.

Kevin Ben-Smith

This is one of the killer features, and one big use case that people use it for is running. If you’re a big runner, a big jogger, or you cycle really, really competitively, a lot of people don’t want to take their phone with them when they go running.

You load everything onto the watch, so you can download episodes. If you have an Apple Watch with internet access and a SIM card, you can also stream directly.

swyx

That's also possible.

Kevin Ben-Smith

Of course, it's basically very limited to just listening and snipping, and then you can see all of your snips later on your phone.

swyx

Let me tell you about this error I just got: “Error playing episode. Substack, the host of this podcast, does not allow this podcast to be played on an Apple Watch.”

Kevin Ben-Smith

Yeah, that's a very beautiful thing. We found out that all of the podcasts hosted on Substack cannot be played on an Apple Watch.

swyx

What is this restriction? What?

Kevin Ben-Smith

Don't ask me. We tried to reach out to Substack. We tried to reach out to some of the bigger podcasters who host their podcasts on Substack to let them know.

swyx

Uh-huh.

Kevin Ben-Smith

Substack doesn't seem to care. This is not specific to our app. You can also check out the Apple Podcasts app.

swyx

Yeah.

Kevin Ben-Smith

It's the same problem. It's just that we actually identified it, and we tell the user what's going on.

swyx

I will say, we host our podcast on Substack, but they're not very serious about their podcasting tools. I've told them before; I've been very upfront with them, so I don't feel like I'm on them in any way. It's kind of sad because otherwise it's a perfect creator platform, but the way that they treat podcasting as an afterthought, I think it's really disappointing.

Kevin Ben-Smith

Maybe, given that you mentioned all these features, I can give a bit of a better overview of what we have.

swyx

Okay, I'll tell you my version. You can correct me, right?

First of all, I think the main job is for it to be a podcast-listening app. It should basically be a complete superset of what you normally get on Overcast or Apple Podcasts or anything like that. You pull your show list from Listen Notes. How do you find shows? I type in anything, and you find them, right?

Kevin Ben-Smith

Yeah, we have a search engine powered by Listen Notes, but in the meantime, we have a huge database of, like, 99% of all podcasts out there ourselves.

swyx

Huge database.

Kevin Ben-Smith

Like, 99% of all podcasts out there.

swyx

What I noticed is that the default experience is that you do not automatically download shows. That's one very big difference for you guys versus other apps, where, if I'm subscribed to something, it automatically downloads, and I already have the MP3 downloaded overnight.

For me, I have to actively put it onto my queue, and then it automatically downloads. Initially, I didn't like that. I think I maybe told you that. I was like, “Oh, this is a feature that I don't like,” because it means that I have to choose to listen to it in order to download it. Is this opt-in? Is this between opt-in and opt-out?

So, I opt in to every episode that I listen to. Then you open it, and it depends on whether or not you have the AI stuff enabled, but the default experience is no AI stuff enabled. You can listen to it, and you can see the snips—the number of snips and where people snip during the episode—which roughly correlates to interest level. Obviously, you can snip there.

I think that's the default experience. I think snipping's really cool. I use it to share a lot on our Discord. We have tons and tons of people sharing snips and stuff, and tweeting stuff is also a nice, pleasant experience. But the real features come when you actually turn on the AI stuff.

Kevin Ben-Smith

I think that was a good basic overview. Maybe I can add a bit to it with the AI features that we have.

One thing that we do every time a new podcast episode comes out is transcribe the episode, do speaker diarization, identify the speaker names for each guest, extract a mini bio of the guest, and try to find a picture of the guest online and add it. We break the podcast down into chapters—AI-generated chapters—with a title and a quick description for each chapter.

swyx

That one's very handy.

Kevin Ben-Smith

We identify all the books that get mentioned on a podcast.

swyx

I don't use that one.

Kevin Ben-Smith

It depends on the podcast. There are some podcasts where the guests often recommend an amazing book. You can also find that again later on.

swyx

So, literally, you search for the word “book” or, like, “I just read blah blah blah”?

Kevin Ben-Smith

No, it's all LLM-based. We have an LLM that goes through the entire transcript and identifies whether a user mentions a book. Then we use the Perplexity API, together with various other LLM orchestration, to go out there on the internet, find everything there is to know about the book, find the cover, find out who the author is, and get a quick description of it.

For the author, we then check which other episodes the author appeared on.

swyx

Yeah, that is killer. For me, if there's an interesting book, the first thing I do is listen to a podcast episode with the writer because they usually give a really great overview already on a podcast. Sometimes the podcast is with the person as a guest. Sometimes the podcast is about the person without them there. Do you pick up both?

Kevin Ben-Smith

Yes, we pick up both in our latest models, but what we currently show you in the app—the goal is to only show you the guest, to separate that. In the future, we want to show the other things more, but that's—

swyx

For what it's worth, I don't mind. If I like somebody, I'll just learn about them regardless of whether they're there or not.

Kevin Ben-Smith

Yeah, I mean, yes and no. We've seen that there are some personalities where this can break down. The best examples for me are Sam Altman and Elon Musk. They're just mentioned on every second podcast, and it picks them up even though they're not on there.

swyx

I see.

Kevin Ben-Smith

We updated our algorithms and improved that a lot, and now it's gotten much better at only picking someone up if they're a guest.

To come back to the features, there are 2 more important features. We have the ability to chat with an episode.

swyx

Yes. Of course.

Kevin Ben-Smith

You can do the old-style searching through a transcript with keyword search, but I think for me, this is how you used to do search and extract knowledge in the past.

swyx

Old school.

Kevin Ben-Smith

The AI way is basically an LLM. You can ask the LLM, “Hey, when do they talk about topic X?” If you're interested in only a certain part of the episode, you can ask it to give you a quick overview of the episode or the key takeaways. Afterwards, you can also ask it to create a note for you. This is really very open-ended.

Finally, there's the snipping feature that we mentioned. Whenever you hear an amazing idea, you can triple-tap your headphones or click a button in the app, and the AI summarizes the insight you just heard and saves that together with the original transcript and audio in your knowledge library.

swyx

I also noticed that you skipped dynamic content.

Kevin Ben-Smith

We don't skip it automatically.

swyx

Oh, sorry, you detect—

Kevin Ben-Smith

But we detect it, yeah. That's one of the things that most people don't actually know. The way ads get inserted into most podcasts is that every time you listen to a podcast, you actually get access to a different audio file. On the server, a different ad is inserted into the MP3 file automatically.

swyx

Yeah, based on IP.

Kevin Ben-Smith

Exactly. What that means is that if we transcribe an episode and have a transcript with timestamps—word-specific timestamps—and you suddenly get a different audio file, all the timestamps are messed up. That's a huge issue, and for that we actually had to build another algorithm that dynamically, on the fly, resyncs the audio that you're listening to with the transcript that we have.

swyx

That's a fascinating problem in and of itself. Do you sync by matching up the sound waves, or do you sync by matching up words? Basically, do you do partial transcription?

Kevin Ben-Smith

We're not matching up words. It's happening basically at a byte level.

swyx

Matching?

Kevin Ben-Smith

Yeah, byte-level matching.

swyx

Okay.

Kevin Ben-Smith

It relies on there being exact matches at some point. Actually, we're not doing exact matches; we're doing fuzzy matches to identify the moment. We basically built Shazam for podcasts, just as a little side project to solve this issue.

swyx

Yeah, yeah. Actually, fun fact: apparently the Shazam algorithm is open. They published a paper and talked about it.

Kevin Ben-Smith

Yeah, I haven't really dived into the paper. I thought it was kind of interesting that basically no one else has built Shazam.

swyx

Yeah, I mean, the one thing is the algorithm. If you now talk about Shazam, the other thing is also having the database behind it and having the user mindset that if they have this problem, they come to you, right?

I'm very interested in the tech stack. There's a big data pipeline. If you share what the tech stack is, what are the most interesting or challenging pieces of it?

Kevin Ben-Smith

The general tech stack is that our entire backend—or 90% of our backend—is written in Python. We're hosting everything on Google Cloud Platform. Our front end is written with—well, we're using the Flutter framework.

swyx

Ah.

Kevin Ben-Smith

So, it's written in Dart and then compiled natively. We have 1 codebase that handles both Android and iOS.

swyx

You think that was a good decision?

Kevin Ben-Smith

It's something that a lot of people are exploring. So up until now, yes.

swyx

Okay.

Kevin Ben-Smith

Look, it has its pros and cons. Earlier, I mentioned that we have an Apple Watch app.

Yeah.

I mean, there's no Flutter for that, right? So you build native, and then, of course, you have to sync these things together. I'm not the front-end engineer, so I'm just relaying this information, but our front-end engineers are very happy with it. It's enabled us to be quite fast and be on both platforms from the very beginning.

When I talk with people and they hear that we are using Flutter, they usually think, "It's not performant. It's super janky," and everything. Then they use our app, and they're always super surprised. Or if they've already used our app and I show it to them, they're like, "What?"

swyx

Yeah. So there is actually a lot that you can do. There are a few concerns, right? One, it's Google, so when are they going to abandon it? Two, they're optimized for Android first, so iOS is a second thought. You can feel that it is not a native iOS app. But you guys put a lot of care into it.

Maybe three, from my point of view, as a JavaScript guy, React Native was supposed to be that dream, and I think that it hasn't really fulfilled that dream. Maybe Expo is trying to do that, but, again, it does not feel as productive as Flutter. I spent a week on Flutter and Dart, and I'm an investor in FlutterFlow, which is the low-code Flutter startup that's doing very, very well. I think a lot of people are still Flutter skeptics.

Kevin Ben-Smith

Yeah.

swyx

Wait, so are you moving away from Flutter?

Kevin Ben-Smith

No, we don't have plans to do that.

swyx

You're just saying about the watch-out. Okay, let's go back to the stack. That was just to give people a bit of an overview. I think the more interesting things are, of course, on the AI side.

Kevin Ben-Smith

As I mentioned earlier, when we started out, it was before ChatGPT, before the ChatGPT moment, before there was the GPT-3.5 Turbo API. So in the beginning, we were actually running everything ourselves: open-source models, trying to fine-tune them.

swyx

What did you use before Whisper for transcription?

Kevin Ben-Smith

Yeah, we were using wav2vec 2.0.

swyx

I see. It was the Google one, right?

Kevin Ben-Smith

No, it was the Facebook one. That was actually one of the papers that, when it came out, was one of the reasons why I said we should try to start a startup in the audio space. Before that, I had been following the NLP space quite closely. As I mentioned earlier, we did some stuff at the startup I was working at before. Wav2vec 2.0 was the first paper that I had at least seen where the whole transformer architecture moved over to audio.

swyx

Yeah.

Kevin Ben-Smith

A bit more generally, it was the first time that I saw the transformer architecture being applied to continuous data instead of discrete tokens. It worked amazingly. The transformer architecture plus self-supervised learning—these 2 things moved over. For me, it was like, "Hey, this is now going to take off similarly to how the text space has taken off."

With these 2 things in place, even if some features that we want to build are not possible yet, they will be possible in the near term with this trajectory. So that's a little side note.

In the meantime, we're using Whisper. We're still hosting some of the models ourselves. For example, the whole transcription and speaker-diarization pipeline needs to be as cheap as possible. We're doing this at scale, where we have a lot of audio.

swyx

What numbers can you disclose? Just to give people an idea, because it's a lot.

Kevin Ben-Smith

We have more than 1 million podcasts that we've already processed.

swyx

When you say 1 million, processing is basically that you have some kind of list of podcasts that you auto-process, and others where a paying member can choose to press the button and then transcribe it, right? Is that the rough idea?

Kevin Ben-Smith

Yeah, exactly. If you press that button, or we auto-transcribe it, first we do the transcription and the speaker diarization. Basically, you identify speech blocks that belong to the same speaker. This is then all orchestrated with an LLM to identify which speech block belongs to which speaker.

Together with that, as I mentioned earlier, we identify the guest name and the bio. So all of that comes together with an LLM to assign speaker names to each block.

Most of the rest of the pipeline we've now migrated to LLM APIs. We mainly use OpenAI and Google models—the Gemini models and the OpenAI models—and we use some Perplexity, basically, for those things where we need web search.

swyx

That's something I'm still hoping for, especially from OpenAI: that they will also provide us an API. No way. Basically, for us as a consumer, the more providers there are, the more competition there is, and that will lead to better results and lower costs over time.

Kevin Ben-Smith

I don't see Perplexity as expensive. If you use the web search, the price is like $5 per 1,000 queries, which is affordable. But if you compare that to just a normal LLM call, it's much more expensive.

swyx

Okay. Have you tried Exa?

Kevin Ben-Smith

We've looked into it, but we haven't really tried it.

swyx

We started with Perplexity, and it works well. If I remember correctly, Exa is also a bit more expensive. I don't know. They seem focused on search as a search API, whereas Perplexity is maybe more of a consumer business with higher margins. I'll put it like this: Perplexity is trying to be a product; Exa is trying to be infrastructure. That's my distinction there.

The other thing I will mention is that Google has a search grounding feature.

Kevin Ben-Smith

We've also tried that. We didn't go into too much detail in really comparing it quality-wise, because we already had the Perplexity one, and it's working. I think the price there is actually higher than Perplexity.

swyx

Really? Google should cut their prices.

Kevin Ben-Smith

Maybe it was the same price. I don't want to say something incorrect, but it wasn't cheaper. It wasn't compelling. Then there was no reason to switch.

In general, for us, given that we work with a lot of content, price is actually something that we do look at. For us, it's not just about taking the best model for every task, but really identifying what kind of intelligence level you need and then getting the best price for that, to be able to really scale this and let our users use these features with as many podcasts as possible.

swyx

Yeah. I wanted to double-click on diarization. It's something that I don't think people do very well. I'm a Bee user. I don't have it right now, but they were supposed to speak, and they dropped out at the last minute. We've had the Bee AI guys on the podcast before, and it's not great yet. Do you use just pyannote, the default stuff, or do you find any tricks for diarization?

Kevin Ben-Smith

We do use the open-source packages, but we've tweaked them a bit here and there. For example, if you mention the Bee AI guys, I actually listened to the podcast episode. It was super nice, thank you. When you started talking about speaker diarization, I just had to think about their use case. With all of the different environments, it can basically be anything. It's completely out of domain; there's no data for this.

I was feeling for them, because our advantage is that we're working with very high-quality audio. It's very controlled, usually recorded in a studio. This is quite an exception, I guess.

swyx

It is kind of a studio. It's pretty quiet. There's consistent background noise, which you can edit out.

Kevin Ben-Smith

Yeah. There's New York.

swyx

It's nice. It's a character.

Kevin Ben-Smith

That, of course, helps us. Another thing that helps us is that we know certain structural aspects of the podcast. For example, how often does someone speak? If there's a 1-hour episode and someone speaks for 30 seconds, that person is most probably not the guest and not the host. It's probably some ad, like some speaker from an ad.

swyx

Okay.

Kevin Ben-Smith

So we have certain heuristics that we can use and leverage to improve things. In the past, we've also changed the clustering algorithm. Basically, how a lot of this speaker diarization works is you create an embedding for the speech that's happening, and then you try to somehow cluster these embeddings and find out: this is all one speaker; this is all another speaker.

There, we've also tweaked a couple of things where we again used heuristics that we could apply from knowing how podcasts function. That's also actually where I was feeling so much for the Bee AI guys, because all of these heuristics are probably almost impossible for them to use. It can just be any situation, anything.

Another thing is that we actually combine it with LLMs: the transcript, LLMs, and the speaker diarization. We bring all of these together to recalibrate some of the switching points—when does the speaker stop, and when does the next one start?

But the LLMs can add errors as well. I wouldn't feel safe using them to be so precise. At the end of the day, just to avoid giving the wrong impression, the speaker diarization we're doing isn't perfect either.

swyx

I basically don't really notice it. I use it for search.

Kevin Ben-Smith

Yeah, it's not perfect yet, but it's gotten quite good. Especially if you take a latest episode and compare it to an episode that came out a year ago, we've improved it quite a bit.

swyx

Well, it's beautifully presented. I love that I can click on the transcript and it goes to the timestamp. It's so simple, but it should exist.

Kevin Ben-Smith

Yeah, I agree. I agree.

swyx

I'm loading a 2-hour episode of The TechMeme Ride Home, where there are a lot of different guests calling in, and you've identified the guest names.

Kevin Ben-Smith

Indeed. These are all LLM-based.

swyx

Yeah, it's really nice. The speaker names—I would say I'm a power user of all these tools—you've done a better job than Descript.

Kevin Ben-Smith

Okay, well—

swyx

Descript has so much funding. They had OpenAI invested in them, and they still suck. So, keep going. You're doing great.

Kevin Ben-Smith

Thanks, thanks. I would say that, especially for anyone listening who's interested in building a consumer app with AI, if your background is in AI and you love working with AI and doing all of that, the most important thing is to keep reminding yourself of what the job to be done actually is here. What does the consumer actually want?

For example, we're delighted by the ability to click on this word and have it jump there. This is not rocket science. You don't have to be Andrej Karpathy to come up with that and build it. I think that's something that's super important to keep in mind.

swyx

Yeah, amazing. There are so many features; it's so packed. There are quotes that you pick up, summarization—and, by the way, I'm going to use this as my official feature request. I want to customize how it's summarized. I want custom prompts, because your summarization is good, but I have different preferences.

Kevin Ben-Smith

Yeah, I completely get your feature request, and I think it just shows that people have asked for it. Maybe, in general, as a way of thinking about the future, I think everything will be personalized. This isn't specific to us.

Today, we're still in a phase where the cost of LLMs—at least if you're working with long context windows like we are—has to be taken into consideration. There are a lot of tokens in an entire podcast, so if we regenerated everything for every single user, it would get expensive. In the future, the cost will continue to go down, and then it will just be personalized.

That being said, you can already do this today. If you go to the player screen and open up the chat, you can ask for a summary in your style.

swyx

Yeah, okay. I mean, I listen to consume, you know. I've never really used this feature. I think that's me being a slow adopter.

Kevin Ben-Smith

No, no—I mean, when does the conversation start?

swyx

Okay. I mean, you can just type anything.

Kevin Ben-Smith

I think what you're describing is maybe an interesting topic to talk about. I told you, “Look, we have this chat; you can just ask for it.” This is how ChatGPT works today, but if you're building a consumer app, you have to move beyond the chat box. People don't always want to type out what they want.

Your feature request, even though it's theoretically already possible, is actually saying, “I just want to open up the app, and it should be there in a nicely formatted, beautiful way, so I can read or consume it without any issues.” I think that's generally where a lot of the opportunities lie in the market right now if you want to build a consumer app: taking the capability and intelligence, but figuring out the best user interface—the best way for a user to engage with that intelligence naturally.

swyx

This is something I've been thinking about as AI that's not in your face. Right now, we like to say that Notion has Notion AI, and there's a little thing there, or some other platform has the sparkle or magic-wand emoji: “That's our AI feature. Use this.” A lot of people don't like it. It should just become invisible, kind of like invisible AI.

Kevin Ben-Smith

100%. The way I see it is that AI is the electricity of the future. We don't talk about how this microphone uses electricity or how this phone uses electricity. You don't think about it that way; it's just in there. It's not an electricity-enabled product. It's just a product.

It will be the same with AI. Right now, it's still something you use to market your product. We do the same thing because it's still something people recognize as new. But at some point, it will just be a podcast app, and it will be normal that it has AI in it.

swyx

I noticed you do something interesting in your chat where you source the timestamps. Is that part of the prompt, or is there a separate pipeline that adds the sources?

Kevin Ben-Smith

This is actually part of the prompt. It's all prompt engineering: figuring out how to provide the context—we provide the entire transcript—and then getting the model to respond correctly in a certain format, and rendering that on the front end.

swyx

This is one of those examples where it's so easy to create a quick demo. You can just go to ChatGPT, paste this thing in, say, “Do this,” and 15 minutes later you're done. But getting it to production level, so that it actually works 99% of the time, is where the difference lies.

Kevin Ben-Smith

For this specific feature, we also have countless regular expressions. They're there to correct certain things the LLM does because it doesn't always adhere to the format correctly. Then it looks super ugly on the front end.

swyx

Why don't you use an LLM for that? That's sort of the AI-native way. Who uses regular expressions anymore?

Kevin Ben-Smith

With the chat, for user experience, it's very important to have streaming. Otherwise, you have to wait so long until your message arrives. We're streaming the text live, just like ChatGPT.

If you're streaming the text and something is incorrect, it's currently not easy to pipe that stream into another stream and get the corrected stream back.

swyx

Yeah, yeah, yeah. Stream it into another stream, get the corrected stream back—that would be amazing. I don't know; maybe you can answer that. Do you know of any way to do it?

Kevin Ben-Smith

There's no API that does this. You can't stream it in.

swyx

If you own the models, you can take whatever token sequence has been emitted and start loading that into the next one, if you fully own the models.

Kevin Ben-Smith

I don't know. It's probably not worth it. What do you think is better?

swyx

I think most engineers who are new to AI research and benchmarking don't know how much regular-expression work goes into normal benchmarks. It's just this ugly list of 100 different matches for whatever criteria you're looking for.

Kevin Ben-Smith

Yeah. No, it's very cool. I think it's an example of real-world engineering.

swyx

Do you have tooling that you're proud of that you developed for yourself? Is it just a test script?

Kevin Ben-Smith

I think it's a bit more. Vibe evals was a term that came up in one of the talks—I think it might have been the first day of the conference. A lot of the talks were about evals, which are so important. For us, it's a bit more like vibe evals.

That's also part of being a startup: we can take risks. We can accept the cost of something sometimes failing a little bit or being a little off, and our users know that. They appreciate that, in return, we're moving fast, iterating, and building amazing things.

With Spotify, or something like that, half of our features would probably be in a 6-month review through legal—or whatever—before they could ship.

swyx

Let's just say Spotify is not very good at podcasting. I have a documented dislike of its podcast features. Overall, they're not very well integrated.

Any other LLM-focused engineering challenges or problems that you want to highlight?

Kevin Ben-Smith

I think it's not unique to us, but it goes again in the direction of handling the uncertainty of LLMs. At the end of last year, we did a sort of Snipd Wrapped, and one of the things we thought would be fun was to do something with an LLM and the snips that a user has.

Three, let's say, unique LLM features were that we assigned a personality to you based on the snips that you had. It was all just a bit of a fun, playful way—

swyx

I’m going to look at mine. I forgot mine already.

Kevin Ben-Smith

I don’t know whether it’s still in the Discord. We all took screenshots of it.

swyx

Ah, okay.

Kevin Ben-Smith

It’s in the Discord. The second one was a learning scorecard, where we identified the topics that you snipped on the most, and you got a little score for that. The third one was a quote that stood out. The quote is actually a very good example: we would run that for a user, and most of the time it was an interesting quote, but every now and then it was a super-boring quote. You’d think, “Why did you select that? Come on.”

For that, the solution was to say, “Give me 5 candidates.” It accepted 5 quotes as candidates, and then we piped them into a different model as a judge—an LLM as a judge. We used a much better model.

swyx

Okay.

Kevin Ben-Smith

With the initial model, as I mentioned earlier, we do have to look at the cost because we have so much text going into it. We use a slightly cheaper model there, but the judge can be a really good model that chooses 1 out of 5.

swyx

This is a practical example. I can’t find it. Bad search in Discord. So, do you recommend having a much smarter model as a judge?

Kevin Ben-Smith

Yeah.

swyx

And that works for you?

Kevin Ben-Smith

Yeah.

swyx

Interesting. I think this year I’m very interested in LLM-as-a-judge being developed more as a concept. For things like Snipd Wrapped, it’s fine. It’s entertaining, and there’s no right answer.

Kevin Ben-Smith

We also use the same concept for our books feature, where we identify the books that were mentioned. Ninety percent of the time, it works perfectly out of the box in 1 shot, but every now and then it starts identifying books that weren’t really mentioned, books that aren’t books, or it starts making up books. We basically have another LLM challenge it. We do the same thing with the speakers, now that I think about it. I think it’s a great technique.

swyx

Interesting. You run up a lot of costs. You mentioned costs: you moved from self-hosting a lot of models to the big lab models—OpenAI and Google. Anthropic?

Kevin Ben-Smith

No, we love Claude. In my opinion, Claude is the best when it comes to the way it formulates things.

swyx

The personality.

Kevin Ben-Smith

The personality. I actually really love it, but the cost is still high.

swyx

You tried Haiku, but you have to have Sonnet?

Kevin Ben-Smith

With Haiku, we haven’t experimented too much. We obviously work a lot with 3.5 Sonnet. For coding, in Cursor, and in general for brainstorming, we use it a lot. I think it’s a great brainstorming partner. But with a lot of things that we’ve done, we opted for different models.

swyx

What I’m trying to drive at is: how much cheaper can you get if you go from closed models to open models? Maybe it’s 0% cheaper. Maybe it’s 5% cheaper. Or maybe it’s 50% cheaper. Do you have a sense?

Kevin Ben-Smith

It’s very difficult to judge that. I don’t really have a sense, but I can give you a couple of thoughts that have gone through our minds over time. We realize that, given that we have a couple of tasks where so many tokens are going in, at some point it will make sense to offload some of that to an open-source model.

But going back to the fact that we’re a startup, we’re not an AI lab or whatever, the most important thing for us is to iterate fast. We need to learn from our users, improve the product, and maintain the velocity of those iterations. For that, the closed models hosted by OpenAI, Google, and Anthropic are just unbeatable because it’s simply an API call. You don’t need to worry about so much complexity behind it. That’s the biggest reason why we’re not doing more in this space.

There are other considerations for the future. We have 2 different usage patterns for LLMs. One is the preprocessing of a podcast episode: the initial processing, including the transcription, speaker diarization, and chapterization. We do that once, and the usage pattern is quite predictable because we know how many podcasts get released and when. We can have a certain capacity, and we’re running that 24/7. It’s 1 big queue running 24/7.

swyx

What’s the queue job runner? Is it Django, just the Python one?

Kevin Ben-Smith

No, that’s just our own. We have it in our database, and the backend talks to the database, picking up jobs and writing them back.

swyx

I’m just curious about orchestration and queues.

Kevin Ben-Smith

We also have a lot of other orchestration where we use Google Pub/Sub.

swyx

Okay.

Kevin Ben-Smith

The other usage pattern is when, for example, a user action triggers an LLM call. It has to be real-time, and there can be moments when usage spikes, followed by moments when there’s very little usage. For that, LLM API calls are perfect because you don’t need to worry about scaling up, scaling down, or handling those issues.

swyx

Serverless versus serverful.

Kevin Ben-Smith

Yeah, exactly.

swyx

I see OpenAI and all of these other providers as the—well, I guess—the Amazon, sorry, AWS, of AI. It’s similar to how, before AWS, you would have to have your own servers, buy new servers, or get rid of servers. With AWS, it became much easier to ramp things up and down.

Kevin Ben-Smith

Yeah, and this is taking it even to the next level for AI.

swyx

I’m a big believer in this. Basically, it’s intelligence on demand. We’re probably not using it enough in our daily lives to do things. We should be able to spin up 100 things at once, go through them, and then stop. I feel like we’re still trying to figure out how to use LLMs in our lives effectively.

Kevin Ben-Smith

Yeah, 100%. I think that goes back to the whole opportunity for a startup. It’s not about letting the big labs handle the challenge of more intelligence. It’s about the existing intelligence: how do you integrate it? How do you actually incorporate it into your life?

swyx

It’s AI engineering. Okay, cool, cool, cool. The 1 other thing I wanted to touch on was multimodality in frontier models. Dwarkesh had an interesting application of Gemini recently, where he fed raw audio in and got diarized transcription out—or timestamps out. I think that will come.

Basically, what we’re saying here is another wave of transformers eating things. Right now, models are pretty much single-modality things. You have Whisper, you have a pipeline, and everything.

Kevin Ben-Smith

No, no, no. We only feed the raw files.

swyx

Do you think that would be realistic for you?

Kevin Ben-Smith

I 100% agree. Basically, everything that we talked about earlier—the speaker diarization, the heuristics, and everything else—in the future would just be put into 1 big multimodal LLM, and it would output everything that you want.

I’ve also experimented with that, just with Gemini 1.5 Flash, for fun. The big difference right now is still the cost difference: doing speaker diarization this way or doing transcription this way is much more expensive than the pipeline we’ve built.

swyx

I need to figure out what that cost is because, in my mind, Gemini 2.0 Flash is so cheap. Maybe it’s not cheap enough for you.

Kevin Ben-Smith

No, I mean, if you compare it to Whisper and speaker diarization, especially self-hosting it—

swyx

Yeah, yeah. Okay.

Kevin Ben-Smith

But we will get there, right? This is just a question of time. As soon as that happens, we’ll be the 1st ones to switch.

swyx

Awesome. Anything else that you’re eyeing on the horizon as you think about a feature or incorporating some new AI functionality into the app?

Kevin Ben-Smith

There are so many areas that we’re thinking about. Our challenge is more about choosing.

swyx

Choosing.

Kevin Ben-Smith

Looking at the next couple of years, there are 4 big areas that interest us. One is content. Right now, it’s podcasts. You mentioned that you can also upload audiobooks and YouTube videos.

swyx

YouTube. I actually use the YouTube one a fair amount.

Kevin Ben-Smith

In the future, we want to have audiobooks natively in the app, and we want to enable AI-generated content. Think of Deep Research and NotebookLM podcast generation—put those together. That should be in our app.

The second area is discovery. In general, discovery as a paradigm in all apps will undergo a change thanks to AI.

swyx

I noticed that you don’t have—so, you have download counts and most snips, right? Something like that?

Kevin Ben-Smith

Yeah. On the discovery side, we want to do much more. Before Elon bought Twitter, there was a lot of talk about bringing your own algorithm to Twitter. That was Jack Dorsey’s big thing; he talked about it a lot. I actually think this is coming, but with a bit of a twist.

I think what AI will enable is not that you bring your own algorithm, but that you will be able to talk and communicate with the algorithm. You can just tell the algorithm, “Hey, you keep showing me cat videos. I know I freaking love them, and that’s why you keep showing them to me. But for the next 2 hours, I really want to get more into AI stuff. Do not show me cat videos.” Then it will just adapt.

Of course, the question is that big platforms like TikTok do not have the incentive to offer that.

swyx

Exactly. That’s what I was going to say.

Kevin Ben-Smith

But we actually are driven by helping you learn, get the most out of things, and achieve your goals. For us, it’s very much in our incentive to say, “Hey, you should be able to guide it.”

That was a long way of saying that I think this will happen a lot in recommendation. I think collaborative filtering will be the first step for RecSys, and then some other fancy stuff.

Maybe to go back to the question that you had before, these were the first 2 areas. The other 2 are voice as an interface and voice AI.

swyx

How does this exist?

Kevin Ben-Smith

Maybe I can first tell you a bit about why I find it so interesting for us. Historically, there has been so much talk about voice as an interface, and it always fell flat. The reason why I’m excited about it this time around is that, with any consumer app, I like to ask myself: What is the moment in my life, or the trigger in my life, that gets me to open this app and start using it?

For example, take Airbnb. The trigger is, “Ah, you want to travel,” and then you open up the app. For apps that do not already have this natural trigger in your life, it’s very difficult for a consumer app to get a user to use it.

swyx

You need a hook.

Kevin Ben-Smith

There’s basically only 1 super-successful app that has been able to do that without this natural trigger, and that is Duolingo.

swyx

Ah.

Kevin Ben-Smith

Everyone wants to learn a language, but you don’t have this natural moment during your day when you think, “Ah, now I need to open up this app.”

swyx

The notifications.

Kevin Ben-Smith

Exactly. The owl memes.

swyx

Exactly.

Kevin Ben-Smith

They gamified it super successfully and made it super beautiful. They are the GOATs in this game. But what’s much easier is when there is already this trigger, and then you don’t have to do all of the streaks and leaderboards.

Now, if you look at what we are doing and our goal of getting people to really maximize what they get out of their listening, we’re interested in a couple of features where we know we can 10x the value that people get out of a podcast. But we need them to do something for that. There is friction involved because it’s all about learning, right? It’s about thinking for yourself.

swyx

Apply the knowledge.

Kevin Ben-Smith

Exactly. You have to be forced to think about what was actually the main takeaway for you from this episode. There’s something that I like doing myself for every episode that I listen to: I try to boil it down to 1 single takeaway. Even though there might have been 10 amazing things, you pick 1—the most important one.

This is an active process, a forcing function in your brain to challenge all of the insights and really come up with the 1 thing that is applicable to you and your life, and what you might want to do with it. It also helps you turn it into action.

This is basically a feature that we’re interested in, but you have to get the user to use it. If this is all text-based, then we’re basically playing the same game as Duolingo, where at some point you’re going to get a notification from Snipd saying, “Hey, swyx, come on. You know you should do this.” Maybe there’s a blue owl.

But if you have voice, you can hook into the existing habits that the user already has. You already have this habit of listening to a podcast. You’re already doing that. Once an episode ends, instead of just jumping into the next episode, you can now have your AI companion come on and have a quick conversation. You can go through these things.

How that looks in detail, we still need to figure out. But this paradigm of staying in the flow relates a bit to what you were saying about AI that is invisible. You’re staying in the flow of what you’re already doing, but now we can insert a completely new experience that helps you get the most out of your listening.

swyx

I think your framing of this is very powerful. I think this is where you are a products person more than an engineer, because an engineer would just be like, “Oh, it’s just chat with your podcast. It’s chat with a PDF, chat with a podcast. Okay, cool.”

But you’re framing it in a different light that actually makes sense to me now, as opposed to previously. I don’t chat with my podcast. Why? I just listen to the podcast, right? But for you, it’s more about retention and learning and all that. You’re very serious about it. That’s why you started the company, and you’re focused on that.

I would admit that I’m still stuck in that consume, consume, consume mentality. I know it’s not good, but this is my default. That’s why I was a little bit lost when you were saying all the things about Duolingo and the trigger. My trigger for listening to the podcast is that I’m by myself. That’s my trigger.

But you’re saying the trigger is not about listening to the podcast. The trigger is remembering, retaining, and processing the podcast I just listened to.

Kevin Ben-Smith

No, what I meant is that you already have this trigger that gets you to start listening to a podcast. You already have that, and so do, I don’t know, millions of people. There are more than half a billion monthly active podcast listeners.

But you do not have this trigger, as you just said yourself, that gets you to regularly process this information. Voice, for me, is the ability to hook into your existing trigger. The trigger I was talking about is that your podcast ends, and you’re still listening. We just continue.

This can be 2 or 3 minutes. I’m not saying it has to be a 60-minute process. I think 2 or 3 minutes can happen completely naturally. If we manage to do that and you start noticing as a user, “I’m freaking out—I’m just now spending 3 minutes with this AI companion, but I’m taking this much more away,” then we’ve won.

Retention is 1 thing, but you start to take what you’ve learned and apply it to what’s important to you—your thinking.

swyx

A lot of people rely on Anki notes, flashcards, and all that to do this. But making the notes is also a chore, and I think this could be very interesting.

I’m just noticing that it’s kind of a different usage mode. You already talked about this: the name Snipd is very snip-centric. I originally resisted adopting Snipd because of that. But now you observe that people are listening to long-form episodes and talking at the end.

The ideal implementation of this is that I browse through a bunch of snips from the things I’m subscribed to. I listen to the snips, I talk with it, and then maybe it double-clicks on the podcast and finds other timestamps that are relevant to the thing that I want to talk about. I was just thinking about that. I don’t know if that’s interesting.

What are your thoughts on voice cloning? I’ve had my voice cloned, and people have talked to me through an AI version of me. Is that too creepy?

Kevin Ben-Smith

I don’t think it’s too creepy in the future. With a lot of these things, society is going through a change. Things seem quite weird now, but in the future they’ll seem normal.

I think voice cloning has already become much more normalized. I remember I was at the NeurIPS conference—I think it was in San Diego?

swyx

No, LA.

Kevin Ben-Smith

LA. It was the Florida one.

swyx

Yeah, yeah, yeah, yeah. Florida.

Kevin Ben-Smith

Everyone says that was peak NeurIPS. I remember there was a talk or workshop by Lyrebird. They actually got acquired by Discord later. They were showing off their technology and there was a huge discussion afterward about all of the moral and ethical implications. It really felt like this would never be accepted by society.

You look at it now: you have ElevenLabs, and anyone can just clone their voice. No one really talks about it as if the world is going to end. I think society will get used to it.

In our case, there are some interesting applications where we’d also be super interested in working together with creators, like podcast creators, to play around with this concept.

swyx

I think that would be super cool if someone could come on to Snipd, go to the Latent Space podcast, and start chatting with AI swyx.

Kevin Ben-Smith

Yeah. No, I think we’ll be there. We want to—obviously, I think as an AI podcast, we should be the first consumers of these things.

swyx

Yeah. I would say that one observation I’ve made about podcasting—this is just the general state of the market, and you can ask me your questions, things you want to ask about podcasters—is that we’re focusing a lot more on YouTube this year. YouTube is the best podcasting platform. It is not MP3s. It is not Apple Podcasts. It is not Spotify. It’s YouTube.

It’s just the social layer of recommendations and the existing habit that people have of logging on to YouTube and getting that. That’s my observation. You can riff on that.

Kevin Ben-Smith

The only thing I would say is that when you were listing your priorities, you said audiobooks first over YouTube, and I would suggest, if I were you—

swyx

Yeah, as in YouTube video podcasts.

Kevin Ben-Smith

I mean, it’s obvious that video podcasts are here to stay.

swyx

Not just here to stay—bigger.

Kevin Ben-Smith

What I want to do with Snipd is obviously also add video to the platform.

swyx

Oh, yeah.

Kevin Ben-Smith

The way I see video is, I do believe—I like this concept of backgroundable video. I didn’t come up with this concept. It was actually Gustav Söderström, the CPO of Spotify.

swyx

Exactly. Exactly.

Kevin Ben-Smith

When I speak with people, it remains true that they listen to podcasts when they do something else at the same time. That’s around 90% of their consumption, also if they listen on YouTube.

But every now and then, it’s nice to have the video. It’s nice if you’re, for example, just watching a clip. It’s nice if they sometimes mention something, like they show some slides or something where you need to have the visual with it. It helps you connect much more with the host as a listener.

But the biggest benefit I see with video is discovery. I think that is also why YouTube has become the biggest podcast player out there, because they have the discovery. Discovery in video is just so much easier, so much better, and so much more engaging.

swyx

For consumers?

Kevin Ben-Smith

Yeah, for consumers.

swyx

Okay. I think that you almost have three different audiences. The vast majority of people for you are the people listening to podcasts, right?

Kevin Ben-Smith

Of course.

swyx

Then there’s a second layer of people who create Snips, right? Who add extra data and annotation value to your platform. By the way, we use the Snip count as a proxy for popularity, because we have download counts, but platforms like Spotify rehost our MP3 file, so we don’t get any download count for Spotify.

Snip count is active: I opt in to listen to you, and I shared this. Those are really, really good metrics.

But the third audience that you haven’t really touched is the podcast creators, like myself. For me, discovery from that point of view—not from your point of view—discovery for me is that I want to be discovered, and I think YouTube is still there. Twitter, obviously, for me, Substack, Hacker News. I try really hard to rank on Hacker News.

When TikTok took this very seriously, they prioritized the creators of the content. For you, the creator of the content was the Snips. But there may be a world for you in which you prioritize the creators of the podcasts.

Kevin Ben-Smith

Yeah, interesting observation. What are some of your ideas or thoughts? Do you have some specific examples?

swyx

Riverside is the closest that has come to it. Descript is number 2. Descript bought a Riverside competitor, and as far as I can tell, it hasn’t been very successful.

Descript has a very, very good niche and a very, very good editing angle, and then just hasn’t done anything interesting since then. Although Underlord is good, it’s not great. Your chapterization is better than Descript’s. Again, they should be able to beat you. They’re not.

Riverside is good also—very, very good. We actually recently started a second series of podcasts within Latent Space that is YouTube-only, because you only find it on YouTube. It’s also shorter. This is a 1.5- to 2-hour thing; the other one is remote-only, 30 minutes, chop-chop. Send it all into Riverside.

Riverside is pretty good for that. Not great. It doesn’t do good thumbnails. It doesn’t do the editing very well; it’s still a little bit rough. It has an auto-editor where whoever’s actively speaking is focused on, and then sometimes it goes back to the multi-speaker view. That kind of stuff. People like that.

But the Shorts are still not great. I still need to manually download them and then republish them to YouTube. I still need to pick the Shorts. They mostly suck. There are still a lot of rough edges there.

Ideally, as a creator, you know what I want. You definitely know what I want. I sit down, record, press the button, and I’m done.

swyx

Yeah. I think you guys could do it.

Kevin Ben-Smith

Okay. So if I can translate that, for you it’s really about simplifying the creation process of the podcast.

swyx

Yeah. And I’ll tell you what: this will increase the quality, because the reason that most podcasts or YouTube videos are made by people who don’t have life experience, who are not that important in the world, and who aren’t doing important jobs.

What you actually want to enable is CEOs to each make their own podcasts. They’re busy. They’re not going to sit there and figure out Riverside.

A lot of the reason that people like Latent Space is that it takes an idiot like me, who could be doing a lot more with my life, making a lot more money, and having a real job somewhere else, and I just choose to do this because I like it. Otherwise, they’ll never get access to me or to the people that I have access to.

That’s my pitch.

Kevin Ben-Smith

Cool. Anything else that you normally want to talk to podcasters about?

I think we’ve covered everything.

I guess, as a last message, go try out Snipd. It’s a freemium version, so you can use and try out everything for free. I’m also happy to provide you with a link that you can add to the show notes to try out the premium version for free for a month if people want to do that. Give it a shot.

swyx

I would say, yeah, thanks for coming on. After you demoed it to me, I did not convert for another 4 to 6 months because I found it very challenging to switch over.

I think that’s the main thing. You basically have OPML import, right? But there’s no way to import all the existing half-listened-to episodes or my rankings or whatever.

For listeners who are interested, I have a blog post where I talked about my switch. Just treat it as a chance to clean house.

Kevin Ben-Smith

That is a good point.

swyx

Yeah. Do new things and refocus your start. Restart 2025.

Great. Well, thank you for working on Snipd. Thank you for coming on. We usually spend a lot of time talking to big companies, venture startups, B2B SaaS, and that kind of stuff.

But I think your journey as a small team building a B2C consumer app is the kind of stuff that we also like to feature, because a lot of people want to build what you’re doing. They don’t see role models that are successful, confident, and having success in this market, which is very challenging.

Thanks for sharing some of your thoughts.

Kevin Ben-Smith

Thanks. Thanks for having me, and thank you for creating an amazing podcast and an amazing conference as well.

swyx

Thank you.

Snipd: The AI Podcast App for Learning — with CEO Kevin Ben-Smith | BidClub