FFmpeg：互联网视频背后的惊人技术 | Lex Fridman Podcast #496
FFmpeg 和 VLC 是以极少传统资本建成的文明级基础设施：Fridman 称 FFmpeg 出现在超过 90% 的视频工作流中，VLC 下载量至少达到 65 亿次。 同一套技术栈服务于家庭影片、YouTube、Netflix、浏览器、电视台、广播机构乃至航天器，形成罕见的开源飞轮：每增加一种编解码器，整个平台的价值都会提升。Kunhya 的表述更具战略意味：VLC 和 FFmpeg 是“二元星系统”，不是竞争对手。
视频的核心经济问题，是用算力换带宽，同时保留人眼和人耳真正感知到的内容，而不是像 ZIP 文件那样复原源文件。 音频可以压缩约 10 倍，视频可压缩 100–200 倍；在质量相当的情况下，每一代编解码器大约再提升 30% 的压缩率，但编码算力需求可能增加 1–2 个数量级。这种不对称之所以成立，是因为内容往往只需“编码一次”，却要分发给数百万观众。
在代码运行数十亿次的地方，手写汇编仍是巨大的经营杠杆，尽管软件行业长期相信编译器和自动向量化。 Kunhya 称 dav1d 包含约 3 万行 C 和 24 万行汇编，而 FFmpeg 全部编解码器的汇编代码约 10 万行；他还举例称，单个函数的运行速度达到 C 的 62 倍。据估计，有 30 亿台设备在解码 AV1，其中 Netflix 和 YouTube 视频分别约有 30% 和 50% 如此，“每一个周期都重要”。
这些项目真正持久的护城河，是一份社会契约：开放访问、严格审查、广泛硬件支持，以及拒绝通过背叛用户来变现。 Kempf 曾多次拒绝价值数千万美元的报价，因为其中涉及工具栏、间谍软件、劫持搜索或广告；他的判断标准是，自己能否“晚上睡觉时对这个决定感到高兴”。LGPL 许可仍允许围绕 FFmpeg 和 libVLC 开发商业产品，但对这些组件的修改必须回馈社区。
大型科技公司经常把志愿者项目当供应商使用，却不给供应商级别的经济回报，最终形成不断扩大的基础设施和安全责任。 节目中最尖锐的例子，是针对冷门编解码器的 AI 生成安全报告：漏洞发现可以低成本规模化，但分诊、修复、对外沟通和固定 90 天期限都由志愿者承担。舆论压力带来了补丁、修复奖金和更多捐款，但 Kunhya 称，FFmpeg 的捐款仍不足以支付 1 名全职开发者。
这里的开源既是全球人才市场，也是精英工程学院，唯一的入场券是优秀代码。 有青少年写出数千行生产级汇编，也有人在战区贡献代码；维护者明确拒绝企业履历崇拜：“也许你是一条狗。我不在乎。”标准之所以维持在高位，是因为数千名短期参与者提交的代码，最终可能由仅约 10–15 名 FFmpeg 核心开发者和 6–8 名 VLC 贡献者接手。
信任来自架构和可验证性，而不是口头承诺：VLC 不收集遥测数据，拒绝政府后门，并在离线系统上构建、分层签名发布。 剩余风险集中在分发渠道和依赖代码：假 VLC 安装包曾延迟数周才启动恶意软件，而真实的媒体进程会调用数百个插件、FFmpeg 和 GPU 驱动。VLC 的应对方式是进程级沙箱，让解码器或解复用器即使崩溃，也无法接触用户文档。
下一个可投资的边缘，将从被动播放转向同步、超低延迟的机器控制，以及更丰富的感官媒体。 Kempf 的 Kyber 通过一条 QUIC 连接传输视频、音频、指令和同步传感器数据；Kunhya 报告称，Windows 到 Windows 或 Windows 到 Mac 的端到端玻璃延迟为 7 毫秒，目标是 4 毫秒。除了 AV2，同一架构还可以扩展到点云、XR、触觉、深度、气味，最终甚至神经流：多媒体就是“面向人类感官的多条数据流”。
1. VLC 的荒谬兼容性，始于把每一种输入都视为可疑对象
Kempf 开场展示的并不是时髦的流媒体格式，而是 VHS：通过 SCART 或 RCA 将采集卡接入后，VLC 可以播放或录制信号，部分模块甚至能直接控制 VCR 摄像机。项目还曾用一个夏天实现 DVD-Audio，包括其定制加密，而当时商业软件已经基本放弃支持这种格式。
FFmpeg 的兼容性长尾延伸到一款老《星球大战》游戏的 10 秒开场：有人实现了它的特殊编解码器，并确保一张光盘上的那段序列能够逐比特准确解码。其背后的档案伦理不言自明——只要贡献者愿意投入，数据的冷门并不是放弃恢复它的理由。
VideoLAN 曾举办比赛，故意制造恶意媒体文件：一个 MKV 每帧都改变分辨率、宽高比和旋转角度；另一个“视频”只有黑帧和动态 SSA 字幕；还有一个文件被描述为同时是有效 ZIP 和 MP3。VLC 能播放这些愚蠢文件，是因为病理样本训练的正是处理现实世界损坏媒体所需的韧性。
交通锥之所以成为分发资产，恰恰因为它“荒唐且可笑”。Kempf 称，VideoLAN 网站 25% 的流量来自搜索“cone player”，而一次愚人节替换活动引发约 1 万封抗议信。一个鲜明的网络梗最终变成了文化符号，也为这个小型非营利组织带来了无需购买的全球身份认知。
2. 按下播放键，就会启动一整套以人的一生为尺度的工程问题
Kempf 将第一阶段描述为：把地址——HTTP URL、本地路径、DVD 或设备——交给操作系统，再得到字节流。随后，解复用器识别容器结构，分离压缩视频、音频和字幕轨道；在此之前，系统里还不存在任何像素或声音采样。
Kunhya 称，播放器必须先探测每条视频流，再选择解码器，因为最多 45% 的文件可能无法由 GPU 解码。不同编解码器变体和 GPU 厂商暴露出的能力各不相同；兼容内容会进入硬件“黑盒”，其余则回退到软件解码。
软件解码首先移除霍夫曼编码或算术编码等熵编码，读取语法信息进行帧内预测，从量化后的频域系数中重建预测残差，执行逆变换，再将残差还原到空间域。只有经过这些步骤，压缩语法才会变成原始图像数据。
最后，图像交给图形系统，原始音频交给声卡，后者再生成供扬声器播放的模拟信号。Kunhya 的提醒避免了这张整齐的流程图制造“事情很简单”的错觉：“每一句话背后都是某个人一生的工作。”每一条箭头背后，都有书籍、学术专长和数千个现实边界案例。
3. 压缩的成功，在于以人类最不容易察觉的方式损失信号
Kempf 的区分至关重要：媒体压缩不是 ZIP。无损归档试图复原完全一致的数据，而常见音视频编解码器会主动丢弃信息——类似 MP3 的音频大约压缩 10 倍，视频通常压缩 100 倍或 200 倍——同时让损失尽可能符合人类感知。
因此，视频处理会从 RGB 转向 YUV 一类的表示方式，将亮度和色彩信息分离。由于视觉对亮度细节更敏感，色度分辨率通常可以大幅降低，先将未压缩数据缩小一半，而大多数观众并不会有意识地察觉消失了什么。
编解码器会把图像划分为块，预测各块内容，用类似离散余弦变换的数学方法转换残差误差，再量化系数并紧凑编码。劣质或损坏图像中常见的方块并非随机瑕疵，而是暴露了大量数学机制所依赖的网格。
每一代编解码器可能在质量相当的情况下将码率降低约 30%，但 Kunhya 补充了隐藏成本：实现这项提升可能需要多 1 个、甚至 2 个数量级的编码 CPU。更高效的压缩本质上是基础设施权衡——以更多前置算力换取更低的存储、网络分发、电池消耗和重复解码成本。
4. 编解码器通过跨空间、时间和观众摊薄冗余来变现
Kunhya 将编解码器的任务定义为去除空间和时间冗余，通常压缩所需的算力远高于解压。只要一次昂贵编码可以服务大量观众，这种不对称就具有经济合理性；但它也限制了实时编码、交互系统和小功率设备。
Kempf 描绘的具体画面是镜头扫过云层或静止背景：物体只编码一次，后续帧再告诉系统复用它。向更远处搜索——例如在 4K 视频的 200 帧范围内寻找匹配——可能找到更好的结果，但内存和算力会把理论上无限的复用变成现实中的优化问题。
AV1、AV2 和 VVC 这类现代名称，代表的是一组编码工具，而不是一种统一技术。类似 Zoom 的视频流可能在 PowerPoint 画面和实时视频之间切换，因此需要不同的预测和压缩工具；编码器的负担，是为每个区域、每个时刻选出最优工具组合。
5. 容器负责组织轨道，但文件名经常撒谎
容器也称 muxer，是承载多条轨道的结构；解复用则负责将它们分开。AVI 来自 Microsoft 早期生态，MOV 来自 Apple，后来演化出 MP4；Matroska 则由开源社区设计，目标是成为更具扩展性、面向未来的容器。
MP4 和 H.264 的混淆，部分是行业自己造成的。H.264 也称 AVC 和 MPEG-4 Part 10，而 MPEG-4 本身涵盖容器、音频和多种视频规范；在实际使用中，Kunhya 称 MP4 有 99% 的时候是 H.264 加 AAC，因此日常口语中的混用也可以理解。
VLC 和 FFmpeg 把
.mp4当作提示，而不是事实依据。文件扩展名会提高 MP4 解复用器的探测优先级，但软件仍会检查字节并尝试其他选项，因为文件经常被错误命名、格式损坏，或由不严格遵循规范的工具生成。VLC 不信任输入的理念，继承自 1990 年代末的 UDP 流媒体时代，当时丢包和损坏都是预期情况。同一理念也让它在部分下载场景中广受欢迎：AVI 元数据可能位于文件末尾，但 VLC 仍会用 Kempf 的话说：“这个文件坏了，但我还是要试试。”
6. FFmpeg 把专业媒体处理变成可编程的商品
Kempf 将 FFmpeg 描述为一组用于编解码器、复用器、解复用器和滤镜的底层库，以及把它们组合成流水线的命令行工具。核心库包括 libavcodec、libavformat 和 libavfilter，同时也整合 x264、libvpx 等外部项目和专用组件。
命令行接口实际上“就是一门真正的语言”。简单任务只需一个输入、一个输出和一种格式，但生产环境中的命令可能长达数千字符，也可能由程序生成；如今用户会让 AI 系统生成这些命令，因为可用的滤镜图和参数空间已经超出大多数人的记忆范围。
Fridman 自己展示的例子，将片头、节目和片尾拼接起来；把视频压暗至黑色；对音频进行交叉淡化；将带样式的字幕烧录进画面；同时接受彼此不同的输入格式。Kempf 将其比作从 shell 中完成 Adobe After Effects 工作——对动态图像而言，这是异常完整的自动化接口。
Fridman 认为，FFmpeg 帮助促成了播客、流媒体和 YouTube 革命。1990 年代售价数十万美元、占据汽车大小机架的压缩设备，最终变成个人即可使用的软件，让个人在技术能力上几乎与大型公司站在同一条起跑线上。
7. 开源许可是社区的宪法
Kempf 用芝士蛋糕作比喻，让源代码开放变得具体：开源不仅给出做好的蛋糕，还给出配方、建造烤箱的说明，以及修改和转售配方的法律许可。软件并非几道厨房步骤，而是许多作者积累的数十亿条微小指令，因此许可尤其重要。
他估计，FFmpeg 历史上有 2,000–3,000 人参与贡献。他们跨越国界、宗教、语言和雇主，围绕一个狭窄的共同目标协作——让多媒体正常运行——除了允许他们协作、分叉和重新合并的许可协议外，几乎什么都可以争论。
MIT 和 BSD 等宽松许可允许广泛复用，有时只需署名；copyleft 许可则要求修改内容以逐步增强的条件回馈社区，从类似 MPL 的方式到 LGPL、GPL 和 AGPL 不等。Kempf 称，这种选择是一份界定商业采用者与社区贡献者关系的“社会契约”。
FFmpeg 和 libVLC 的重要库组件使用 LGPL：专有应用可以链接这些组件，而无需开放整个产品，但对组件的修改必须回馈。其他 VLC 版本仍适合使用 GPL；iOS 和 Apple TV 应用则使用 MPL，因为 Apple App Store 的条款会使 GPL 分发变得复杂。
8. 重新许可揭示了代码库背后的人类所有权
将 libVLC 从 GPL 改为 LGPL，需要获得 350 多名版权持有人的同意，因为 VideoLAN 没有要求版权转让。Kempf 追踪旧邮箱地址、走访工作场所，甚至见到一名工厂工人——他已故的儿子曾贡献过几行有用代码。那场对话最终谈的是一个人的一生和一张照片，而不是行政清理。“把事情做对很重要。”
这次变更让 libVLC 得以进入专有游戏和移动应用，也支持围绕集成开发的咨询业务。但它同样说明，FFmpeg 无法简单改变许可来阻止云服务商：数千名贡献者，包括已经去世的人，仍然拥有其作品的权利，而这些作品的影响可能早已超越最初提交的代码。
9. 项目按代码判断身份，因为维护者要继承后果
Kempf 的精英主义规则非常绝对：“也许你是一条狗。我不在乎。”国籍、雇主、年龄和学历都不能证明技术质量；即使补丁来自知名公司的工程师，也可能因为不符合保护社区的代码标准而被拒绝。
严苛门槛反映的是维护算术。Kempf 统计，VLC 约有 5 名核心开发者，FFmpeg 有 10–15 名，而历史贡献者达数千人；如果一个贡献者留下来的概率只有约 1%，核心团队就必须在工作、家庭、意外和兴趣变化让作者离开后继续维护代码。
Linus Torvalds 代表了更极端的表达方式，不过 Kempf 指出，他最强烈的批评通常针对已经认识他的子系统维护者。理想的区分，是攻击一个人和宣布代码“糟透了”；这种文化可能令人难受，但全球基础设施的可维护性和正确性不能妥协。
语言简短也有现实原因：志愿者往往在完成一天的本职工作后审查补丁，且多语种沟通以欧洲为中心。英语中的语气细节并非所有贡献者都掌握，因此一句让美国读者感到敌意的话，可能只是疲惫或直译，而非针对个人的轻蔑。
10. VLC 起源于学生在校园内转发卫星电视
École Centrale Paris 的校园由学生大规模管理，因为校友非营利组织拥有这块场地。学生负责房间、公用设施、商店、广播、电视和网络。这种罕见的责任，让年轻工程师既有权力，也有必要亲自建设基础设施，而不是等待学校安排。
其 1980 年代的令牌环网络，在 Doom 和 Duke Nukem 让延迟变得明显后，终于无法忍受。约在 1994–1995 年，寻求更快网络的学生找到 Bouygues；该公司的 CIO 提议安装一面大型卫星天线和解码器，通过网络向约 1,500 名学生分发信号，而不是给每个人配独立接收器。
“Network 2000”通过 155 Mbps 的 ATM 网络传输 MPEG-2 传输流、视频和音频。被破解的原型机运行 45 秒后崩溃，但演示持续了 40 秒；由于内存泄漏，团队安装了 64 MB 内存，而不是常见的 8 MB 或 16 MB。一场本应一次性使用的展示，最终成为基础。
Christophe Massiot 和另一名学生后来创建 VideoLAN，用于网络视频，并花了约 3 年说服学校将其开源，最终在 2001 年成功。Kempf 于 2003 年加入，成立独立的 VideoLAN 非营利组织，并在 2005 年 VLC 只剩 2 名活跃开发者时将项目维持下来。
11. 拒绝掠夺式变现，保住了 VLC 的信任与生存
Kempf 确认自己不止一次拒绝过数千万美元，但他否认自己反对金钱的神话。这些报价涉及捆绑工具栏、类似间谍软件的浏览器改动、劫持搜索或广告——这些商业模式会伤害用户、引发分叉，并很可能让项目在 3 年后死亡。
他的诚实反事实很有启发性：如果 Netflix 这类合法服务提出有用的集成方案，答案可能不同。界线并不是商业与纯粹之间的二选一，而是以合乎伦理的方式赚钱，还是通过用户没有明确请求的机制利用分发信任。
最难缠的出价者声称，其“下流”的付款可以资助另一个开源项目。Kempf 看出了这种道德漂白：“这不是这么运作的。”出卖项目还会侵占许多其他贡献者共同创造的成果；对他个人而言，关键是晚上睡觉时仍能为自己的选择感到自豪。
12. AI 安全发现，变成了对志愿者注意力的拒绝服务攻击
Kunhya 的批评始于一个让步：Google 是开源最大的支持者之一，也雇用了能力很强的安全研究人员。冲突在于，Google 使用 AI 生成 FFmpeg 漏洞报告，给志愿者标准的 90 天期限，只提供有限资金，却在修复尚未准备好时就公开宣传 AI 的成果。
报告中的目标可能只是一个 1990 年代的冷门游戏编解码器，但安全流程说得仿佛每一把锁都守着 Fort Knox。Kunhya 的类比是：用工业规模的设备撬开业余爱好者家里的挂锁，然后要求这名爱好者紧急重新设计锁具，而发现者既不提供资金，也不参与施工。
自动化报告篇幅很长、数量很多，而且全部令人警觉——“高、高、高、存在漏洞、可怕、可怕、可怕”。另一名研究人员将一个可能让单个像素变色的整数溢出评为 7.5 严重度；Kunhya 认为，这种反复升级会训练维护者和用户忽略警报，成为安全领域的“狼来了”。
Alex Strange 对激励机制的批评之所以有力，是因为发现者可以获得奖金、会议声望、署名、Logo 和宣传，而“你修复它时，没有人会替你做这些事”。争议之后，Google 开始提交补丁，并为修复问题设立奖励，部分纠正了发现与修复之间的不平衡。
13. 企业经常要求服务级别的紧迫性，却不购买服务
Microsoft Teams 的例子发生在 XZ 争议之后：一名经理因为某个 FFmpeg 问题影响到 Microsoft 的显眼产品，便将其标为高优先级。当志愿者建议签订长期维护合同后，Microsoft 只提供了几千美元的一次性付款，尽管其企业规模十分庞大。
Kempf 的边界很简单：“我们不是你的供应商。”公开 Bug Tracker 不是签约供应商的 Jira；除非公司购买，否则就不存在 SLA。那些本应教会企业理解这一点的开源项目办公室，往往没能把产品团队和其技术栈底层的社区联系起来。
舆论压力仍然有效。捐款大幅增加，但 Kunhya 称，依旧不足以支持 1 名 FFmpeg 全职开发者；尖锐发帖也迫使相关方回应：VLC 曾超过 1 年无法在 Google Play 更新，VideoLAN 作为 Windows 前 10 大应用之一，却没有普通的联系渠道。
14. FFmpeg 账号把冲突转化为招募、教育和杠杆
Kunhya 将网络争吵称为“说唱对战”或 WWE：攻击可以带有戏剧性，却不必演变成永久的个人敌意。这个账号的目的不只是嘲讽，而是让人看到 FFmpeg 并不是拥有数百或数千名带薪开发者的 Kubernetes，而主要是一群下班后在地下室工作的人。
最有效的挑衅，是为被行业忽视的贡献者正名。“FFmpeg 里的青少年写过比 Google 工程师更多的汇编代码”，指向 Daniel Kang 和 16 岁的 Ruikai Peng 等人。他们发现并修复问题，没有把一个 3 天就能解决的开发 Bug 变成带品牌的 CVE 宣传秀。
Kempf 承认，账号的语气偶尔越界，包括涉及 Theo 的争议期间；他打电话给 Theo，让参与者冷静下来，并划定界线。但开源社区自建项目的知名度因此大幅上升，Tim Sweeney、John Carmack、Andrej Karpathy 和 Fridman 等知名支持者也放大了这些工作。
15. 贡献者因视频而来，因精通、影响力和乐趣而留下
Kempf 给潜在贡献者的建议是“做你喜欢的东西”。电影、动漫、字幕，以及让图像显示出来的满足感，会吸引人入场；而 16 毫秒的帧预算会迫使他们学习 C、汇编、SIMD、CPU 流水线、内存层级、I/O 和实时系统纪律。
FFmpeg 之所以成为“有史以来最好的学校”，是因为世界级审查者会暴露代码的每一个缺陷，而代码没有任何藏身之处。回报是异常清晰的影响力：程序员可以告诉祖母，“我做这些，是为了让你能播放视频”，而不必描述某个雇主之外没人会见到的内部发票门户。
16. VLC 和 FFmpeg 像二元星系统一样彼此增强
Kunhya 的简洁类比是：“VLC 之于 FFmpeg，就像 Android 之于 Linux。”VLC 依赖 FFmpeg 的解码广度；FFmpeg 则通过 VLC 接触损坏和异常文件；两边开发者重叠，捐款也曾跨项目流动，一个环境发现的故障会强化另一个环境。
x264 让这种耦合不仅是技术性的，也是商业性的。Kunhya 估计，许多 FFmpeg 流水线中超过 80% 依赖 VideoLAN 的编码器；他还称，很多公司采用 FFmpeg，正是为了获得 x264——这个开源实现让 H.264 编码变得异常优秀。
依赖图远大于每个品牌自己的代码库。Kempf 称，在 Windows 上编译 VLC 涉及约 1,600 万行代码，其中 VLC 自身只有约 100 万行，FFmpeg 约 200 万行；而 FFmpeg 又整合了 x264、Opus 以及许多其他独立库。
VideoLAN 更广泛的项目组合包括 libdvdcss、libdvdnav、libdvdpsi、libbluray、libVLC、x264、dav1d、空间音频项目和 checkasm。因此，生态没有单一中心：“提醒一下，里面是 FFmpeg”低估了项目之间相互提供能力、测试、触达和贡献者的架构。
17. FFmpeg 的历史，是一代代人解决不可能兼容问题的历史
Fabrice Bellard 提供了最初概念。Kunhya 将 2000 年代称为 Michael Niedermayer 的时代：在 Windows Media Player、RealPlayer 和各种独立编解码器包割裂播放世界的时期，全面支持 DivX、Xvid 以及无数 MPEG-4 Part 2 变体。
原生解码取代了臃肿的专有播放器和可疑 DLL 软件包，但真正的工作近乎西西弗斯式。一套中国 CCTV 系统可能实现了某种古怪的 MPEG-4 ASP 变体；要在不破坏其他所有变体的情况下支持它，就意味着将问题复制到数百万个边界案例中。
逆向工程始于对 Windows Media 和 RealMedia 的专有实现，参与者包括 Benjamin Larsson 和 Kostya Shishkov 等人。约在 2008 年，H.264 成熟和高清视频兴起，开启了另一个时代；Paul Mahol 和 Kostya 等专家开始处理那些供应商从未提供可用开源实现的编解码器。
VLC 1.0 约在 2009–2010 年到来，此时兼容性栈逐渐成熟。它的承诺——无需编解码器包、没有间谍软件、直接播放——并不只是界面上的简洁，而是多年重建彼此不兼容的专有格式、同时维持所有早期解码器运行的消费端结果。
18. 逆向工程师把可执行二进制当作规范
Kostya Shishkov 的标志性说法是“二进制规范”：文档并非必需，因为可执行文件本身会揭示编解码器的行为。Kempf 称他接近天才，能够重建 20–30 MB 的二进制数据；Kunhya 估计，即使只有 1 MB，也可能代表约 1 个月的工作。
GoToMeeting 成为 VLC 用户的重要需求，是因为旧录音需要专有解码器，否则 VLC 无法播放。Kempf 发起悬赏；约 2 个月后，Kostya 带着优雅的代码回归，还开了几个涉及 Kempf 姓名的玩笑，并提供了能够让这些会议记录在未来 ARM、RISC-V 或移动系统上继续播放的支持。
Kunhya 描述了这场考古工作：找到解压模块并挂钩，导出参考 YUV 输出，打开反汇编器，通过模式识别熵解码和变换，然后逐条机器指令跟踪。很长时间里，屏幕上看不到任何熟悉的东西；一个错误的系数缓冲区，就可能把推理带进完全错误的分支。
他的 CineForm 工作始于一个幸运的动画样本：画面包含平坦色块，复杂工具很少。更多样本逐步揭示新的分支，直到实现完全匹配；一个企业档案中 10 万个文件，可能成为验证的“金矿”。现代编解码器的逐比特精确定义，要求独立解码器对每个样本输出完全相同的结果。
19. FATE 保护组合硬件矩阵中的可移植性
FFmpeg 的自动化测试环境 FATE 持续在志愿者托管的机器上测试编解码器、滤镜、编译器、操作系统和指令集。其面板覆盖 Apple Clang、GCC、Visual Studio、Intel 编译器、macOS、iOS、tvOS、ARM、x86、PowerPC、RISC-V，以及过去的 DEC Alpha 等冷门平台。
这套矩阵捕捉到的不只是程序员回归问题。编译器有时会错误编译有效的 C 代码，而解码器中 1 个比特或 1 个像素的错误，可能沿着依赖帧扩散成大面积画面损坏；在本地运行正常的改动，仍可能在 GCC 11 或某个旧 Apple 平台上失败。
Kempf 所面对的可移植性负担，横跨 Windows XP 到 Windows 11、macOS 10.7 到节目中所说的 macOS 26、iOS 9 到 iOS 26，以及 Linux、BSD、Solaris 和 OS/2。支持 iOS 9 需要混用旧版 Xcode 和 SDK 组件，像一台“弗兰肯斯坦”，以便在 Apple 放弃 ARM32 后仍保留 ARM32 输出。
这种向后兼容具有直接的用户价值：有人可以继续把 iPad 2 当作正常的播放器，而不用购买新硬件。Kempf 将汇编和旧系统支持连接到一门逐渐失传的技艺——充分优化软件，让仍能工作的设备不会仅仅因为上游工具链前进就变得过时。
20. 汇编把架构知识直接转化为吞吐量
汇编表达的是处理器自身的指令，而不是让编译器把更高层的意图翻译出来。FFmpeg 关键使用的是 SIMD，即单指令多数据：一条操作可以同时将一个值加到约 16 个元素上，这天然适合视频像素的块和向量。
Kunhya 报告称，常见的函数级提升为 10–50 倍，明确不是 10%–50%；其中一个函数被称为比 C 快 62 倍。他的公司 Open Broadcast Systems 将 FFmpeg 的理念用于体育直播系统，包括一套手写汇编矩阵，将每一代 CPU 上的各种打包 10-bit 视频格式相互转换。
Kempf 将低层工作的重新升值归因于硬件限制：AI、视频、内存和网络需求的增长速度，已经超过单核性能提升。增加核心数只能解决部分问题，而优化那些每秒执行数十亿次的小型内核，可以实现“每一美元投入获得更多算力”——节目中提到的 FP8、FP4 量化和 Microsoft Phi-1.5 也体现了同样的约束驱动思路。
专家程序员还会“滥用机器”，例如用加密指令执行无关的视频操作，或在受控库内部绕过操作系统调用约定。这些收益要求掌握寄存器、缓存流量、指令延迟、ALU、流水线和数据布局，而普通 C 抽象无法将这些信息传递给编译器。
21. dav1d 证明手写优化仍然重要
AV1 出现时，甚至 Alliance for Open Media 的参与者也认为其复杂度要求硬件解码。Kunhya 和合作者坚持认为，在硬件部署前的多年过渡期内，软件解码不可或缺，并围绕“每一个周期都重要”打造 dav1d。
Kunhya 给出的规模对比非常鲜明：dav1d 约有 3 万行 C 和 24 万行手写汇编，而 FFmpeg 全部编解码器的汇编代码约 10 万行。一份代码构成统计——79.9% 汇编、19.6% C、0.5% 其他——成为“巅峰视频编解码器应有的样子”的挑衅性图景。
优化目标覆盖约 30 亿台设备；Kunhya 称，Netflix 视频约 30%、YouTube 视频约 50% 已经采用 AV1。dav1d 可以只用 1–2 个 CPU 核心解码 720p，把原本被认为必须依赖专用芯片的规范，变成可部署在现有机器上的软件。
实现会在运行时检测硬件能力，并为不同 x86 代际、AVX、AVX-512、ARM64、Neon、SVE、SME 和 RISC-V 选择不同内核。尽管已有多年实际案例，批评者仍认为编译器自动向量化或 intrinsics 足够；反驳仍然来自经验：“差得远。”
22. 汇编必须通过真实问题传授，否则这门技艺会消失
Kunhya 拒绝按指令逐条编写教材，因为那是在用语法而不是对话的方式教编程。他的汇编课程从高中数学、指针、寄存器和实际向量问题出发；贡献者学习后提交了可运行的 FFmpeg 补丁，说明这门技艺不必依靠师徒式传承。
他的审美类比是驾驶 Spitfire：“只有你和处理器。”操纵杆与控制面物理相连，飞行员可以把飞机推到设计师预期之外。LLM 对汇编的理解超出他的预期，但训练数据稀缺，导致它们即使被要求修复，也会反复生成无法逐比特一致的修改。
23. Rust 能改进新系统，但无法抹去成熟 C 或不安全汇编
Kunhya 重视内存安全，但认为“Rust 单独取代现有基础设施”的说法带有“非常强的世界语氛围”。如果重写项目只能实现 coreutils 的 85% 或 90%，并不等于成功，因为“最后的 1% 需要 99% 的时间”；生产采用要求功能等价、ABI 稳定、测试无缺陷和广泛编译器支持。
Kempf 已为 VLC 编写 Rust 模块，Kyber 也主要用 Rust 构建。他认可 Rust 在从零开始的解析和网络系统中的价值：所有权和边界检查，使 Rust 成为内存敏感型新系统中“更好的 C++”，尤其当整个项目可以共享一种语言和安全模型时。
他的默认规则仍是“不要重写”。写代码比读代码容易一个数量级；新人看不到未记录的业务逻辑或历史边界案例，往往能迅速重现 80%–90%，随后才发现剩余行为承载着大部分累积智慧。
用 Rust 重写 dav1d 的 C 代码，并不能让手写汇编变得安全，因为汇编仍可访问任意内存。Kempf 更倾向于研究“安全汇编”：类似 checkasm 的编译期插桩，检查优化内核是否始终处于允许的内存区域内，而不是在性能边界上摧毁 Rust 的安全保证。
24. 分叉可以强化项目，但维护压力也会击垮个人
Fridman 转述称，FFmpeg 于 2011 年从 Libav 分叉，起因是治理、领导和开发流程冲突，而不是编解码器本身存在根本分歧。一些 Linux 发行版似乎曾在一段时间内转向 Libav；随着时间推移，FFmpeg 吸收了 Libav 的大量工作，贡献者回归，审查流程稳定下来，留下的项目成为更广泛的超集。
Kempf 认为，分叉是开源改变僵局的合法方式，并举出 GCC/EGCS 以及 KHTML/WebKit/Blink 的例子。Fridman 的反驳来自人的层面，而非技术层面：即使机构冲突最终有助于项目，也可能耗尽最初凭热情维持项目的少数个人。
如今更大的倦怠威胁还包括“AI 垃圾”——糟糕的报告和补丁把审查成本转嫁给维护者。Kempf 将其与 XZ 联系起来：攻击者不断向一名孤独维护者施压，甚至在不合时宜的时间骚扰，直到他把提交权限交给一个有条件破坏项目的人。
滥用甚至可能演变成肢体威胁。VLC 在 2009–2010 年左右停止支持新的 PowerPC 后，Kempf 收到过装有粉末、写着“PowerPC forever”的死亡威胁；粉末最终证明是面粉，不是炭疽，但警方和受惊的家人都是真实存在的。他称，这段经历锻造了厚脸皮，而任何志愿者都不应被迫拥有这种东西。
25. x264 击败神圣指标，优化的是观众真正看到的内容
早期编解码器研究重视由均方误差推导出的 PSNR，即使它会激励系统把小误差均匀铺满画面，造成明显的细节模糊。x264 开发者则直接在普通笔记本上用眼睛判断输出，关心人的视频是否看起来好，而不是实验室指标是否上升。
两项创新推动了这次转向：感知率失真将人类感知和块能量纳入考量，自适应量化则根据视觉复杂度重新分配比特。ParkJoy 片段包含跑步者、水面、草地、树叶、倒影和运动；它显示，启用这些工具后，即使 PSNR 更低，画面也可以明显更好。
Kempf 还提供了来自 Planet Earth 的更严苛样本：画面被鸟群填满，数百万个运动形状会压垮普通编码器。动漫则增加了渐变、色带、特殊数字纹理、多层日语读法和复杂字幕；缺乏官方发行渠道的字幕组社区开发了类似 Aegisub 的工具，并提供了改善开源编解码器的反馈循环。
专业用户最终也押注同一套逻辑。Kunhya 提到 Warner Brothers 工程师 Chris Henderson 将 x264 用于 Fringe 盒装套装，也提到一名工程师在 Cinema Paradiso 上进行比较；不过对于高度电影化的作品，他仍偏好实体光盘，因为精心制作的 Blu-ray 在不需要奢华电视的情况下也可能更好看。
26. 新一代编解码器通过扩大搜索空间节省带宽
从 MPEG-2 到 DivX 时代的 MPEG-4、H.264、HEVC 或 VP9，再到 AV1，基本组件始终是变换、帧内预测、运动补偿和熵编码。节目给出的粗略规律是每代提升 25%–50%，概括为 30%；但 AV1 编码所需 CPU 可能约为 H.264 的 2 个数量级。
额外算力用于搜索更大的块、更多分区、更多预测方向和更多面向具体内容的工具。因此 YouTube 会广泛编码 H.264，但对热门内容重新编码 AV1：只有当交付体积减半、且覆盖数十万或数百万观众时，昂贵的服务器工作才具有经济合理性。
I 帧是完整画面；P 帧依赖更早的画面；B 帧可能同时依赖过去和未来的画面，因此解码顺序与显示顺序不同。一个 GOP 可以跨越约 250 帧，而帧内刷新可以逐步重建干净区域，避免之后插入完整 I 帧。这种压缩之所以像魔术，是因为观众从未看到它背后的依赖图。
27. 自适应流媒体在本地很简单，端到端却极其复杂
流媒体播放器下载编码片段，判断它们是否足够快地到达，再在约 7 种预制分辨率之间切换。Kempf 称，降低画质可以简单到：当一个片段的下载时间超过其播放时长的一半，就立即反应；而如何在不来回震荡的情况下重新升档，才是更难的控制问题。
Kempf 认为，音频切换可能比视频变化更刺耳：从完整 AAC 切换到 Spectral Band Replication 配置，会让声音突然变得像罐头；视觉质量变化则可能被忽略。观众能容忍 30 FPS 的体育视频画质相当差，但音频出现故障，立刻就会意识到发生了变化。
系统规模带来新的复杂度：一场直播活动必须采集内容、路由、在没有 QA 时间的情况下编码多种分辨率、加上 DRM、通过 CDN 分发，并在浏览器和异构设备上解码。接受 10–20 秒延迟可以简化网络问题；一旦要求互动控制，这种余地就消失了。
28. VLC 的安全模型同时假设媒体和分发渠道都可能恶意
WikiLeaks 的 Vault 7 文件描述过一个被植入额外 DLL 的修改版 VLC：受害者观看电影时，它收集、加密并传输文档。攻击并没有利用官方 VLC，而是利用了用户对假冒版本的信任；播放 2 小时期间正常的风扇噪音和 CPU 占用，恰好成了掩护。
中国攻击者针对印度用户的类似滥用，也曾促使印度封锁 VLC，直到 VideoLAN 发起抗辩。据报道，一个德国假分发网站已经存在 12 年：其安装程序会等待约 3 周才下载间谍软件或广告软件，从而逃避简单分析，而搜索排名却让它排在官网之前。
VideoLAN 的官方应对异常严格：不收集遥测数据，不知道用户观看什么；构建机器离线运行，并先编译自己的编译器；发布采用双重签名。两家情报机构曾要求留下后门，Kempf 的回答是“不”，并补充说，如果被强制要求接受入侵，“我们会关闭它”。
沙箱仍必须隔离 VLC 约 500 个插件、FFmpeg、着色器和供应商 GPU 驱动中的正常 Bug。团队正在像 Chrome 一样，将解复用、解码和滤镜拆分到不同的受限进程中，但多媒体沙箱必须在不复制数据、从而牺牲性能优势的情况下，传输每秒数百兆甚至数十亿比特的数据。
29. Kyber 将编解码器纪律应用到机器控制回路
Kempf 的新公司瞄准的是远程操控：此时视频是反馈，而不是娱乐，应用包括机器人、无人机、远程车辆、潜艇、云游戏、手术，以及最终对自主机器进行远程观察。其承诺是通过“让距离消失”，实现技能投射或力量投射。
Kyber 通过一条加密 QUIC 连接复用视频、音频、鼠标、键盘、游戏手柄指令和传感器流。它还校正时钟漂移，让多台摄像机、GPS、控制事件和训练记录保持同步；否则，机器人模型可能从彼此不对应的观察和动作时间戳中学习。
可靠性部分来自前向纠错：传输几个百分点的冗余数据，由接收端重建丢失数据包，避免等待会增加延迟的确认信号。CES 展示中，一辆 3D 打印的小型探测车位于拉斯维加斯附近，通过法国的摄像头和一块小型主板进行控制。
目标是 4 毫秒的玻璃到玻璃延迟，相当于约 240 Hz。Kunhya 报告称，Windows 到 Windows 或 Windows 到 Mac 的延迟为 7 毫秒；其中约 3.5 毫秒来自 NVIDIA 硬件编码器，2 毫秒来自 Intel 解码器，因此剩余预算的大部分由编解码器本身的延迟占据。
30. AV2 的竞争，既是压缩效率之争，也是专利经济学之争
AV2 是 Alliance for Open Media 继 2018 年 AV1 规范之后推出的下一代标准，目标是在质量相同的情况下将带宽降低约 30%。Kempf 预计会有 AVM 编码器，也可能有 SVT-AV2 等其他实现，以及 VideoLAN 的 dav2d 解码器——拼写为 D-A-V-2-D，被描述为“dav1d 2”，延续 dav1d 的递归命名玩笑。
平行的标准线包括 H.264 或 AVC、H.265 或 HEVC，以及 H.266 或 VVC；ITU 和 ISO 的命名反映了共同的机构历史。节目有意将每代 30% 作为实用化简，同时承认屏幕内容或其他高度匹配的场景可能获得 70%–80% 的提升。
专利经济学将两大阵营分开。H.264 的许可相对可控，但 HEVC 集合了 MPEG LA、HEVC Advance、独立权利人、无上限索赔以及大量主要用于建立专利地位的工具；Netflix、Google 和 Meta 可能需要每年考虑数亿美元，转而资助免版税替代方案。
Alliance for Open Media 在标准化过程中必须考虑专利，迫使团队进行“双重创造”：一边提升压缩效率，一边避开受保护工具，有时甚至创造出类似“我不敢相信这不是 B 帧”的东西。法国拒绝软件和数学专利，也让 VLC 可以实现某些格式；Kempf 估计，如果合并计算许可费，每位用户的成本将超过 200 欧元。
31. 档案工作者把 FFmpeg 当作未来文明的罗塞塔石碑
Kunhya 强调了与 Dave Rice 及 British Film Institute 等机构相关的档案社区。成员重视开源，不只是因为预算有限，更因为播放软件必须在机构、供应商和硬件更替后继续存活；相比之下，英国 New Domesday 项目在几十年内就变得难以读取。
FFmpeg 的 C 实现是这项押注的核心。Kunhya 预测，1,000 年后 C 仍然会像拉丁语一样可理解：“也许不再是日常语言，但与逻辑和数学足够接近，未来工程师仍能重建编译器，并理解历史比特流如何变成图像。”
档案工作者资助了 FFV1，这是一种无损编解码器，目标是快速软件编码、抗损坏，以及存储位翻转时将损害限制在局部；他们也支持 GPU 加速。他们的要求是不丢失整个 GOP，并用数学证明细微的历史证据没有被压缩掉。
他们的领域知识也回流到软件：1950 年代磁带色度学、电影扫描仪、老化胶片、过时磁带磁头和存储恢复。面对多到机构无法全部保存的资料，决定哪些电影和磁带能够存续，成为 Kunhya 所称的“数字监护”；他还特别指出，这个词是自己创造的。开放工作流也让没有富裕档案馆的国家能够由志愿者教授基于 FFmpeg 的保存方法。
32. 多媒体将从图像和声音扩展到所有同步感官
Kempf 对多媒体的定义很宽：它是面向人类感官的多条数字化数据流。FFmpeg 和 VLC 已经要处理点云、体积视频、RGBD 深度、XR、空间音频和 3D 资产；VLC 甚至支持过液压“4D 影院”系统中的同步触觉控制。
合乎逻辑的延伸，是加入触摸、运动、气味或神经接口轨道的容器；Fridman 开玩笑说可以有立体气味，以及面向人脑的
ffmpeg -i。Kempf 毫不讽刺地回答：只要一种定时数据流代表了人能够感知的东西，架构最终就应该能够为它接入模块。因此，维护者的职责不是亲自实现所有未来设备，而是保持架构的可扩展性。Kunhya 称，VLC 每年约有 150 名贡献者，FFmpeg 约有 300 名；核心团队创建第一个模块和接口，之后的专家便能添加第二、第三种实现，而不必重做整个框架。
现有部署已经足以拓展想象：FFmpeg 为 Mars 2020 探测车压缩图像，VLC 监控 CERN 长达 27 公里的加速器环，Formula 1、SpaceX 和 European Space Agency 都使用过这些工具。Kunhya 最喜欢的总结是“FFmpeg，没有什么超出我们的能力范围”；他最后留下的纪律同样持久：“从错误中学习，但不要后悔。”
The important thing is: Is your code good? We care about excellent code. We don't care who you are. Maybe you're a dog—I don't care, right? I don't care where you come from. I need to look at your code.
“Oh, yeah, but I'm an engineer at this very large company in Italy, Germany, or the U.S.” We don't care. We care about the quality of your code because this is what defines our community. It means that we have a lot of people who contribute who are from very different backgrounds and are very introverted. Sure, but that's okay, right?
FFmpeg is probably one of the biggest CPU users in the world. Everything we've just said in the past couple of minutes—every sentence is someone's lifetime's work. There are books about every sentence, so the level of complexity in many cases is inordinate.
FFmpeg has 100,000 lines of assembly for all the codecs.
For all codecs. Mm-hmm.
And just this one has 240,000. Every cycle matters. We're talking about probably 3 billion devices that are going to decode video nonstop because, for example, 30% of the video from Netflix is now in AV1, and 50% of YouTube.
This is what peak video codecs should look like: 79.9% assembly, 19.6% C, and 0.5% other.
What's incredible is that with those tweets, which are factual, people get crazy. For the last two years, they go crazy: “No, intrinsics is fine. The compiler—”
“You can optimize your compiler. Auto-vectorization—it's your fault. You don't understand.”
And we've tried that forever, right? For two years, and two years later, after showing hundreds of examples of handwritten assembly: “No, no, no, you're doing it wrong. The compiler can do this.”
The intelligence agencies tried to say, “Can you put a backdoor in VLC?”
Yes. 2 of them.
What did you say?
No. Well, I was a lot less polite.
Basically saying, “Hell no.”
If we had to compromise our software, we would shut it down. This is clear.
Any tweets, Kieran, you regret?
Tweets I regret?
Or is it like that—how does the French song go? “Regret nothing.”
Don't regret anything. No, it's because regrets are attacks on your mind.
The following is a conversation all about FFmpeg and VLC with Jean-Baptiste Kempf and Kieran Kunhya. FFmpeg is an open source software system that is the invisible backbone behind YouTube, Netflix, Chrome, VLC, Discord, and basically every platform that touches video or audio on the internet. It can decode, encode, transcode, stream, and play almost any video or audio format ever created. To me, it is one of the most incredible software systems ever developed, and it’s all done by volunteers. VLC is also a legendary piece of software. It is an open source media player that plays basically anything you throw at it, any format, any platform, no ads, no tracking. It has been downloaded over six billion times, and again, for me, it has been one of my favorite pieces of software ever, with the most legendary logo, which I, of course, had to honor in this conversation by wearing the VLC traffic cone hat the whole time. So again, above all else, thank you to the incredible volunteer engineers who put their heart and soul into this code that has been used and loved by billions of people. Thank you. And about the two great engineers and human beings I’m talking to in this episode, Jean-Baptiste is the president of VideoLAN and is a key figure behind VLC and FFmpeg. Kieran is a longtime codec engineer, FFmpeg contributor, and the man behind the now infamous FFmpeg account on Twitter/X that I recommend everybody follow for the memes and for the unapologetic celebration of open source and great low-level software engineering. Let me also say that it’s inspiring and humbling that so much of modern civilization rests on software built by people who are not chasing fame or money, but are obsessed with the craft of engineering. We live in a world where billions of people consume video every day without ever thinking about the invisible machinery underneath it. But that machinery matters. Open source infrastructure matters. It is one of the great examples of human beings quietly collaborating across borders to build something useful, durable, and elegant for the rest of us. And so this conversation is not just about codecs and media pipelines. It is also about the deeper spirit of engineering and generosity that makes projects like FFmpeg possible. Again, I can never say it enough. Thank you. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback, and so on. And now, dear friends, here’s Jean-Baptiste Kempf and Kieran Kunhya.
So the legend goes that VLC can open everything. What's the weirdest thing that you know it can open?
There are a ton of people using VLC to record VHS videos, right? You just plug it in with a capture card, and you can basically record VHS video.
How does that work?
Basically, it's those types of capture cards where you can put in a Péritel or RCA, and VLC can play those types of cards. There's a module that allows you to control some of those VCR camcorders directly.
We support DVD-Audio lately, right? We spent the summer working on DVD-Audio support, and no one is making any DVD-Audio support. There are custom encryption schemes.
What about Lucasfilm?
Oh, yeah, and there are, of course, all the weird codecs supported by FFmpeg—game codecs.
The first 10-second opening sequence of one Star Wars video game—someone went and implemented that and made sure it was bit-exact on one disc that existed at one time, for one little sequence in the game.
Funnily, at one VideoLAN conference, we made a competition to make the weirdest and most horrible file ever and see if VLC could play it.
What did it end up being? What's the file?
It was an MKV file made by Derek, in which each frame was changing resolution, aspect ratio, and rotation.
Did it work?
Yes. There was another one where the whole video was actually animated subtitles, right? SSA.
Yeah, I remember that.
Each frame was a black frame, but on top of that there was an animated subtitle for each frame.
There was a file that was a valid ZIP and a valid MP3 at the same time or something like that.
So, yeah, we'd made a competition of stupid files.
And it worked. It opened all of the stupid files.
Yes.
By the way, for people who aren't familiar, I'm wearing a hat. Would it be fair to say this is the best worst logo of all time—the cone?
Yeah, by far, right? The logo of VLC is so iconic. We're a team with a small number of people, and the icon is known everywhere. I go to the middle of nowhere in India or China, and people know the cone, right?
Twenty-five percent of the website traffic that comes to our main website is “cone player,” right? So many people don't know VLC. They know the cone player.
That's the thing they Google for: cone player.
Yeah. They go on Google and put “cone player,” and they download VLC, right? So that's iconic.
Once we tried to change it as a joke, right? We said it was going to be a type of caterpillar construction, and we announced that during April Fools' Day. We had around 10,000 emails saying, “No, don't change the logo,” and so on, right?
It's so iconic. It's so distinctive. If you want to do a video player, you're going to put a play button on a TV, right? That's the YouTube logo, right? It's unoriginal. This one is orange, right?
Yeah.
It's very bright and it's weird.
And it's ridiculous, absurd, and hilarious. It becomes a meme, and a meme becomes culture.
And you keep it, and you know that in 20 years you're still going to have the cones and remember, “Oh, yeah, that was a video player.”
Yeah. And we'll talk about the mission of FFmpeg, being kind of the archival aspect of it. You can think that 1,000 years from now, we'll have all these videos that only VLC can open.
Human civilization has already destroyed itself multiple times, and the only thing that will remain is this. The cockroaches will be crawling around, and it'll be the VLC logo with some of the archival footage that VLC can open. The aliens will show up, and they'll press play and get to see it all.
Well, I really, really hope so, right? But there are also so many memes where people say, “Well, I'm sure I can put a pancake inside my DVD drive, and VLC will play it.”
Can they?
No, we tried. It doesn't.
It doesn't.
But we actually have a video of us trying that. It didn't work.
A codec for physical reality—I don't know what that would even look like.
There was a guy who did that, right? He printed a small cone, like the ones we distribute as goodies, and inside he put an RFID chip, which was his way of playing a movie. He put it on an RFID player, and when he put it on there, it was playing The Last Star Wars and so on.
Instead of having DVD boxes, he had VLC cones all around, and he plugged that in. Those were like physical objects.
The thing we're talking about is everything around video codecs, video encoding, video decoding, video streaming, and the video player client that I'm wearing on my head—the entire ecosystem enabling free media.
We'll talk about FFmpeg, VideoLAN, VLC, and all the other incredible video technology used probably by billions of people. So, JB, you're the lead developer behind the legendary VLC player. Kieran, among many other things, you're the man behind the legendary FFmpeg handle on Twitter. Both of you have spicy opinions, I would say.
So today we want to talk about FFmpeg and VLC. For context, people who aren't aware—and I'm sure basically everybody listening to this has used these two technologies, probably regularly, without knowing it—FFmpeg underlies basically most video on the internet, including YouTube, Netflix, Chrome, Firefox, of course VLC, and countless other video platforms. It's estimated that over 90% of video processing workflows, online and offline, involve FFmpeg.
VLC has been downloaded at least 6.5 billion times, but likely that number is much higher because it's impossible to really count. Virtually any operating system supports virtually any media format, the limitation being that it can't open pancakes. Can we just lay out some of the basics to help people understand what's involved in all of this? When we press play on a video player like VLC, what happens? How does it go from the file or the stream to the pixels on the screen and the sound on the speaker? What are the big stages to be aware of?
There are several stages. The first stage is to get from an address, which is a type of URL, to a byte stream. This would be, for example, HTTP, a file, or a DVD. You give the path to the media, and it gives you a stream of data.
The stream needs to be cut up by what's known as the container, the demultiplexer, or demux. We'll try to keep the jargon light throughout this, but it needs to start demarcating video and audio frames. It gets data from the operating system, blocks at a time, and starts cutting these frames up into compressed data.
It then needs to start doing simple parsing of the video frames, mainly to figure out whether that codec is GPU-decodable or needs to fall back to software. We're very used to assuming the GPU will play all of these things and that there'll be hardware acceleration. I think up to 45% of files are not GPU-decodable, so these need to be probed and detected. There can be variants of a given codec, some of which are decodable on the GPU. Different GPU vendors might have different capabilities, so those need to be detected.
If it's GPU-capable, you pass it through to the GPU black box. If there's a software fallback, that means you first need to do entropy decoding, removing the mathematical coding of the bitstream. This uses capabilities such as Huffman coding or arithmetic coding to decompress the mathematical layer of the bitstream.
We then need to start reading the syntax elements for intra prediction. Intra prediction is like still images of the video—your I-frames. This operates in the spatial domain. You have a residual because your prediction isn't quite matching reality. You've made a prediction, but then there's a little bit left, and that's what's known as the residual. This is stored in the frequency domain, and the coefficients are quantized to compact their representation. We then need to do the inverse transform to bring them back to the spatial domain and apply these residuals.
A lot of the process of decoding is that this thing is compressed, and you have to predict the highest-quality thing that's supposed to go there. An I-frame is the best representation you have spatially. Then there's a lot of temporal compression that can happen, depending on the codec, and then you're predicting. You're predicting what the reality was that was captured in this raw form.
Yes, because what people don't realize is that the compression on video and audio is 100 times. People don't realize how compressed we make things. For audio, when you move from normal audio to MP3, you compress by 10 times. When you move to video, you need 100 times, 200 times. You need to remove all the details that you don't care about, because all the compression that we do—and that's very important, people forget about that—is intended to be viewed by humans.
All the codecs, whether for audio or video, basically mimic how your ear works, and a lot of things about the response of the ear. The same goes for your eyes. For example, with video, we don't work in RGB. Everyone expects us to work in RGB, but we don't. We move to YUV, which is basically one component for luminance, or brightness, and the others for color. This matches your eyes, where inside your eyes you have cones and buttons. Some of them respond to brightness, and others to colors.
We need to compress a lot, and so we need to degrade the signal. In order to degrade it, we need to match human perception, and this is why it's so difficult. We need to use the maximum mathematical power—very complex technologies. We move to the frequency domain, as Kieran said. We do a ton of dequantizing in order to get the best compression while still making it look good.
You're trying to compress in order to maximize the highest-quality thing for human perception.
That is correct, and this is very important. Compression is not like a ZIP file. With a ZIP file, you have data in and you get data out, and you try, with all the ZIP compression, to arrive at the limit. Here, we are degrading the signal. We need to degrade both the audio and the video signal in the best way possible.
We can do that, but it involves, first, a lot of theoretical knowledge about how the eye works, and also a lot of mathematical changes and a lot of mathematical tricks. For example, when you move from RGB to YUV, what we do very often is scale down the resolution of the color compared to the brightness. Most of the time, just this, without compression, divides the size by 2, but most people don't see it.
Then you go to very complex mathematical transformations. Of course, there are Fourier transforms, which, de facto, are not Fourier transforms; they are like discrete cosine transforms, but it's the same idea. In the frequency domain, we split the video into blocks. That's why, when it's wrongly decoded or badly encoded, you see those blocks.
We arrive at compression rates that are insanely high. Each generation of the codec is about 30% less data for the same quality, and this requires a huge amount of computational power.
No, no, but you should elaborate. It's 30% better, but it requires an order of magnitude, perhaps even 2 orders of magnitude, more compression power. That's the big difference.
What do you mean by compression power?
Sorry, CPU power to achieve that level of compression. You have to be able to leverage the CPU and sometimes the GPU, like you mentioned. We should mention that a lot of this programming is done at the lowest possible level of the stack, whether it's C and, of course, as the legendary FFmpeg Twitter handle reemphasizes over and over, a lot of assembly.
What happens globally is that you have an address, which gives you, with the operating system, a stream of bytes—a stream of data. This is the first step. The second step arises with demuxing, where you're going to separate audio, video, and subtitles into different tracks. Then, on each of those tracks, you're going to decompress or decode them: audio with an audio codec, video with a video codec, and subtitles with a subtitle codec.
Once you've decompressed those things, you have raw images, and then you're going to talk with your graphics card and your screen and display them. The same goes for the audio: you're going to talk to your audio card, which then converts the signal to analog and sends it to your audio speakers.
Everything we've just said in the past couple of minutes—every sentence—is someone's lifetime's work. There are books about every sentence. The level of complexity, in many cases, is inordinate. Every sentence has thousands of people working on it in the industry as a whole, and books have been written about it. There's a lot of detail, a lot of subtleties, and a lot of both academic and practical realities, both of which matter.
We mentioned codecs, but I don't think you mentioned containers. What are the actual containers for some of the stuff we're talking about? People are familiar with MP4, MOV, and MKV. What are containers versus the thing that goes inside?
The container is what we also call the muxer. When I say demuxing, it means decontainerizing. If you look at the terms, mux means multiplexer, and demux means demultiplexer. A codec is actually coder-decoder.
Containers are a collection of multiple tracks. It's what normal people call the file format, but it's a bit more subtle than that. The most well-known one, of course, is MP4, but when I started, it was AVI. AVI was the video format from Microsoft, and MOV, M-O-V, which became MP4, was a format from Apple.
In the open-source community, one of the people who is still active on VideoLAN is called Steve Lhomme, and he started the Matroska format, which is a bit more complex and more future-proof. There are many others.
There's a pretty common thing—and maybe it'll even happen in this conversation—that people confuse the container and the codec. They confuse MP4 and H.264, for example. Is that a horrible violation?
No, it's not, because technically the name of H.264 is MPEG-4 Part 10. MPEG-4 is actually a meta-specification that has several things in it. There's Part 2, and there are also audio codecs. AAC, de facto, is MPEG-4 audio. There are actually several video codecs inside the MPEG-4 specification.
One of them is MPEG-4 Part 10, also called AVC, also called H.264. Right? So it's completely the fault of the industry to make things difficult to understand. It's very difficult, so people then don't understand why sometimes you talk about MPEG-4 Part 10, when you mean H.264, and why it's not MP4.
So you can technically shove in all kinds of different codecs inside containers, and horribly so.
But broadly speaking, though, MP4 is understood to generally be H.264 plus AAC audio. 99% of the time, that's that, and the rest are de minimis—the small effects, edge effects really, compared to that. So it's not the end of the world. There are people who do get annoyed by that.
But also, in reality, something like VLC—the file may say .MP4, but it may be something completely different, and that's one of the challenges both FFmpeg and VLC have: the real world is a completely different place from a 3-letter file format.
And this is very important to say. For example, in VLC and in FFmpeg, we discard the file format, right? We look into the file to understand what's in it because so many people say, "Oh, it's a video, it must be MP4," but technically it's an MOV or maybe an MKV. We analyze in real time everything that we have, and we don't trust the format.
So what information does the fact that it's .MP4 give you?
It helps, right? It gives you a hint. Just like, "Oh, it ends in .MP4. I'll start by opening, probing it with the MP4 container demuxer to see—well, it should be that." But I don't trust it, and if I'm lost, I say, "Okay, maybe I'm going to try it." So it bumps the priority of the module.
So, just to take a bit of a tangent there, the dumb thing is, if you try the MP4, but it turns out it's a different codec than you would have expected, most players just break there.
Yes.
This is why VLC is popular. The reason is that VLC was just a client of a streaming solution called VideoLAN from a very long time ago, from the late '90s. And when you're playing video which is on UDP, on a network, it might be damaged. So you don't trust your inputs, and this is very important in security: you don't trust your inputs.
Everything in VLC is prepared to work with broken files. It's a philosophical idea from the beginning, and everything is engineered into that. And it's a culture. So, for example, VLC became very popular because, a long time ago, when people were pirating content—which they do a lot less today—
And none of us ever have—
No, of course not. The metadata to play some files like AVI is at the end of the file, right? And when you're downloading, you don't have that. So VLC was just like, "Hey, this file is broken, but I'm still going to try to interpret it," and this was very useful.
We've hinted at the awesomeness of the various different stages. We hinted at the awesomeness of codecs—the depth, the richness, and the complexity of everything involved there. Let's try to define what a video codec is. What's involved there? What does it mean to compress something? You already started to hint at it, but can we elaborate a little bit more?
So there's a huge amount of redundancy in any video, both spatial and temporal, and the point of any video codec is to remove this redundant data and use mathematical properties as part of this reduction process. More often than not, it uses several orders of magnitude more compute to compress because that's more costly, both financially and in CPU resources, versus the decompression. So it's asymmetric in that respect, often because compression is done once, but there could be lots of viewers of a file.
So you take that information and compress it by 100x or 200x, removing redundant information and using mathematical properties to make that small, but also have properties such as error resilience. As JB suggested, VLC in the beginning was used to play UDP network feeds, and UDP network feeds lose packets. Some of the design goals of a codec are also to be recoverable. You need to actually be able to join a stream. It's not necessarily a file. You need to join, get on the decoding process, and start decoding.
And to give a more concrete image to people who are not familiar, when you're going to see any type of movie, the camera is going to pan and travel. And you realize that, for example, all the background is the same for a minute, or 30 seconds. So you can reuse the cloud that you see in the background; you can reuse that from one frame to another. And the more memory you have, the more power, the more comparisons you can make, and so the more compressed you can be. Most of the modern codecs are basically doing that.
So, just to make it even more explicit, what is video? Video is a bunch of pixels, often RGB. You have 3 values, and you have a grid of pixels, and you have, let's say, 24, 30, or 60 frames a second, and you just have all these pixels repeating and showing different stuff 30 times a second. The philosophical, technical question is: how can I compress all of that, store all of that at 100x?
Michael Niedermayer
Yep. Or 1,000x, right?
1,000x.
Michael Niedermayer
The target is 1,000x, right?
And the goal, when you say redundancy, what is redundant? Meaning stuff that, at best, humans wouldn't notice if it was missing.
So, for example, you have a picture of a cloud, and in the next frame, it's still going to be the same cloud, so it's redundant. You could just put it once and not do it again. Or you have a black background behind me, for example. The black is the same in the whole picture.
So you can say, "Well, in this picture, take the pixels that you have on the top left and the one on the top right. I'm not going to give the value. I'm just going to tell you it's the same as the top left." And then you can say, "For frame 1, reuse something from the previous frame or the previous, previous frame," and so on and so on.
Basically, it's unlimited, but then it's limited in terms of memory or compute power. For example, if you need to compare pixels on 200 frames in the past at 4K resolution, it's a huge amount of compute.
And then when you're showing it, you have to do the decompress of all of that. So is the codec—the encoding and decoding—a coupled process that you're developing?
Michael Niedermayer
Yes, exactly. And those are 2 different trade-offs. Are you going to compress more? But then it might be more difficult to decode. Are you going to make it a codec that is more complex to encode and easier to decode? Are you going to make a codec that is easier to encode because you need to be fast, but then the client side, the player, is going to spend more time?
That's why you have so many different types of codecs: it's not always easy. And to make it even more complex, modern codecs like AV1, AV2, or VVC are actually not codecs. They are a collection of tools. There are multiple tools, multiple codecs in the same codec to get more compression depending on the image.
So, just to elaborate, codecs like AV1 and VVC have a much wider audience. It could be screen-share content, it could be video, it could be animation. All of these require different coding tools.
So what happens these days is that a collection of tools are put in and called AV1, called AV2, called VVC, to allow for different use cases. You may be on Zoom and sharing your PowerPoint, and then you need to show the audience a video. That codec needs to start changing its tool set depending on the content to compress in a different way.
And like you said, there's a bunch of incredible engineers behind each part of that, each part of the tools that make up AV1, for example. We've kind of danced around it. We talked about VLC—the logo, the hat. Let's talk about FFmpeg. What is FFmpeg exactly?
FFmpeg is basically the low-level libraries for codecs, so compression and decompression, muxers and demuxers, and filters. The core is this, and then you have several tools which allow you to create a type of pipeline to process any type of video file. And it's used as a library absolutely inside everything from VLC to Chrome to your smart TVs. Basically, any video that you see online usually uses FFmpeg.
And FFmpeg in itself has all those types of tools, and sometimes depends on other libraries like x264, libvpx, and others. So it's really now the de facto tool to process images.
From a philosophical level, I think it's incredible that your home videos, your grandmother's home videos, and trillion-dollar corporations are effectively on a level playing field, using the same technology stack. It wouldn't be a surprise if these big companies just had 3,000-line FFmpeg commands. There are some that use the API, but there are some that just have long command lines.
So, yeah, there's a bunch of tools, literally command-line tools: FFmpeg, of course, FFprobe. There are libraries: libavcodec, libavformat, libavfilter. But FFmpeg on the command line is legendary because there are so many parameters. You can customize everything to hell.
It's a language. It's an actual language.
It's an actual—yeah, you could think of it as a programming language.
Yeah, of course, I'm sure. Most people are going to take FFmpeg, file in, file out, and specify the format, right? But you can—We've seen thousands of characters, and we've also seen people generating command lines programmatically to make FFmpeg.
There are a ton of people using AI to generate command lines for FFmpeg because they have no idea what they are. But you can specify so many filters right on the command line, right? So FFmpeg is this collection of toolboxes for multimedia processing that everyone uses. Everyone who's watching your videos is also using it, right? You're on YouTube—it's FFmpeg on the server side. On the client side, it's probably Chrome, and you're using FFmpeg there also. You're using OBS to record. Well, it's FFmpeg, right? You're using a ton of important, big, professional boxes. It's very possible that some part of FFmpeg is running inside them.
I mean, there's so much to give people an idea. I use FFmpeg a lot, on everything—trivial stuff like taking a video, adding an intro and an outro, and fading one into the other. What is it called? Dip to black, where it dips and then shows the next video, and doing the same thing with audio. There's a cross-dissolve of the audio: it quiets the audio and makes it loud again. Then there's a bunch of stuff like showing the captions on-screen, like baking the captions in. You can customize the font. You can do all kinds of layering of audio and video. There are a million things, and of course, all of that works magically with basically any codec. Anything you can shove in on the audio and video side, it works.
But if you look at it, for example, you can do things that you would do with Adobe After Effects on the command line in FFmpeg, right? It's very interesting because, for example, for imaging, there isn't such a tool. There are a few tools, but not with the breadth of FFmpeg.
So ImageMagick has a similar kind of—
Yes, but you will not—
...spirit, but it—
...do some filters, complex filters. You don't have the equivalent of Photoshop on the command line, right? But for video, you have FFmpeg on the command line.
Yeah. It's incredible. It's an example of a thing where a bunch of great people get together, get a vision, and stick by that vision for many years, which is incredible.
The vision behind VLC and FFmpeg is that we make everything that's very complex easy to use for normal people—for everyone. Our goal is to make something that's insanely complex technically easy to use, right? People use VLC; they drop in a file. They don't realize how complex the file is, but they play it. Or people put any type of thing inside FFmpeg with complex filters, and it just works magically, right? This is our mission: make very complex things easy.
We wouldn't be here, and you wouldn't be here, if this required a traditional television studio setup. It's tools like FFmpeg that democratize this. The podcast and streaming revolution—the YouTube revolution—was caused by this. FFmpeg was a big player in that because it democratized technology that, once, in the 1990s, required equipment that cost hundreds of thousands of dollars to do compression. It was the size of a car, and now everybody has that at almost exactly the same level. That's something that's so remarkable.
It gave voice to a lot of people. Just to clarify, when we say, "You wouldn't be here," we mean not the human, but the podcast.
The podcast. Oh, sorry. You as a... Sorry.
I would still say VLC did not have anything to do, on a biological level, with creating me as a human.
But it's like, you realize also that everything moved from text to images, and images to video, right? Look at social networks. Video is everywhere. It's the most powerful medium there is, right? When you see Shorts and Reels and TikTok, it's amazingly powerful. Video is amazing for that, right? But the complexity is important.
This is what people don't realize. I mean, this really gave power to the individual all across the world. That's real freedom. I can't believe it, but we still haven't mentioned the obvious thing for people who aren't familiar: it's open source, and there's an open-source community of users and developers behind it. It's really a movement. We'll talk in a bunch of different ways about the community behind it, but can you speak to the open-source element? When we say, "What is FFmpeg?" it's an open-source project.
Yeah. So FFmpeg, VLC, x264, VideoLAN—everything we do is fully open source. For the people who don't understand what open source is, my usual analogy is a chocolate cheesecake. Usually, when you want to buy your cheesecake, you go to a bakery, and they give you the cheesecake. Another way of having a cheesecake is to have your grandma give you a recipe for how to make it. When we do open source, we give you the chocolate cake, and we give you the recipe to remake the same cake, but at the same time tell you how to build the oven and how you're allowed to modify the recipe and resell it to someone else.
This is because software is just a very long recipe of small instructions. Computers are not very clever; they go very, very fast. A normal program has tens of billions of instructions instead of the tens you have in a chocolate recipe. A lot of the software industry was about selling software, where you just have the final cheesecake. In open source, we give you everything, and that managed to get a lot of people to work together, right? Then you decide that you're going to make the best program, the best recipe for video, and you create communities.
In FFmpeg, since the beginning of FFmpeg, probably 2,000 to 3,000 people have contributed from the beginning, right? It's exactly like the Linux kernel, right? The Linux kernel has probably 10,000 people contributing everywhere, and they get together—mostly online, right? So they virtually get together to create the best tool for something. On FFmpeg and VLC, it's just like: this codec doesn't work, so I'm going to work on the codec, and I'm going to add support for this file inside FFmpeg, so it will be beneficial to everyone. Because, again, we work for the greater good. We work for everyone, and that is what open source is.
And we should mention that, depending on the licensing, you could probably build a billion-dollar, maybe even a trillion-dollar company as a wrapper to—
Well, yes—people do, right? There were a lot of problems, mostly with cloud providers, who are basically running some open-source tools in the cloud and just giving you an API to access them. There were a lot of databases, like MongoDB or Elasticsearch, that changed their licenses in order to avoid those types of scenarios.
A question we get a lot in FFmpeg is, "Why don't you do that?" And you can't. We have thousands of contributors, some of whom aren't even alive anymore. It would need all of their agreement to do that, and JB will maybe, a bit later, talk about how challenging that process was in VLC to do the relicensing.
The license is a social contract, in terms of Rousseau, de facto, of the community. The community doesn't agree on much besides the license. People go around and discuss because of the license, and that also allows those licenses to fork, right? Sometimes the community splits, but it's possible because of the license to merge back. We've seen that so many times, right? GCC and EGCS in the past.
We've seen, for example, all the web browsers, right? They started as, like, KHTML, which became WebKit and then became Blink, right? So open-source licenses are the core of the community, and people are coming from all around the world, with very different types of religion and political borders. They work in the same way on a project to solve a specific problem, and the specific problem we're working on is to make multimedia easy for everyone.
Looking it up on Perplexity here, looking at the different open-source licenses: most major open-source licenses fall into 2 buckets—permissive, with very few conditions, and copyleft, with share-alike requirements for derivatives. Below is a brief practical summary of the main ones you'll see in the wild.
MIT License, BSD, ISC, Apache, GNU GPL, GNU AGPL. Where's LGPL? Yeah, LGPL. Let's see. There's the Mozilla Public License. There's the Eclipse Public License. It goes on. There's a lot of variety. I mean, I think the really popular ones are MIT, GPL, LGPL, and BSD—Apache as well. Sometimes you'll see Apache. Unlicense, that's an option: attempts to dedicate code to the public domain with a fallback permissive license.
There are many licenses for many different things. What people don't understand is that the public domain is something that doesn't exist worldwide. All open-source licenses use copyright law—the international copyright law—in order to give rights regarding how you use the software or how you modify it. It's de facto a copyright-license contract that you give to the end user or to the developer.
So you have the first ones, which are basically very permissive: MIT, BSD. You give the code and basically do whatever you want, right? You take it, you modify it, you do what you want. This is popular for JavaScript and the type of BSD operating system.
So one of the parameters is whether they require attribution—meaning, if you use the code, you have to say—
Yes. In those types of permissive licenses, some require you to say if you use it, which is called attribution, and some don't. Then there is the other part of licenses, which are copyleft, where you need to give back to the community your modifications, with different strings attached.
There are some weak copyleft licenses, like the Mozilla Public License, some that are a bit stronger, like the GPL, and some that are very strong, like the AGPL. All of those are different types of licensing that depend on what your goals are and how you want to structure your community, which is why I spoke about the social contract, because this is very important to understand.
FFmpeg and VLC are mostly GPL or LGPL. The Linux kernel is GPL, but Android is Apache. A ton of JavaScript frameworks are mostly MIT. All the BSD kernels—OpenBSD and NetBSD—are, of course, BSD. It’s a philosophical choice in how you want people to contribute back, basically.
So I think you talked about that: You moved at one point from GPL to LGPL on certain parts of the project. Can you describe the difference between the two, and what does it take to move to, I guess, a more permissive license? That direction is more permissive; LGPL is more permissive than GPL.
Yeah. You have to realize that you can always go from more permissive to less permissive, right? Of course, those licenses are basically statements, and if you restrict, you can always restrict more. In a GPL project, you can take MIT code, but you cannot do the opposite, because they are more constrained to match.
Indeed, I changed the core of libVLC—the engine of VLC—from GPL to LGPL. There were 2 reasons to do that. The first one is so people can use the VLC engine, libVLC, in third-party applications. A lot of applications that play video on your phone or tablet actually have the VLC engine in them, which calls FFmpeg.
That was one of the ways to create one of the companies I created, which does consulting and integration of those types of applications, where you integrate VLC into third-party solutions, like inside game engines or stuff like that. With GPL, you couldn’t do that, because that means you needed to open-source everything, and a lot of commercial companies don’t want that.
So you can create a company with LGPL, you can create a company around it. You can do a commercial thing. You don’t have to open-source it. So that’s a big, big leap. So you could play video in your game.
Yes.
The problem is, I’m a game developer, and I want to play some videos, and I don’t want to be forced to open-source the entire game just to play those videos.
So that’s where the consulting business—the libVLC LGPL—allows you to do that. The LGPL, the Library GPL as it used to be known, allows you to do that.
And FFmpeg is exactly the same. LGPL forces you to give back what you change in this component, this library, which is why it’s Library GPL. You can use FFmpeg as LGPL in any type of application, even non-open-source applications, but you need to give back the modifications you made to FFmpeg. The same applies to libVLC.
Is it limiting from an open-source perspective to go GPL? If your library, if your code is GPL, it means you’re basically discouraging companies from building a business around it, right? Is that fair to say?
Yes. It depends on the company, but for a company whose business model requires the application to be closed-source, yes, it’s limited. That’s why, for example, I moved to LGPL.
The second reason is a bit more obscure: The terms and conditions of the Apple App Store for iOS make it very complex to have GPL applications on it, while it’s easier to have LGPL applications on it. So VLC on Windows, Mac, and Linux is GPL. The core is LGPL. But on iOS, the iPhone version and the Apple TV version use a different type of license called the MPL. And yes, I went and changed the license, and it was a long story.
Yeah. So I think basically, to change the license, you have to contact all the contributors.
Yes. It’s very important to understand that open-source projects are what we call, in U.S. copyright law, joint works, or in civil law, collective works or collaborative works. You work all together toward the same goal, and then you create one piece of software, which is one release. But the copyright is kept by all the individuals.
Some open-source projects don’t do that. They force copyright assignment, but this is not what we do. We’re communities. Everyone has copyright on what they changed. That copyright stays even if, at the end, your contribution was deleted, because the new contribution was based on your previous one, right?
So if you want to properly relicense, you need to find all the contributors. At that time, I had to contact more than 350 people. Sometimes, they’re just an email address, right? So you need to actually track them down. I actually traveled to some place to find someone I had located online. I went to their job and said, “Well, you licensed that. Do you want to change from GPL to LGPL?”
Most of the time, they don’t even care. They wanted to help VLC. But it also brought me into very complex situations. I arrived at the workplace of a person who was a factory worker, and I said, “Well, I need you to sign that,” because it was his son who had died who had actually written the code.
I had to explain all those aspects of open-source licensing. No, I was not a company trying to rip out the 2 lines or 5 lines that this guy wrote, but they were useful, and the whole community agreed on that. I had no idea he was a factory worker. I was a lot younger—it was 14 years ago—and I was almost in tears.
It’s very difficult, right? We are talking about people’s lives. We talked about the photo of this guy. It’s important to do it right and to do it correctly. But yes, that means tracking down everything, because every contribution counts.
There are some projects that don’t respect that, and they do relicensing a bit aggressively. But as I said, it destroyed the whole heart of the community, because we only agree on the license, so that’s important.
I would emphasize that the community is such a wide-ranging group of people. There are people in the Syrian war zone with electricity part-time. There are people from all walks of life—rich, poor, young, old. So it’s quite remarkable to get a group of people aligned on something. That’s an achievement in itself.
Yeah. It’s incredible. And a lot of them are introverts, so coming to find them and getting them to answer an email might be quite difficult.
Most of us are introverts, right?
You need to be more precise. You have extremely introverted people, extremely, extremely introverted people, and introverts, right? It’s just a whole spectrum of different people. It doesn’t matter. The important thing is: Is your code good? Is your code great? Is your technology great?
We care about excellent code. We don’t care who you are. Sorry, we have no idea how to check. We cannot check, right? Maybe you’re a dog. I don’t care, right? I don’t care where you come from. I need to look at your code.
This is important because people don’t understand that. They come to the community and send in some patches, and they get rejected. They don’t like that because we’re saying, “Sorry, it’s not up to our standards.”
“Oh, yeah, but I’m an engineer at this very large company in Italy, in Germany, in the U.S.”
We don’t care. We care about the quality of your code, because this is what defines our community. That means we have a lot of people who contribute from very different backgrounds, and they’re very introverted, sure. But that’s okay, right?
So one of the legends of the community is, of course, Linus Torvalds, who created Linux and is a longtime maintainer of the Linux kernel. As the legend goes, he can be pretty harsh in this meritocratic process of reviewing the code and saying it’s not good enough. Can you just speak to the legend of Linus Torvalds?
Linus is one of a kind, right? I would even go so far as to say that what he did with Git is more interesting than what he did on the Linux kernel. He’s very harsh, but what people don’t see is that usually, when he’s harsh, it’s toward people who are maintainers of part of the kernel. They know him, right? So he’s not that harsh with everyone.
The thing is, what he created in his room is basically powering every server online, right? Even at Microsoft’s cloud, called Azure, I’m quite sure 70% or 80% of the servers are running Linux. All your Android phones are running Linux. What he did with the power of open source is amazing.
The quality of the Linux kernel is very high, and yes, it’s difficult, but we cannot compromise on that. We cannot compromise on quality, because in the end—and you have to understand this—the core community of VLC is 5 people. The core community of FFmpeg is 10 to 15 people, and we are the ones who are going to maintain your code.
If you have 1,000 contributors over the timeline and just 10 stay, it’s a 1% chance that someone comes and stays. One percent. People will change jobs, change wives, have children, have accidents in life. They’re going to change jobs, whatever. They’re most likely not going to come back.
We are the ones who are going to maintain your code. It needs to be maintainable. It needs to be excellent. Sometimes that means you need to rework your work, because it was good, but it’s not excellent. We need excellence because we are very few people maintaining something that is critical for the whole.
But we should also mention that there is some spiciness, some harshness, to the language that’s sometimes used when you’re keeping this high bar of excellence. Is there something to say to that?
It’s true, right? It’s also the fact that, for example, what we’re doing is low-level.
It's extremely technical. You get into this community, and the tone gets very much like a subculture. People who arrive from the outside are basically not known to the subculture. Most of those people around FFmpeg and VLC—we do VideoLAN DevDays, or VDD, every year—are so much fun in real life, and they love it. But it's true that when you're online, sometimes you don't realize how it comes across. But that's okay.
It's a culture. You get this in gaming culture. The way people communicate can be pretty harsh and intense, and everyone understands that the way you show love and respect just looks different in different communities. Sometimes it depends. If it's a book club, usually people are going to be much sweeter. If it's an open-source project that's very high-stakes and used by millions of people—
But it's very often insults that you see, for example, in gaming, right? And so Linus' tone is a bit unusual even for the open-source community. It's more harsh on the results, saying, "No, this is not good. This is crap." Those are the types of things that you will see.
Try not to make it about the person; make it about the code.
Yes.
It's very matter-of-fact, and I think you've got to look at it in terms of the fact that FFmpeg is developed almost entirely by volunteers. You've got to imagine that someone's done a hard day's work at their day job, and they come home. Terseness might be a thing, and that's not something to take personally.
You're tired, you're busy, but you still care about this open-source stuff. You may not be able to explain and handhold someone through every subtle detail.
Also, you have to realize that most people don't speak English as their native language. This is especially true for open-source projects like FFmpeg and VLC, which are mostly centered out of Europe. Sometimes people from the US are not very happy about the tone, but most of the time, they also don't know better. English is a difficult language. There are so many subtleties in tone and so on that you don't have if you're not a native speaker. So often it's also difficult in those types of communities because of different cultures and languages.
So as the legend goes, JB, you repeatedly turned down millions of dollars to keep VLC open-source and free for everyone, without ads. Take me through the reasoning behind that decision to leave millions of dollars on the table.
Yeah, that's almost a meme, right, on Reddit or—
There literally is a meme on Reddit.
9GAG, yeah. See, there's—
You looking like a wizard in the VLC hat on Reddit: "This is JB, the creator of VLC media player. He refused tens of millions of dollars in order to keep VLC ad-free. Thanks, Jean-Baptiste Kempf. You can even summon him on Reddit."
Yeah. Usually, if you see it, people tag me, and then there's me saying, "Good morning." I got 24,000 upvotes, which is great. My karma on Reddit is amazing, at least on that account.
The question needs to be answered first: What is the story about VLC? Yes, it's true. I refused dozens of millions of dollars several times. I could be a multimillionaire and be somewhere on the beach, but I did not do it because I thought it was not moral and it was not the right thing to do. It's very important for me to feel that I work for the greater good, that I work for people. I don't want it to be just for myself.
But the reason is also that I did not feel completely legitimate to do that. Let me explain why. The VLC story is a very weird story.
In France, we have universities, and we have a type of top college. Those top schools of excellence are engineering schools, business schools, and basically law and medical schools. But they're outside of the university system, and in order to enter them, you spend 2 years working like crazy on math and physics to enter the best engineering schools.
One of the schools is called École Centrale Paris. It has changed its name since, but it was called École Centrale Paris. Because it was Centrale, they had to move it after World War II because it was too small, and they wanted to move it to the center of France, to a place called Clermont-Ferrand.
The alumni decided that this was not okay. It was the school that Eiffel attended, the one who built the Eiffel Tower. So they said, "No, no, we are an amazing, great school. We cannot do that." They bought a piece of land south of Paris, very near Paris, and it became a campus managed by a nonprofit organization of the alumni.
Because of that, everything on the campus was managed by the students. The university did nothing. The radio, the TV station, the supermarket, the library, defining who was going into which rooms—everything was managed by the students.
That's amazing. That's an amazing experiment, that it all didn't go to hell quickly. It somehow flourished.
It worked great, and I learned so much in my life doing those side activities. You're 22, and you need to run your campus, or else you don't have electricity. So you care about that.
Anyway, in the '80s, they did a full experiment of deploying a network, mostly sponsored by IBM and 3Com, which was a Token Ring network. Token Ring is something that probably almost no one knows about anymore. It's a networking technology where you don't have routers. Everyone is linked in a ring, and when you want to send a message, you talk to your neighbor, who's going to pass the message to the next one, who passes it to the next one, in terms of a ring.
The issue with Token Ring is, of course, that it's very slow, because every computer on the network needs to open the message and see if it's okay—"Is it for me? No, it's not"—and then send it back, like a token traveling around the ring. In the '80s, you were doing some Telnet and sending emails at university. That was okay.
But then the '90s started, and video games started. When you have high latency in video games, basically, you die. So in 1994 and 1995, around the time Doom and Duke Nukem were coming out, they wanted a faster network.
The students went to the university and said, "You know what? We want a faster network. We need to work, and also play video games." The university told them, basically, "I'm sorry, we cannot help you. You understand that the campus is not ours. You manage it, so do something. You should go and see some partners of this university."
They went and saw the CIO of Bouygues, a large French company that was doing television in France. He said, "Well, you know what? The future of video is satellite." Today, we know it's not, but at least it was a good idea.
In 1995, with the first satellite dish, he said that instead of having 1 satellite dish and a large decoder for each of the 1,500 students, they could build an enormous dish, use only 1 decoder, and send the video directly over the network. That required a very fast network. Today, it's obvious, but at the time it was the first attempt to do video streaming.
So they built this project, which was called Network 2000. Of course, we were in the '90s, and everything futuristic was called 2000.
Yeah, 2000.
They did the Network 2000 project. It was completely hacked together. It crashed after 45 seconds, but that was okay because the demo was 40 seconds. It leaked memory, but that was okay. They put 64 megabytes of RAM in it instead of the 8 or 16 megabytes you had, and the demo should have stopped there.
That was the Network 2000 project by the students.
What was the format of the video that they had to work with?
MPEG-2, because satellite used MPEG-2 TS for transport, MPEG-2 video, and MPEG-2 audio at that time.
The project should have stopped there. Everyone was happy. They had an amazing ATM network at 155 megabits per second. They had probably one of the best networks in Europe at that time, and they stopped the project.
Six months or a year later, 2 students arrived and said, "Well, you know what? Maybe other people care about video streamed on a local network." They created the VideoLAN project. One of them was called Christophe Massiot, a good friend of both Kieran and me, and they started the project.
It wasn't even open-source yet, and they spent around 3 years getting the school to agree to make it open-source. The university wanted to get some money because, due to the intellectual property and copyright of the students, it wanted to monetize these MPEG-2 decoders.
Just to be clear, what was the main application? Streaming on a local network?
It was streaming on a local network.
By the way, that's just to state the obvious: This is before YouTube. This is before—
10 years before YouTube. You had a Pentium 60 or 75. The main machine was a 486DX at 33 megahertz.
Bear in mind, television was the main form of video at the time. You could get new channels. In the '90s, having even 1 new channel, when you grew up with 4 channels, was a big deal. Having a 5th or 6th channel was a big deal, and having this satellite service with dozens, even hundreds, of channels was so groundbreaking.
Especially because this was a university where you had a ton of different nationalities. There were a lot of people who wanted different channels. In the end, they had several dishes on different types of satellites, because, for example, a lot of people were coming from the Maghreb or the Middle East, so they went to different types of satellites.
Anyway, the solution worked great, and they started the VideoLAN project.
The VideoLAN project has several solutions, and some are completely crazy, like how to create multicast on a unicast network, but let's not get into that. It's too complex. But the VideoLAN client part is what became VLC. Actually, they basically strong-armed the university to force it to open-source it, because the university did not understand that. In 2001, it's still early, but basically, yes, the university agreed in early 2001 to make it open source. I joined the project in 2003 because that's when I joined the university.
So the first thing is, I'm not the one who created VLC, because actually no one did, right? It just naturally emerged from the VideoLAN project. So I arrive in 2003, and then I created the open-source nonprofit organization called VideoLAN. I took everything out of the university to create a nonprofit project and something sustainable.
It's true that I spent more time than anyone on VLC and VideoLAN. That is for sure. But it's a continuity of a previous project, VideoLAN, the student project, which is a continuity of the Network 2000 project, which is a continuity of that and that.
It just kind of naturally emerged from the VideoLAN project. And we should mention that—again, you said it just, but to make it clear—VideoLAN, as it was at the time, was a set of technologies around video, and the VLC, what you called the client, that's the thing that most normies—
That's correct.
—think of as the thing that pops up when you click on a video and play it.
I'm sure there were moments along the way when you were thinking, "What is the future of this from an open-source perspective?" As the internet is blowing up, there are companies—I mean, for people who don't remember, there were companies making huge amounts of money.
I can tell you that in 2005, the project should have died, and I made it continue. At some point, we were only 2 active developers. I thought it was great technology and was useful, and it would be useful, and I made that my life and my time. I made it grow from a few hundred thousand users to millions of users, to what we have now, which is probably billions of versions of VLC around the world, used everywhere.
So that's a bit the story of VLC. There are a ton of very funny stories around that. Many people from around the world work on it, like you said, in Syria or in the middle of nowhere in India.
Along the way, I got several offers, which were either to bundle toolbars—you remember those horrible toolbars, which were basically spyware—or change your web browser or your search engine, or even put advertisements inside VLC. And I didn't like that, right? I'm not against money. I'm very happy to make money. I created several startups, and one that I hope is going to work very well.
It's the fact that I believe that you need to make money ethically. There is a right way of doing that, and doing sneaky advertising or stealing data is not the correct way, right? For example, if Netflix arrived at some point and said, "Well, we want to put Netflix inside VLC," probably the story would have been different, right? But they didn't. The only people who came to us were shady ad companies.
If I do that, I would have a ton of money, right? And then 3 years later, the project is gone, right? Someone forks it and something else happens.
So it's not even necessarily ads or any of that; it's the shadiness. So you had a good radar, a good threshold of, like, "No, this compromises the spirit of what this is supposed to represent."
But it's for me, right? I'm very selfishly like, "I need to go to bed at night and be happy about what I've done." Maybe it's my upbringing, maybe it's my parents' fault or whatever, but I believe there is right and wrong. This was the right decision at the time. It still is. I want to be proud of what I've been doing. And if I had sold out, I would have betrayed so many other people who work here.
Yeah, well, I should say that me and most of the internet thank you for that decision. It's inspiring for others, I think, who are pushing the open-source movement forward—that it's okay to make these kinds of huge sacrifices if you believe it's right. And I think in that case it was right, and it was the reason that VLC became as successful as it was, because it's an embodiment, a symbol of freedom and what the open-source community can create.
Yeah, and it's a service for so many people around the world, and this is important.
We should emphasize that in the 2000s, it was really normal to download a program and have it secretly install some spyware. It was buried in very faint text or in the license text box that nobody reads, at the bottom: "Oh, I will be installing this toolbar and changing all these things." It was very common to install a program to do something at the time.
To put yourself in the mind of a developer at that time, I think it's very easy for everybody listening to this to convince yourself to take a few thousand dollars to do it. To say no to much more money takes guts and takes vision.
The last offer I had was obscene, and they said, "Yeah, but imagine with all that money you could build something new, open source." The mind trick was difficult. But for me it was just like, "No, this doesn't work like that. This is not the right thing, so I don't do it." Again, it's not that I don't like money or whatever. It just wasn't right.
Well, once again, thank you from me and from the rest of the internet. Let me talk a little bit more about the open-source movement, about the fact that, as you say over and over and over and over, FFmpeg and many open-source projects are built by volunteers.
There's been a bit of drama recently, Kieran, on the interwebs, on Twitter. You have a spicy style on Twitter that I think articulates and celebrates all the incredible developers and development and the code, especially assembly, that's involved in building some of these codecs and building some of this incredible technology. But that brings us to a bit of a debacle that happened. Tell me the full saga of what happened with the Google security engineers.
Just to be clear, Google is one of the biggest supporters of open source out there. They have been for a long time. It's just that I think some things went a bit overboard this time.
FFmpeg itself—and this is not a secret; it's on the homepage—processes untrusted data. There can be security issues when you parse untrusted data. That's very normal. But recently, what changed was that Google started using AI to create security reports on an open-source project, FFmpeg. Volunteers had to deal with that. They did. Google provided very limited funding, and they even went to the media first, announcing how good their AI was before the issues could be fixed.
And this is in the public forum.
Yeah, this is all public.
So reporting an issue, using AI to find an issue in the code—which is a security vulnerability—and then reporting that publicly before you're able to fix it?
Yeah. It's announcing how good their AI is. They provided a standard 90-day industry deadline without really understanding the nature of volunteer-driven development. In addition, this vulnerability was on an obscure 1990s game codec.
Let's look at it from their standpoint to begin with. Can you steer me in their case?
Yeah, sure. They have substantial resources working on the security of open-source projects that are ubiquitous, and they've used a lot of compute to do that, as well as very expensive and very capable security researchers. Their viewpoint is that they are contributing by doing that.
But I think that's where opinions differ. It opened up a lot of interesting fissures, I would say. It does seem that there's a portion of the security community who look at themselves a bit like building architects who never have to go to the site. Going to the site is something that is a little bit beneath them—the actual day-to-day construction. They're there to do their security things, and it's someone else's problem.
The security industry also has a very aggressive tone. The language they use is extremely aggressive. They use very strong language like, "You will get popped." To Joe Public, "get popped" means something quite bad. For them, it means to get hacked.
The way I would look at it personally is a little bit like the padlock on your home. The padlock on your home, or the lock on your home, is there to protect against the capabilities it's there to protect against. It's not there to protect nuclear secrets. It's not there to protect Fort Knox.
It could be looked at as them using AI at a level of scale to go and pick those locks and then say, "Hey, your lock's not secure. You need to deal with this." Whereas, actually, they're the ones with the resources to be able to fix this. But that seems not to be something they'll contribute to, either in terms of patches or financially.
The scale of AI is the issue. The bug reports are very wordy. It's almost a denial of service by AI-generated bug reports on very niche codecs. The other issue the security community has is that everything is marked high priority: "This is the most important thing in the world, and you need to deal with this. High, high, high, vulnerable, scary, scary, scary," on a game codec used on one disk in 1993.
And that's where the dichotomy lies. Going around telling everyone that their padlock's not safe—well, that's a hobby project of somebody.
The safety of that codec is commensurate with what that person thinks. It’s their hobby. It’s good that they’re analyzing its security, but it doesn’t need a big, scary warning saying, “This is a critical vulnerability.”
We may also have recently seen another quote-unquote vulnerability. It wasn’t at Google in this case, but a filter could overflow and have an integer overflow, and one of your pixels could be the wrong color. This was marked high—7.5 severity—in red.
At some point, the security industry needs to realize that you can’t keep crying wolf like this, because this just leads to people putting password stickers on their PCs. You can’t just keep crying wolf every day. I appreciate that their modus operandi is to create as much scare and fear as possible, but from Google’s standpoint, at the end of the day, they need to contribute either financially or with patches.
Google uses FFmpeg at a scale that you or I probably couldn’t even contemplate—millions of CPU cores. Yes, they contribute in areas mostly regarding their own products, such as VP9 and AV1. But in a wider sense, there’s a disproportionate level of contribution. Yes, they fund students. Yes, they fund Summer of Code.
Alex Strange is a former FFmpeg developer, I think, posting in a personal capacity.
So he posted about security engineers on Hacker News. His post reads:
“The problem with security reports in general is security people are rampant self-promoters (Linus once called them something worse). Imagine you’re a humble volunteer open-source developer. If a security researcher finds a bug in your code, they’re going to make up a cute name for it, start a website with a logo. Google is going to give them a million-dollar bounty. They’re going to go to DEF CON and get a prize, and I assume go to some kind of secret security people orgy where everyone is dressed like they’re in The Matrix. Nobody is going to do any of this for you when you fix it.”
That’s basically commenting on the misaligned incentives for the different people involved.
The problem here is the disproportion of means for discovering an issue compared to patching it, right? This is the biggest issue. After that debacle, Google made some changes.
They are now starting to send patches.
They also now have reward pools for fixing issues. So it has changed a bit because of that debacle. It’s good, right? But we’ve seen this with Google, and we’ve seen other large companies saying, “You need to fix this bug because it’s critical in our product.”
Can you explain the XZ fiasco? The FFmpeg tweet reads:
“The XZ fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion-dollar corporations expect free and urgent support from volunteers. Microsoft Teams posted on a bug tracker full of volunteers that their issue is high priority. After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead. This is unacceptable. We didn’t make it up. This is what Microsoft Teams actually did.”
Then you give the image and the details and all that kind of stuff, showing that these trillion-dollar companies are not giving much money or much support.
They think an open-source project is a traditional vendor with an SLA. They think a public bug tracker is actually a third-party vendor’s Jira, where you can do all of these things. It’s not. It is there to report bugs.
I think the thing that made this particularly heinous was the name-dropping of Microsoft—the name-dropping that this is a visible product. If this was just a general bug report, I think that would have made it a lot better.
Yeah, so they literally said, “This is a big deal because a lot of people are using it at Microsoft.” I wonder what happens psychologically. I think what happens in these companies—maybe you can correct me—is that they just think of FFmpeg as a vendor that Microsoft surely is paying a huge amount of money to.
They kind of assume that in their interaction, and nobody anywhere on the stack is going, “Wait a minute. Shouldn’t we be giving millions of dollars to FFmpeg?”
This is a very big problem in large companies. We’re talking about some companies, but it’s the same everywhere. When we talked to that person, he was just a manager on one project in Microsoft Teams. He had never really discussed things with the open-source community. He had no idea.
Usually, there are what we call OSPOs—open-source program offices—in those types of companies, and they’re the ones who are supposed to discuss things with open-source vendors or open-source communities. But they often don’t explain that correctly internally.
We’re not your supplier. If you want me to be your supplier, I’m very happy. I will send you a contract and SLAs. I’ve created 5 companies that are doing that around open-source projects, so that’s okay.
We should say that some of the spicy tweets that Kieran, you’re behind, and some of the debacle produced results.
Yes.
Positive results.
Donations have increased substantially. They’re still not enough to cover even a single full-time developer, but both on an awareness level and a technical level, there’s substantially more awareness of the importance of FFmpeg as a result of X and what’s happened.
I can say it served its purpose. People realize the level of importance FFmpeg has.
And on VideoLAN, it’s the same, right? For example, for more than a year, we couldn’t update VLC on Android because of a bug on the Google Play Store. The only way we got someone to answer was to put out a very spicy, as you say, tweet saying that we were going to stop distributing VLC for Android. We have around 100 million people using that, and then someone from Android actually came and discussed it with us.
We had the same issue with Microsoft, saying that we were going to stop distributing VLC on the Windows Store. Unfortunately, we’re so small that the only strong power we have to solve those issues is blaming them on social networks, because it snowballs and now they listen to us.
Large companies often have difficulty talking to us. VLC is probably one of the top 10 pieces of software used on Windows. I’m not part of Microsoft’s ISV programs. I don’t have a point of contact at Microsoft, while I’m sure any other software—Adobe, Spotify—has a point of contact.
I don’t have that. Raising awareness works. It’s sometimes very spicy, with a lot of drama. X and Twitter are okay for that, but it’s efficient.
Everybody listening to this should go follow FFmpeg on Twitter, on X, and follow VideoLAN on Twitter, on X. Go donate to FFmpeg.
And thank you, Lex. Over the years, you’ve been a supporter of FFmpeg and VideoLAN on X, giving us shout-outs and appreciating what we do.
FFmpeg for life.
For example, Tim Sweeney, John Carmack, and a few others—very high-level people—have raised awareness of our X accounts, and that helped a lot as well.
Karpathy as well.
Karpathy, yes.
Karpathy as well, yeah. I mean, also, outside of the fact that so many people use it and it’s so impactful on the world, it’s also a great representation of a great open-source project: the value of assembly and C, and making sure that you take programming seriously for real-world systems.
Michael Niedermayer
It’s not just that. We’ll talk about assembly later, I’m sure, because that’s a whole topic in itself. But it’s also celebrating people like Andreas Rheinhardt, who does maintenance. He is, I believe, unpaid, as a volunteer. He’s doing massive refactorings.
Andreas Rheinhardt and Anton Khirnov are rewriting ffmpeg.c with threading. Celebrating those guys means celebrating the untold labor that’s gone into this, which actually doesn’t change anything from the user’s standpoint. The files are exactly the same, but wow—the airplane has been rebuilt while it’s in the air.
Christian Garcia said, “As a teenager running this account,” referring to the FFmpeg account, and you responded, “Teenagers have written more assembly in FFmpeg than Google engineers.” But you’re also pointing out that there are a lot of incredible contributors who are teenagers.
Like JB said, we don’t care who you are, where you’re from, or what you do. Teenagers have written thousands of lines of assembly over the years. I want to give a shout-out to Daniel Kang back in the day.
We’re also highlighting the work of people like Ruikai Peng. He was 16, and some of his first contributions to FFmpeg involved actually finding issues and fixing them, putting some of these quote-unquote security researchers to shame by doing so at 16.
There are no barriers. There’s no barrier requiring you to study at college under a particular person and understand these things. You can learn C, and let’s be honest, it’s from the K&R book. Learn C. You can learn assembly. We’ll talk about that maybe a bit later. You can contribute to world-class technologies.
In VLC, one of the oldest contributors, called Felix, is the one doing everything on Mac and iOS. He started working on VLC when he was 16.
We had a guy called Edward Wong, who used to be a Google Summer of Code student and stayed for 3 years at VideoLAN. He was 14, right? Google Summer of Code and Google Code-in were programs where we basically had students or high schoolers who wrote a ton of assembly for x264, VLC, and FFmpeg. Everyone can contribute.
Michael Niedermayer
And he also did a good job because he didn't play the alarmist CVE hype. He didn't create a CVE—which is like a public disclosure of a security issue—and do these big, scary red 7.5 high-priority ratings. He just fixed an issue in Git after 3 days. He didn't need to go and play a big security drama about it.
I think I posted, “The kids are all right.” Whereas there's a portion of the security community—as Alex said, I'm not saying all security people do this—that likes to hype itself up by creating drama. They would have happily raised, “This is a high-priority CVE, 8.0,” or whatever, on an issue that was actually in Git. It wasn't even in a release; it was in development, and 3 days later it was fixed.
Well, I just want to put a little bit of love out there. Much love and respect to Google engineers. Like you said, they're some of the best software engineers in the world, and they do contribute a lot, even on the security front.
I'm also a big fan of Theo. Much love to Theo. He was part of this debacle and drama a little bit. I think when you just zoom out on the grand arc of human history, the drama contributed positively to everybody involved. Donations went up. It brought more attention to the topic and allowed everybody to bicker in a way that ultimately got them to figure out what FFmpeg is all about.
So the way we looked at this is like it's a rap battle at the end of the day. No, but it is. We say stuff, but we can leave it on X. X is a perfect place for an international rap battle. You say stuff. I say stuff about your mama, but it doesn't mean I have an actual personal issue with her. That's what it looks like.
The Theo situation, JB can maybe expand, went a little bit too far, and there was a little bit of an issue. But it's just a bit of fun. It's a bit of a rap battle. It's a bit of WWE. Everyone's having a bit of fun on X. It doesn't need to be taken seriously.
The teenagers thing—that guy was a Google employee saying, “Hey, there are other ways to run an open-source business.” Just go and—oh, man—have a bit of fun. That's the point of this account. Furthermore, if you can teach people about the ways of open-source projects, assembly, et cetera, by doing that, I think there's a lot to be offered here.
It's not dunking on people for dunking's sake. It's showing the story that I think X learned: these are not big corporate open-source projects. This is not Kubernetes, where there are hundreds, maybe thousands, of people paid to develop this stuff. These are just people in their basements, in their spare time.
If you can address that topic in a fun and entertaining way, I think that's the good thing. That's the value of X and the reach we have.
Michael Niedermayer
And to be honest, even at Google, Google is one entity, but so many different people. There are a ton of Google engineers we work with all the time, and even Google—from YouTube to Chrome to Chrome Media to the rest of Google—those are very different types of entities. But what we do is efficient.
For Theo, it went a bit too far. I calmed everyone down. I had him on the phone, and we said, “Okay, this goes too far,” and so on. But in the end, it's a rap battle, and it's positive for the project.
The awareness we have of open source—and I mean true open source from communities—has increased dramatically in the last 2 years, and this is useful.
What do you think motivates all the incredible contributors we've been talking about? What's the engine? It's so interesting to see. Like you said, they're sitting in the basement. What's the driver? What's the engine there?
Michael Niedermayer
There are many drivers, but weirdly, the main one is that what we do in multimedia plays videos, and video is cool.
For example, we have so many people in the community who arrive because they loved watching anime. This is the advice I give when people ask me, “What should I work on in open source? How do I start?” My answer is always the same: work on something you love. I'm working on VLC because I love movies, and I love watching the same movies over and over, even if my wife hates me when I do that.
It's interesting because it's a topic that you like. That's the first thing that usually brings people to VLC and FFmpeg. The second thing is that, technically, because we search for excellence, this is the best school ever.
If you're good at C and FFmpeg, and if you know how to write assembly, I assure you you're going to be one of the best programmers ever, even if you're working on writing TypeScript. This is the most amazing thing to do. You have to get reviews from some of the most seasoned programmers ever, who are going to look at every part of your code and tell you why it's not great. It's like we are the best teachers you've ever had in programming.
Andrew Kelley started Zig. He was an FFmpeg developer and started Zig after his FFmpeg school. I mean, it’s the place to learn so many aspects of programming in the real world, in something used by billions of people.
You have nowhere to hide. You have to be open and honest about your flaws and how you can learn and be better.
What's also interesting in multimedia is that you have 16 milliseconds to display a frame. It's not like a game engine, where you can basically slow down and wait a frame. You need to be good. There is no choice; otherwise, you don't have your video.
Because of how codecs work, if you miss a frame, you're going to destroy the look of the video. You need to be good. You need to be perfect to get the right result.
It's not just pure programming in the mathematical sense. A lot of people don't understand that, in order to program correctly in the open-source multimedia community, you need to understand how computers work. When you write assembly, you need to understand CPU pipelining. You need to understand how SIMD works and how the ALU works. You need to understand how I/O works.
This is what I think is missing for a lot of engineers and software engineers today: understanding what we call computer architecture. Seriously, some of the debates are like, “Should we use this assembly instruction or this one?” People say, “Well, no, it's going to be like 3 cycles on this type of CPU and this one,” and that has a massive impact on the output.
We should expand. FFmpeg is probably one of the biggest CPU users in the world. It's probably running, as we speak, on easily 100 million—on the order of 100 million, maybe even a billion—CPUs. So every instruction matters. The impact, at least in terms of CPU, is massive for everything that we do.
So first you come because it's an interesting subject, then you stay because it's excellent, and in the end you're very proud of it because it's in the hands of everyone.
So many people say, “I'm working for whatever consulting company, and I'm doing some portal to download invoices for your PG&E.” Wow, great. So many jobs are like that. You're not going to tell that to your grandma. But if you go to your grandma and say, “I do this so that you can play video on your laptop,” she understands.
This is very important because you're working on VLC, FFmpeg, or H.264. It's in the hands of hundreds of millions of people, and you have an impact. You can be proud of yourself. In addition to making a great résumé, all those things are why people contribute.
Yeah, those are side effects. My favorite quote on this topic is from John Collison. He said, “The world is a museum of passion projects.”
Everything out there is a passion project. In open-source multimedia and open source in general, you can do that so much faster. There's such a faster network effect. I can open a café, and that can be my passion project, but I have to get building codes, build a building, find a location, and do all sorts of things.
In the software world, that passion project can move quickly. It can be amplified by the network effect, and that amplification can be more than the sum of the parts. You can find people interested in extremely obscure things, have a network effect, and make something that is truly amazing.
On that topic of passion projects, Tim Sweeney actually said in a reply to a tweet complimenting JB, quote, “Many things in the world only happen because an awesome person decides to do it. This is the case with VLC.”
That speaks to something interesting to me. It does seem that a small number of people, sometimes 1 person, can create something incredible in the software world. You said this over and over and over. I think JavaScript is an incredible thing created initially by a single person. Some programming languages, like Python, C, and Java—just 1 person has this vision and this design and brings it, sometimes over a weekend. That's the initial spark.
Yes, Linus built Git in 2 weeks. Wow.
It changed the world, Git. I mean, it really changed the world.
Linus's passion project: “Hey, I'm uploading this tarball to an FTP. Deal with it.”
But for me, it's not just in software. I believe in individuals who are going to change the world, and it's with a good vision. “I want to do that. It is useful. It will be useful.”
Whether it’s going to build trains, cars, rockets, or something else, I believe people who believe in themselves and have a vision can have a huge impact on humanity.
Let’s actually zoom out before we zoom back in. We’ll keep going up and down the stack. We’ve been talking back and forth about VLC and FFmpeg. Kieran, you said that FFmpeg and VideoLAN, VLC, coexist, and that there’s no central point of importance. It’s what you call a binary star system: they succeed because of each other. Can you explain the difference and how they interact? Are they competitors?
I don’t think they’re competitors. The simple answer—the short answer, before I go into detail—is that VLC is to FFmpeg as Android is to Linux. They depend on each other, but they coexist because of each other. They are a binary star system, as I used the analogy.
By the way, I feel horrible that I only recently learned that Alpha Centauri, the closest star system to us, is a triple star system.
When you start doing the physics, it’s a nightmare, right?
Hence the three-body problem. But anyway.
A lot of FFmpeg pipelines involve the x264 project, which is a VideoLAN project. I would put a finger in the air and say 80-plus percent of those pipelines are dependent on a VideoLAN project. VLC, obviously, as we’ve discussed, is a VideoLAN project. It uses FFmpeg, giving it reach and exposure to weird files, and historically used some donation money to fund FFmpeg development. We’ll talk a bit about some of the reverse engineering later.
So it’s a binary star system. They work and feed off each other. Many of the developers are shared. There’s no central location; it’s a virtuous cycle of working together.
We should mention that x264 is the encoder for the H.264 video standard. H.264 is the standard; x264—
—is the open-source implementation of the standard—
—that’s used by basically everybody.
For everything. That is the main driver of this. When you think of an MP4 file that has the H.264 codec in it—
If it came from a software environment, like a data center or somewhere, the chances are it was created with x264.
That’s a VideoLAN project. So, in the VideoLAN graphic, it sits in the VideoLAN world.
And VideoLAN has a bunch of stuff in it. Go to the VideoLAN website; there are a bunch of icons.
If you look, there are so many libraries, right?
libdvdcss, libdvdnav, libdvdpsi, libvlc, of course, VLC for Unity, libbluray—there are many more.
There are so many more, right? Lately, the dav1d project that we might talk about is the last project from VideoLAN. It’s everywhere. We also have libspatialaudio, which we recently announced. We have—
checkasm.
We have checkasm, which is an insane project, but amazing. x264 is one of those VideoLAN projects. In my opinion, x264 was the most amazing encoder ever designed, and this helped the adoption of FFmpeg.
A lot of people and large companies went through FFmpeg because they wanted to use x264, and x264 increased the popularity of FFmpeg. VLC also had its popularity because it played so many files that were created by FFmpeg. There are many projects that are intertwined and work together.
Unfortunately, there’s a thing on X where VLC is mentioned and people say, “A quick reminder that it’s FFmpeg inside doing the actual work.” As I said, that’s not the case. We work together.
To give you an idea, when I compiled VLC for Windows, I compiled around 16 million lines of code. 1 million of those are inside the VLC repository, and FFmpeg in total is probably around 2 million. That means so many dependencies are outside.
If you also look at FFmpeg itself, FFmpeg integrates third-party libraries like x264, libopus, and so many others. We all depend on each other.
That’s why I was hoping to do this episode, because it joins FFmpeg and VLC. They’re really two of the same, as you said—a binary star system—and we’re all just orbiting it.
Can we give a shout-out to some of the people along the way? We didn’t really talk about the history of FFmpeg. Maybe you can tell me about Fabrice. Can you tell me about Michael Niedermayer and some of the key figures here?
Let’s talk about the eras of FFmpeg, because there are key eras and key people who made this possible. Fabrice Bellard created the concept, and then I would call the 2000s the Michael Niedermayer era.
Some of the key things he got done were exhaustive support for DivX and Xvid at the time, along with all sorts of weird variants of what’s known as MPEG-4 Part 2. This predates the MPEG-4 Part 10 that we’re used to. These were the video codecs of the 2000s, when there were flavor after flavor of weird decoders.
At the time, you needed a new player to play every different type of file format. There was Windows Media Player for Windows Media formats, and RealPlayer for RealMedia formats. The other key thing in FFmpeg at the time was native decoders for those formats.
I remember being a teenager and figuring out that there was one player that could play and decode all these files without requiring separate, bloated players. When you downloaded RealPlayer, there was a ton of other stuff in there, a ton of ads, and a ton of other things. Having a simple library that was fast led to that.
Then I think 2008 onward was a big change, because that’s when H.264 reached maturity. This was the beginning of high-definition video, and H.264 was the key decoder for that. I’d call that the late 2000s and the 2010s, when the big reverse engineers came along and did astonishing work.
The beginning was a single player that could play Xvid, DivX, Windows Media, and RealPlayer. That was already a massive achievement: no codec packs, no weird stuff you had to download with strange ads and spyware.
VLC 1.0 was released around that time—in 2009 or 2010—and that’s when it exploded.
Yeah, without codec packs, it just works across all these different formats.
De facto, all the codec packs are FFmpeg inside VLC, plus we have other modules for all types of codecs.
Back at the time, in the 2000s, there were weird codec packs, with DLLs coming from this place and DLLs coming from that place.
With a lot of spyware.
With spyware and all that. It wasn’t reliable; you didn’t know what you were getting. Having a single open-source player—or a single open-source playback module—that could do this was a huge improvement.
The thing to emphasize is that the task Michael took on in the 2000s was Sisyphean. The number of edge cases was beyond comprehension. You could have a Chinese CCTV system that implemented one weird variant of MPEG-4 Part 2, known as MPEG-4 ASP, and you had to fix that without breaking everybody else—times a million.
So you said that’s where a lot of the reverse engineering was happening.
It started in the 2000s with the Windows Media stuff, because that was proprietary. It started with RealMedia, with Benjamin Larsson.
Kostya Shishkov.
Kostya Shishkov—that era. Those were the key groundwork. Then, in the 2010s, it was the Paul Mahol and Kostya era, building and implementing some of the most difficult codecs.
JB maybe can talk about GoToMeeting 4 and GoToMeeting 5—
What’s GoToMeeting?
Let’s talk about this amazing Ukrainian guy called Kostya, who at that time was living in Germany and who was in love with Sweden. A lot of the people in the community are very clever, but he’s one of those who are borderline geniuses. He was able to reverse-engineer extremely complex codecs.
We do a bit of reverse engineering with Kieran, but clearly not at this level.
No, no, yeah.
He reverse-engineered binary blobs that were 20 megabytes.
Yeah. Just for reference, reverse-engineering a 1-megabyte binary blob is probably on the order of a month of work, and this guy was doing 20- or 30-megabyte blobs. Maybe we’ll talk about the subtleties of how you do that in a minute.
He did that for fun. GoToMeeting was a big problem with VLC because it was the number-one feature request for a long time, so I put up a bounty. At some point, the guy said, “Okay, JB, I’m going to do it.”
He did it in a matter of 2 months, and then he explained how he did it. He was just like, “I looked at the code, and this looked like DCTs that I used to see in WMV and so on.” He did that, and the funniest part is that the code he wrote is full of jokes. There are a ton of JB jokes—my name—and Kempf and Kostya jokes inside the code.
The code is beautiful.
One of the things I want to comment on is that I’ve gotten a chance to speak to some of the developers, some of the people working at the assembly-language level, and they always make everything sound like it’s kind of easy. There’s a kind of humility, because maybe the level of what’s required to do this stuff is so high that everything else seems easy. I guess that’s the lesson to take away from it.
In the community, some of the most impressive people are the ones doing reverse engineering and the ones doing the assembly folks. Both of those types of people are amazing.
x264, for example, became amazing because of a guy called Loren Merritt, who was from the University of Washington, I think.
At the time, yeah.
He made everything great and fast by doing a ton of assembly. This is like the golden era, I guess, where so many things got done.
If you look at Kostya, for example, he looked at the world as a binary specification. He didn't need documentation or anything. It was, “I have a binary, and I can figure all of this out.” He regularly used the phrase “binary specification.” “Ah, you know, it's not a problem.” He would go away, and he would come back, and he would do interesting stuff.
Can you actually speak to the details or add color and texture to what it takes to reverse-engineer a blob?
Yeah. So let's look at GoToMeeting, for example. It's a good one because I record a meeting on GoToMeeting. How do I play it back without needing this GoToMeeting player? There may not even be a player. I may need to send a recording of a meeting to someone who doesn't have a player or whatever.
First of all, there's a ton of other stuff there. There's an actual video-conferencing client. You need to go and find the actual module doing the decompression. It may be easy or it may not be easy to find. You need a way to actually dump the YUV data from the module. Often, this involves opening it in a disassembler, trying to guess where the hooks are to incorporate that module, and running that module natively to decode a sample file.
You need to figure out where this module is doing the decoding process and find a way to hook in and output the raw YUV data, because you will need that as a point of comparison for when you actually do the reverse engineering. You'll need to be bit-exact, or in some cases close to bit-exact.
Then you open up your disassembler and use a lot of intuition to figure out where the DCT is and where the entropy coding is. There is a kind of—not a rule book, but there's always a pattern of some sort. For example, with GoToMeeting, you know it will be a screen codec—there are a lot of screen-codec tools. There are also different variants. Often, I think there's GoToMeeting 4 and 5—
Well, 2 or 3, 4, I think.
2, 3, 4.
This is key, because GoToMeeting is something that not many people know anymore. You know about Zoom and Teams and so on, but fast-forward 10 or 15 years, and there's a GoToMeeting executable for 32-bit Windows. It's like, “Yeah, but I'm on Android, I'm on an iPad, or I'm somewhere else.” How are you going to do that? I'm going to be on RISC-V or ARM. Those are blocked, but there are tons of files we need to support in the future. This is why this type of work is exceptionally useful for humanity.
I just have to say, though, that reverse-engineering process is mind-blowing. It's crazy. It's kind of like—I've been reading a lot and interviewing archaeologists. You have so little signal. Over time, you get so much experience, you understand the structure of the original code, and you can start inferring the basics. But you're like—
We're like archaeologists with a little brush, trying to reconstruct the entire human civilization.
Kieran is too humble, but Kieran has done some reverse engineering also.
Of CineForm, yeah. I did that at the time, before it actually led to the open-sourcing of that work.
CineForm, nice.
So, in parallel to doing the binary side, you obviously have samples. In many cases, you don't have many samples, so you have to figure out what all the different flavors are.
CineForm, for example, is actually a collection of different approaches and toolkits within that codec, because often it grows naturally. The hard part is finding a sample that gets you somewhere to start without having to implement 10 different other things. So you start there.
Thankfully, at the time I found a sample by pure chance that had a lot of flat blocks. It was animation, so that really helped a lot because it wasn't using particularly complex coding tools. You could get somewhere and then build up and build up until you figure out, “Hey, here's a few bits here. I missed this. I missed this branch that it does.” Then you go, “Oh.”
So when we say samples, you mean sample videos, and then you're tracking and trying to infer what this codec is doing by observing the sample and then looking at the machine code—
The machine code saying, “Ah, I have a byte. This byte is 6. Take this branch.” And in a different sample, oh, it's—
That's nuts, man. That is nuts.
So you see, this is nuts. Then you go to things like GoToMeeting.
Imagine two orders of magnitude more complexity: a guy alone somewhere in Germany doing that. For a long time, you're in a black box because a decoder has so many steps: the entropy decoding, the intra prediction, the motion prediction, the IDCT, and so on. For a long time, you don't see anything, so you're debugging purely in memory.
Debugging guesswork.
You may have the buffer that the coefficients are stored in completely wrong, so you may be going down a complete rabbit hole thinking it's this, and then, oh damn, that's not it. That's something else.
And you're doing that on binaries that are tens of megabytes, millions of instructions, right?
You're stepping through the debugger one instruction at a time, going, “Hey, this instruction changes this. This does this.”
Pausing the program at the CPU level. It's—
Pausing it, yeah, at the CPU level, watching what's going on and trying to figure out—
Sometimes you need to be in a VM so you can pause the VM.
Yeah, pause the VM and dump the memory, because some of the codecs could have encryption. There could be DRM on there, so you need to dump the memory from a virtual machine.
When I joined École Centrale Paris in 2003, Jon Lech Johansen basically broke the DVD specification and created DeCSS. He showed us how he was breaking a DRM, which was FairPlay from Apple. What he did on his laptop—and I was young, I was 21—was just mind-blowing because he was basically debugging Windows inside a type of VM. I was like, wow. It's incredible, mind-blowing, and inspiring.
From your experience and from what you've seen in the community, does it get discouraging?
People help you. People send you samples. People are keen. Sometimes you don't have access to an encoder, so this is even more difficult because you just ask, and you have to ask for samples. I remember VideoLAN used to tweet for samples at one stage: “Hey, I need this obscure sample.”
For a long time I was saying, “Oh, I need this codec, and I need this codec.” If you were unlucky, you'd get nothing or you'd get one or two. Sometimes you'd find a goldmine. It was like, “Yeah, my company has 100,000 of these files because we're dependent on it for some reason.” Those are kind of the best, because then they can test bit-exactness across the huge range of coding tools.
Can you explain bit-exactness?
Bit-exactness means that most, but not all, video codecs—certainly from about the 2000s onward—have a bit-exact definition. Every implementation must produce exactly the same bits, bit for bit, in exactly the same output data from a decoder.
For a large number of samples?
Mark Thompson
For a given sample. Lex's implementation, JB's implementation, and my implementation of H.264 must match bit for bit.
That wasn't the case with MPEG-2 in the '90s—probably fair to say one of the biggest mistakes the video industry made. I think the people who were in the room in '92—I suspect both of us were in diapers—have acknowledged that. I would give a shout-out to Yuriy Reznik; he's acknowledged that that was one of the big mistakes of the era.
And you're saying the encoders needed to be able to run tests for bit-exactness. I mean, that's a nice thing to guarantee. There's a parallel sort of development here in the way a web browser works, which takes HTML and displays it, and there's no bit-exactness across the different engines.
I would point out that FFmpeg is unique in the sense that it has been a winner-takes-all scenario. Browsers are a good analogy because a browser has to parse a lot of different content and render it in a particular way, like a decoder. But there still are multiple browser engines. There's Firefox's, there's Chrome's, and there are a few Japanese ones that are pretty decent.
That hasn't been the case in multimedia in general across a wide range of codecs. FFmpeg has kind of won it all, I suppose, in a sense, because every new codec added is actually worth more than the value of that codec itself. It makes the whole thing better.
Man, this is really cool.
Telco.
Oh. And so he was more connected to companies. RealAudio, RealVideo, right? That was—
Oh, yeah.
—very important at that time.
Mark Thompson
Zencoder, Brightcove, Contex. This, man, I need to hang out with Yuriy. He's legit. And he's one of the nicest people—
Slack guy, yeah.
Mark Thompson
—ever, right? For example, for my startup that I'm doing right now called Kyber, I met Yuriy every year at the Mile High Video Conference, which is in Denver. And he gave me so many good ideas and good things. He's a really amazing person.
He tells us how—
Mark Thompson
How great it is to even know us. And then you look at that and I think it's the other way around, Yuriy.
That reminds me of something that you mentioned to me about FATE testing and the insanely rigorous process that's used to test everything that's incorporated into FFmpeg. Can you take me through the testing process?
Yeah. FFmpeg has a system called FATE, the FFmpeg Automated Testing Environment. Because FFmpeg runs on so many different operating systems and can be compiled with so many different compilers, there are a crazy number of configurations. So you can see the absurd combination of compiler variants, operating-system variants, and instruction sets. At the top, macOS has tons of different variants because it has iOS and tvOS.
Well, I'm looking at a page, fate.ffmpeg.org. Eighty-one minutes ago, 76 minutes ago, looking at the different architectures, the operating systems, the different compilers, Apple Clang versions—
Mark Thompson
The combinations are crazy.
—the combination is insane. RISC-V—
So these are all run by volunteers. The ones at the top, for example, the Macs I host in my office, host all sorts of different stuff. Other people host other things. So it's really there to make sure that, because FFmpeg does quite complex C code, you do have miscompilations. The compiler will sometimes compile C code incorrectly. For example, this happens once in a while.
Oh, there's a log of all the compilations.
Mark Thompson
Yeah, a log of all the compilations and all the tests. I think one of the other ones will show all the tests passing.
If you click, you can see all the tests back. All tests successful.
Mark Thompson
In logs/test, yeah. So you see all those tests are passing for all the different codecs and all the different filter transformations. The level of scale is quite crazy on all the combinations.
Oh, that's nuts. That's nuts.
Mark Thompson
It's not just a matrix at this point. It's like a pivot table of different combinations. And it's a key part of what we do because you may be able to test something locally and make a change, but actually that breaks GCC version 11 on Mac or something like that, and you're able to then fix that. We also have miscompilations, so the C code—sometimes the compiler can have a bug in it where it creates the wrong output, and that can have quite a big effect sometimes on a video because of the way frames have dependencies. Even a small change in the output can cascade to quite big glitches.
You see PowerPC, you see RISC-V, you see ARM.
Mark Thompson
There was PowerPC, there was RISC-V, and there was weird stuff in the past like DEC Alpha.
You see Visual Studio, different versions of Clang or GCC.
Mark Thompson
Visual Studio, Intel compiler, Apple Clang, you name it.
What are some of the pain points? Do you have emotional triggers, maybe nightmares, about a particular operating system, a particular container-codec combination?
For me, it's really easy because I have a day job. The company I started builds equipment for broadcasting sports matches between TV stadiums and studios, for example. We have to work with 10-bit video, and 10-bit video has a set of challenges because you can't process 10-bit data natively on a CPU. That means you have to put it in 16 bits, so you have 6 wasted bits. There are different packing formats to pack the data more efficiently because when you send that over a network, you need to save that 40%. For example, on PCI Express, you may only have enough bus bandwidth to do that.
Internally, we have about—I think—a 5-by-5 or 6-by-6 matrix of every single format to every single other-format conversion. Some are industry ones, and some are internal to our own hardware that we build. One of them I sent you, and they're all written in handwritten assembly. They all support different CPU generations. So this is really traumatic, handling all these different combinations times a million.
By the way, the company you're talking about is Open Broadcast Systems.
Yeah, so no relation to the free OBS streaming service. But JB and I have started companies, broadly speaking, around the FFmpeg-VLC ethos, so that's really low-level work. In most companies, this wouldn't be written in assembly. It would be accepted that C is fast. As you can see from that, C is not fast.
So here it says 62 times faster than C.
Mark Thompson
Yeah. So it's taking that ethos of doing low-level programming and real-time programming and using that for commercial applications. JB and I have started companies around that, in many cases hiring developers from the open-source community to use that ethos. That's a great example of some of the things we're doing. In most companies, it would be, say, "I'll write this in C, it's fast, and we're done," but actually you can get a lot better.
For me, some of the headaches we have are around operating systems that are difficult to support. If you look at VLC, and thanks to FATE and FFmpeg, the last version of VLC runs on Windows XP and still runs on Windows 11. We work on macOS 10.7 through the latest macOS, whatever it is—26. We work on iOS since iOS 9. We're actually on iOS 26. We support many types of Linux, BSD, and Solaris. The last version still runs on OS/2. There are maybe 10 users of OS/2 in the world, and 1 of them is maintaining VLC.
Then you realize that this very small team around VLC, using FFmpeg codecs and all the other ones, supports more operating systems than Microsoft, Google, or Apple, and those companies have an infinite amount of power and resources. The worst, for example, is iOS. In order to build on iOS 9, we need to do some very clever mixing of several versions of Apple's Xcode IDE and SDK and create a type of Frankenstein version so that we can still support iOS 9, which isn't supported at all by Apple's compiler, and still run on ARM32 on iOS 9. And you've seen on FATE that it was still supporting iOS 9, right?
My headaches are mostly related to supporting so many operating systems. It's important because we receive so many messages from people saying, "Hey, thank you. I still have my iPad 2 to watch movies," and it still works on iOS 9. It's also an impact of not forcing people to buy new hardware when it works fine if you optimize it correctly. That brings us to what we were saying about assembly. It's also fighting the fact that you need to buy something new nonstop when you could optimize more, which is a lost art.
You've got to tell me about this lost art, the carriers of the flame of assembly. What is assembly? Why is it beautiful? Why is it challenging? How does it work?
When you write assembly code, you write it using the instructions the actual processor is using directly. Most of the time, you would write in a language—let's take C as a good example—and the compiler would use that to create assembly-language and machine-code instructions for you based on your C code.
There's a specific flavor of assembly that we use in FFmpeg called SIMD: single instruction, multiple data. For example, say I want to add 5 to a number in scalar assembly, where you work on an individual element. I have the number 10, and I want to add 5. I use the add instruction, add 5 to 10, and get 15. With SIMD, I can have a whole vector of 16 different numbers. They could all be different. If I want to add 5 to that, I can run 1 instruction, and that 1 instruction sums all 16 elements.
That, as you can imagine, lends itself very well to video. Video is a pixel grid, so I can perform operations on multiple pixels at the same time. The key thing that we do differently in FFmpeg is that we don't use any abstractions, or any major abstractions, on top of that. There's a part of the world that uses what's known as intrinsics. These are C functions that behave very similarly, but not quite the same, as writing assembly by hand. The registers that data is stored in on the CPU are allocated by the compiler for you.
The key thing to understand is that when we write SIMD, we have a 10x to 50x speed improvement—not a percentage improvement. That function is 62x—
That's nuts.
—on the FFmpeg account, which, as you know, posts and tweets a lot about that to try to say, "Hey, we are doing this stuff."
You are a person who sees the beauty in assembly, but it's also extremely useful for these kinds of applications to—
—significantly outperform even C, which is crazy.
It is necessary, right? One of the projects that we need to talk about is called dav1d. dav1d is a decoder for the format that was created by the Alliance for Open Media, which is a video codec called AV1.
For people who don't know, we've been talking about H.264. AV1 is another hugely popular standard and codec that's increasingly taking over the internet.
When this format was launched, many people said, especially people from the Alliance for Open Media—which includes Google, Netflix, Amazon, and Mozilla—"This format is so complex, it must be done in hardware to decode it." I arrived with a few other people, mostly Ronald Henrik, and Martin, and we said, "We need to have an extremely good software decoder because it's going to take time to have hardware." And so we wrote this project, which is beyond insane.
We are talking about 30,000 lines of C, but 240,000 lines of handwritten assembly, right?
Handwritten assembly, 240,000 lines. That's incredible. Some of the stuff we're talking about is probably among the biggest assembly codebases.
To give you an idea, and Kieran can correct me, but I think FFmpeg has 100,000 lines of assembly for all the codecs.
For all codecs. Mm-hmm.
And just this one has 240,000. It's a VideoLAN project, of course, and it is optimized to the maximum, because the motto when we started the project was, “Every cycle matters,” right? Every cycle matters because dav1d is used in VLC and in some software AV1 playback stacks.
We are talking about probably 3 billion devices that are going to decode video nonstop, because, for example, 30% of the video from Netflix is now in AV1, and 50% of YouTube, right? And you often don't have a hardware decoder because not many devices have a hardware decoder. With dav1d, we realized that with 1 or 2 cores, you were able to decode 720p correctly. So it is literally—
Yeah, that's dav1d.
—incredible, right? That's dav1d. Look at that, Lex.
Yeah, this is another spicy tweet from you. This is what peak video codec should look like: 79.9% assembly, 19.6% C, and 0.5% other.
That's almost 80% assembly.
And what's incredible is that with those tweets, which are factual, people get crazy. They are unhappy, right? For the last 2 years, they go crazy: “No, intrinsics is fine. The compiler is...” They go, “You can optimize your compiler, auto-vectorization. It's your fault; you don't understand.” And we've tried that forever, right?
For 2 years, and 2 years later, showing hundreds of examples of handwritten assembly: “No, no, no, you're doing it wrong. The compiler can do this.”
So we should actually articulate this a little more clearly. The intuition there from the software engineering folks—there are a lot of great software engineers, great engineers, and great machine learning people—is that when you have code like... Okay, let's just take an example: C++. There's a compiler that's doing a lot of the optimization.
Yes.
And the presumption is, if you have a good enough compiler, if you continue to improve the compiler, you're going to generate code that can perform at optimal performance. You cannot possibly beat it.
Yes.
And you're consistently challenging that thought, that if you do—
By orders of magnitude—
—handcrafted assembly can outperform C.
The 2 things that they tell us are, “Yeah, but modern compilers have auto-vectorization,” right? Because SIMD that we're doing is vectorization. And it's not even close, right? It's not like 5% or 10% slower; it's multiple times slower.
So can we—I don't know if you can say something philosophically, because there are a lot of great software engineers, great engineers, and great machine learning people. Karpathy will listen to this and say, “What's the intuition he's supposed to get from this? What are we supposed to take away?”
Karpathy learned assembly because of the tweets, by the way. He went, “Oh, I think this is a movement.”
He's like, “Let me figure out what's happening here.”
No, no, he—and you know the way he documents his work and so on. Philosophically, what's important to realize is that we passed the time when hardware was going so much faster, right? We are at the end of Moore's law. We have limitations for AI and for memory.
You need to go down in the stack and optimize more to get more power from what you have, because our requests for power—CPU power, GPU power—are exploding while the hardware is not exploding in speed, right? So what people do is add more cores, right? But at some point, you can add 250 cores, right? So what we do is take every inch of the machine.
Not just that, not just that. We abuse the machine. We use the machine in ways that the creator didn't expect. Sometimes we use an instruction that's completely unrelated to what we do. We use a cryptography instruction in video processing to do nothing related.
One of the other things that we do, for example in dav1d, which is a bit crazy, is that we don't use the function-calling convention from the operating system.
We should explain that.
That is extremely complex. But basically, usually when you move from one function in code to another, there is a way to save the registers, the state of the CPU, to enter another function. And this is standard.
It's a bit complex. I would simplify this a bit. dav1d does things to abuse the calling convention. You could define the calling convention as: I've written a function and I want to call another function. How is the data shared between the functions? Because there's a convention, what's known as a calling convention, and what dav1d does, for optimal reasons, is create its own calling convention sometimes.
So if I want to call Lex Fridman's library, we've got to agree on a convention so that I can share data with you in the assembly language space. And one of the challenges in assembly is every operating system—well, not every operating system, but there are at least 4 that I can think of on x86: Linux 32-bit, Windows 32-bit, Windows 64, and Linux 64. They all have their own calling conventions.
But dav1d takes this even further. For speed reasons, it does its own calling convention within itself to bypass the rules of functions and say, “Okay, actually, I'm going to call a function this way because I know it's within my library.” Does it have to be special to every single operating system?
Well, if it's custom, no. But the challenge is, in general, yes, and in terms of each instruction set. The thing to also emphasize is that we do this on every instruction set. Every instruction set has its own handwritten assembly, which is even more crazy.
That matrix has gotten bigger in recent years because of RISC-V, because of ARM64, because of the new SVE. There's SME. x86 has AVX-512 and AVX. So we do runtime processor detection. We see what the machine FFmpeg is running on, or dav1d is running on, is capable of, because you could be on a laptop from 2008 where this isn't there. At runtime, we set function pointers accordingly, and then from then on, off you go.
Or you could be on a machine with RISC-V.
Yes. And in all that, we don't even respect the calling convention of the operating system in order to be faster, because we know that we're going to be called from within our binary, so we can share data without saving all the registers in the common way. That can lead to loading and saving registers on the L1 and L2 caches, and gets us faster.
That's why I said understanding CPU architecture—computer architecture—is key. This is also why it's handwritten. I don't know anyone; I've never heard of any other project than dav1d doing that. This is why Kieran calls it an art, right? It is an art.
I think in a mass world, there isn't something on billions of devices. I know there are some specialist industries. I know in high-frequency trading, they take this really seriously, where they're receiving feeds from a market and they need to react within X number of microseconds, and so the instructions matter.
But that's not a mass-produced thing that's on a billion devices. That's hyper-specialized, running on hyper-specialized hardware. We're running on all hardware from—
Sorry to linger on it, but that's a really counterintuitive, almost revolutionary idea here: that there's a huge amount of value to assembly. What are we supposed to take away from that?
There's a bunch of people listening to this who are basically like—sorry, for myself included—I programmed for many, many years in C and C++, going up the standards of C++, fell in love with C++, even metaprogramming and so on, and then transitioned more and more, because of machine learning, about 15 years ago to Python.
And so, for me in this Python world, JavaScript world, now vibe coding, where I'm just using natural language, sitting in my jacuzzi, drinking a drink, and just talking to the computer, why is the value to go back all the way down to the low level? What's the intuition?
Because you can get more power per dollar invested, right? And sometimes it's going to be a problem that's limited by your hardware. A good analogy is what you see in quantization in LLMs, right? People are doing, “Oh, I'm going to do that in FP8 or FP4 or some crazy things like Microsoft Phi-1.5,” because you're constrained by memory, because you're constrained by the machine you can run.
At some point we're doing real time, and I believe this is going to happen on AI inference also, where at some point you need to get faster, and you cannot always get more powerful hardware, right? So you need to analyze code and see where the mission-critical things are, where the things are that are called nonstop.
For example, dav1d is a good example. It's going to be run billions of hours per day. That makes sense. It doesn't make sense to be on the glue of FFmpeg CLI.
No. This has to do with it as well—we'll talk about it more—but your new effort, your new company, Kyber, is doing that kind of thing for ultra-low latency, so the slogan is, “Every millisecond counts.” And when you're actually extremely highly constrained in some dimension—
We are also arriving at a point where we've done so many great things, but the hardware is getting back to us, right? Because cost is increasing, because we need more power, and so you're limited by either your CPU, your RAM, or your networking, and you need to optimize. This is where value is going to be.
Especially because doing AI is going to help do the programming of business, right? The core thing that you will not be able to vibe code is optimization for the hardware, to make it as fast as possible.
There's this nice repo with the assembly lessons. First of all, do you think developers should learn how to program in assembly, and how would you go about learning it? What is asm-lessons?
I personally wasn't happy with the way assembly is taught in books and online because it's very grammar-focused. In general, you don't learn a language by learning the grammar and the structure. You learn a language by asking someone what their name is, and you start from there and solve real problems that you have when you want to communicate.
You don't learn sentence structure—this is the interrogative, this is the adverb, and so on. All the assembly books seem to be doing it like that, going through every instruction, even ones that aren't really relevant, explaining what they all do. It actually doesn't really change much.
The other problem that we have in our community is that assembly is taught hand to hand, person to person, like blacksmithing one by one. That's the only logical analogy, and that doesn't really scale online. It doesn't do other things.
I've started a set of assembly lessons in the way it's done in FFmpeg, which is a little bit different from the way assembly is used in general. The other big use case of assembly is in embedded devices, in really low-power, cheap devices, and that's completely different from what we're doing here.
I think it would be good if you could highlight the requirements, which are quite simple: high-school mathematics and C. Actually, not even C—really, it's pointers. To emphasize, yes, we've talked about how brilliant this stuff is, but high schoolers like Daniel Kang have written assembly in FFmpeg.
I think there have been contributions because of these lessons. It's really about trying to get this dying art to continue because we've shown it's possible with dav1d to produce something amazing. There's still a lot of codecs in FFmpeg that are only maybe partially assembly-optimized.
It really starts with the basics and continues by explaining a lot of the jargon and syntax. It doesn't really try to explain interrupt handlers, interrupt instructions, and all of these different jump targets. It actually makes this really vector-focused.
It describes all kinds of registers: general-purpose registers, vector registers. Really nice examples. Oh, this is cool.
It's a classic example of FFmpeg. Some of this assembly language is really beautiful, and I think it's beautiful because it's kind of like flying a Spitfire. It's really aviation at its purest, but also pushing the aircraft beyond what the designer thought was possible.
We're abusing, for example, cryptography instructions sometimes to do certain things. There's a level of beauty and art where it's really you and the processor. There's nothing in between. It's you and the joystick of the cockpit, and you move that joystick, and it's physically connected to the ailerons. You can push that plane beyond what it can normally do, and there's a level of beauty and amazingness to that.
But I don't think the sort of person-by-person assembly that someone taught me, and that I've taught multiple people, is going to work in the long run, just because of the particular flavor and the way that we do it.
I was going to say wizards handing it down. I realize I look like a wizard wearing this hat. But you're basically like the sages, the wise sages, handing down the craft. Can I ask you about LLMs? Can they help?
They had more of an understanding than I expected, but they still have limitations. I've asked them questions, and they still start making modifications. Then I go, "Is it bit-exact?" "No." "Fix it." And then they just go and do the same thing. There isn't the corpus of information, like Stack Overflow, to work on.
There is not enough data to train on, and this is the biggest issue. I started my career doing some assembly for Itanium, right? Itanium is a dead processor type, which was developed by Intel and HP a long time ago when they wanted to do 64-bit computing. They lost, and then we got AMD, who did AMD64, which became x86-64.
Itanium was extremely interesting in the sense that those were processors with a ton of computing power to do floating-point operations and FMAs, which is similar to what we need now for LLMs. You could pack 3 operations per line that could be loaded. You had an output of 6 billion operations per second, but the bus—the memory bus—only allowed 1.5, right?
Your CPU was 4 times faster, so you had to do crazy things to pack things in memory or reuse the registers. Those types of semantics couldn't be handled by any language, right? I have the Itanium programming book because Intel did amazing books, but that's exactly what Kieran says: if you don't know what you're going to do, it's impossible to read. It's a ton of jargon and so on.
Those lessons are amazing because they're targeted to a real problem, and you can do it yourself.
And people have. There are patches, and they said, "I studied your lessons, and here's my first changes."
That's amazing.
Part of the lessons is a framework called x86inc, written by Loren when he was working on x264. It allows you to do more with that, so you don't care too much about different calling conventions. We had a lot of students who gave code to x264 using that a long time ago, right?
It's really doable, and I believe it's necessary to understand assembly language, even if you don't use it much, to understand what's going on inside your computer. That will make you a better programmer. I assure you that, by doing that, you will understand some of the architecture of the memory inside your computer, right?
Understanding registers, L1, L2, L3, RAM, SSD, disk, and so on is very important because then you have a good programming culture that will make you a better programmer.
What do you think about the Rust programming language? Because that's a bit of a meme.
We have very different opinions with Kieran.
I think it's valuable what they're doing in terms of memory safety as a concept.
Can it achieve some of the speedup that assembly achieves?
Not assembly by hand, no. I think that's a given. C, potentially, but I see it as having a very big Esperanto vibe. It's like, "We're going to solve this, and we're doing this in a particular way."
Meaning it's a bit too utopian?
There's a lot of focus on the self-importance rather than solving real-world problems. It reminds me of the Sinclair C5. Sir Clive Sinclair of Sinclair Research built a car, and he said, "Oh, everyone will be traveling around in one of these electric cars."
Rust reminds me of that. I think the community doesn't quite understand that, in order to get people to move, you have to build something that's as good as, if not better than, what you have now. Yes, people are doing Rust rewrites, but if they only do 85 or 90% of the feature set of what we need, like with things such as coreutils, that last 1% takes 99% of the time.
To use Elon's famous quote, "Prototypes are easy." This kind of stuff is easy. But to get a real electric car, you have to make a car as good as, if not better than, what we have now, and Rust isn't at that stage yet.
I don't think anyone would object to seeing Rust code in FFmpeg, but it needs to work as well and support the same unit testing as everything else. It needs to be flawless. It can't just randomly break. They can't just randomly break the ABI when they want to.
It needs to have more. I think it still has only 1 compiler implementation. It's got to be as good as, if not better, and saying, "Here's my utopia of memory safety," isn't enough, even though we probably all agree that that's the goal.
I've done a ton of Rust, and the 2 major topics I had were adding Rust modules inside VLC. One of the reasons VLC got popular, and one of the main architectural decisions, is that VLC is a very small core with a ton of modules, right?
You can write modules in C, C++, Objective-C, and anything that is basically interoperable with C. We did some Rust modules, so I have experience with that, and I wrote some of it. Also, my new startup, Kyber, is an open-source project mainly done in Rust.
Rust is extremely good in the sense that it's a better C++ that cares about memory and allows you to do things with memory ownership that no one else can do so far. However, it's great when you start a new project from scratch and do everything in Rust. But it's not very good when you interoperate with existing parts.
Some of the Rust community believes that they need to rewrite everything and everything will be better with Rust. The answer is no. In all my years of being an engineer, manager, CTO of a startup, and so on, I'm almost always saying, "Don't rewrite," right?
Is that the initial instinct for a lot of people when they show up to a codebase, probably before LLMs? It's probably because they don't understand the wisdom of the way things have been done in the past.
They say, “Well, we need to rewrite it.” Hence why there are a thousand JavaScript frameworks.
But the reason is the following, and this is very important to understand: it is an order of magnitude easier to write code than to read code. You see that also with LLMs. They can write code, but analyzing code is a lot more difficult.
When you arrive at a very complex piece of code, you don’t understand it because it’s so much more effort to understand code from someone else when you don’t have their thought process. Often, I joke about some languages, mostly Perl, for example. Imagine I am at my maximum intellectual efficiency in programming, and I write the best code ever. I will not be able to understand myself 6 months later because reading code is more difficult.
So very often, you arrive and don’t understand all the wisdom, all the business logic, and the reasons things were done that may not be documented. And you say, “Well, I’m going to rewrite it.” The thing is, no, you don’t, right? That’s what Kieran said: “I’m going to rewrite coreutils in Rust.” Then, of course, you arrive very quickly at 80%, then 90% takes a bit more time, and then you’ve got the last ones.
For new projects, Rust is great. Everything related to parsing files and networking, because of the borrow checker and bounds checker, is amazing, and there is nothing else. To answer a bit differently for Rust, imagine I take a piece of software like dav1d or x264, which has a ton of runtime code in assembly. I rewrite the C part in Rust, so it’s more secure. Yes, but then you arrive at the assembly, and you can jump anywhere in memory because we’re doing handwritten assembly.
So even if I rewrite the C part in Rust for security reasons, I break all the security when I write handwritten assembly because we can jump anywhere. In my opinion, we need to do something that is secure assembly: compile-time checking of the assembly. That’s similar to the checkasm projects we’re doing on dav1d and x264 with VideoLAN—to start instrumenting your assembly at compile time to check that it’s not jumping anywhere in memory.
Otherwise, you might rewrite a part of the C in Rust, but if you want to have the same performance, you’re going to have inline assembly, and so you destroy your whole security model. That’s a bit what I think about Rust.
No, I just want to say: on a personal level, I’m so in awe of assembly. Once in a while, seeing speed improvements of 62× never gets old. I run our internal test suite at work and just see that I’m still in awe at the gains we have.
Well, there’s a source of joy and happiness with programming for different reasons. But I think one of the greatest sources of happiness is in the optimization of code, and it sounds like you’re at the cutting edge of that.
I was like, “Whoa, that was cool.”
In the community, I want to speak about 2 people who are wizards of assembly. The 2 of them are actually working and living in Northern Europe, Sweden and Finland.
Henrik Gramner knows so much about Intel x86 assembly that when we ask questions at Intel about things, they tell us, “Why are you asking us? You have Henrik. Henrik knows better.” He knows the cycles of almost all the SIMD instructions across all the CPU generations. “Oh, yes, this is a P4, this is a Nehalem, this is a Core 2,” and so on.
That person is the best person on assembly in the world. And he’s the nicest person you’ve ever seen. He arrives, and you don’t see that he’s amazing. The other one is called Martin—Martin Storsjö—and he’s doing mostly the same on Arm: NEON, iPhones, Androids, and so on. He codes in assembly on his phone, editing it with the crappy virtual keyboard while watching his kids play in the playground. That is wizard level.
Yes. So when you’re programming assembly at that high level, part of that is knowing the architecture that you’re programming on.
Yes. On Arm in particular, yes.
Arm in particular. But x86, I mean, these are complicated architectures, right?
Yeah, but Arm is, in some ways, more complicated. x86, with out-of-order execution, is not so bad. With Arm, you really need to understand all the different generations of Arm processors because they’re all different. There’s the A72, et cetera, et cetera. There’s the Apple variant, there’s this variant, there’s that variant, and you need to write code that works efficiently on all of them.
With x86, broadly speaking, you have Intel, AMD, and subvariants, but generally speaking, something fast is going to remain fast on all of the variants. With Arm, it’s a much more complicated ballgame.
We’re taking a nonlinear journey through history here, but we’re talking about Michael Niedermayer, and I wanted to ask about this. For a time, there was a split between FFmpeg and Libav.
Yes. In open-source projects, sometimes you disagree.
You have such a nice way of putting it, yeah.
The good thing is that, because of the license, you’re allowed to basically do your own, right? This is normal, and it has happened all the time. At one point, there was GCC 2 and EGCS, which then became GCC 3. There is what we saw with KHTML, WebKit, and Blink. It is the same process.
Also, when I want to do a new feature today in VLC, I fork, I do my thing on my own, and then I merge back into the community. There was a split in the open-source community around FFmpeg, which became Libav and FFmpeg. After a few years, the community merged back and people moved on. It’s a bit of drama that is normal in the open-source community, but forks are important because they change the status quo of a community.
Not talking about FFmpeg and Libav here, the GCC fork made GCC a ton better because some people wanted to fundamentally change the architecture to make it faster. Of course, it’s always a question of people and so on, but in the end, you realize that FFmpeg today is better than it was before the fork. Now, we’re back all together.
I spent a lot of time—and Kieran can say this—in the community. It’s not often, to be honest, very well explained because a ton of the reasons are not public. But I think that’s normal, and that’s good.
Yeah, I mean, you’re making it sound really nice, but there were battles—pretty heated battles—inside open-source projects. It’s a very passionate community, and you have to define the direction of things in a distributed way.
So, here, looking at Perplexity: “FFmpeg and Libav split in 2011 mainly over project governance, leadership style, and development processes, not because of a fundamental technical disagreement. FFmpeg effectively absorbed Libav’s work, while Libav withered and most distributions and developers moved back to FFmpeg.”
That was a weird experience because I’m a Linux user. From that perspective, whether it’s Ubuntu and so on, all of a sudden, I think for a little bit, Ubuntu switched to Libav. Am I remembering correctly?
12, 14, something like that. Yes, something like that.
And then they switched back to FFmpeg. I was like, “What is happening?” You get to feel the ripple effects of the different internal debates that are happening.
To be fair, on Apple, when you type GCC, you get Clang. They did something like that as well.
Yeah. So to me, the fork was heated drama, but most of the development from Libav was merged back into FFmpeg, right? De facto, FFmpeg got a superset of Libav, and that gave the user—in the end, we work for the users—a larger set of features and a ton of things that were discussed.
For example, the debate on reviews and on how we push are things that are now completely settled in FFmpeg and follow what mostly everyone in the community agrees on. De facto, everyone who was active on Libav came back to work on FFmpeg because the disagreements were fixed, and in the end, FFmpeg is stronger than it was before.
I know people love drama, but—
Well, my main concern—I understand, and I think that, looking at the long history, it’s all for the good. But I am concerned because there are so few humans who are critical to the success of open-source projects, and I have seen it be a psychological toll on people and sometimes lead to burnout.
You have these incredible people who are at the core of open-source projects. There’s a moment that happens because the motivation for doing it ultimately is that you’re passionate about it and it makes you happy. Then, at a certain point, you wake up and it’s like, “This has been a bit too much heat from the drama.” At the project level, the project continues and often flourishes, but sometimes there are individual humans who are just like—
But—
…“I’ve had enough.”
Yeah, but it’s not just about forks. What you’re referring to is one of the most challenging and most interesting parts of open source today: maintainer burnout. AI is a problem because of that.
Daniel Stenberg, who is the maintainer of curl and probably one of the best promoters of open source in the world, is against what he calls “AI slop.” He’s, by the way, a member of the European Open Source Academy with me, so I’m very humbled to be in the same community as him. He says it gives a ton of fake reports or…
Bad reports, bad patches, and then a lot of maintainers have a lot of burden to maintain the software. This is straining the mind of open source developers much more than forks. For example, the XZ fiasco was because there was 1 guy maintaining it, and he got basically hammered by 2 attackers who were asking him questions nonstop at weird times at night to block him. At some point, he got fed up and said, “Okay, I can’t do that,” and gave commit access to the attacker. Burnout in the open source community is something that exists, but mostly it’s about maintaining things, right?
No, for sure. But I wonder, how do we help that? Those people are so important. The human beings are so important to the core of these projects.
So, for example, now I am maintaining a ton of multimedia and non-multimedia libraries as a maintainer because the maintainers got fed up, right? Some are on VideoLAN, some are outside of VideoLAN. Sometimes you need a tough skin, right? Because you get—it’s not really attacks—but, “Oh, this is not working, this is not working,” and you feel it personally. And this is also why the XZ fiasco was a problem, right? They don’t realize that, in the end, you have—it’s like the same graph where you see everything, and it’s just 1 random open source project that is maintaining the whole internet.
The Nebraska thing, yeah.
You see the one, right?
Yeah, this is the meme. I mean, it applies to a lot of open source projects. But this is all of modern digital multimedia infrastructure, and then that thing at the very bottom that everything relies on is FFmpeg. It’s true. And then there’s usually a handful of folks who are maintaining that.
And FFmpeg or VLC, right, you have a community of 10, 15 core developers. They’re not the worst open source projects. XZ, which is in even more installations, is 1 person, right? There is 1 guy—
libxml is—
Yeah, libxml, right? There was a big stop. No one is maintaining libxml anymore, which is the parser, the only library that is able to parse XML everywhere.
All the crazy edge cases of XML under ridiculous circumstances, and they get attacked by security researchers because there’s 1 other crazy edge case that they haven’t thought of. It’s like, yeah, but the body of knowledge to actually resolve that is massive.
There is 1 guy maintaining all the time zones for everyone who is in the middle of—I think, was it Nebraska or—
Yeah, it could be, yeah—
South Dakota? The mental health of open source maintainers is something that large corporations don’t care about or don’t see, right? It’s just like, “Oh, yeah, I’m just doing an open source report,” and so on.
Some of it is financial, but some of it—and people should definitely support open source financially, all across the board—but some of it is also spiritual, on a basic human level. There’s something that happens with this image of FFmpeg and so much of the internet depending on it, where people almost talk down to the folks who are carrying these projects forward and maintaining them.
In the security community, they certainly did. That was one of the things—I think that argument came out because there was a portion of the security community who were like, “No, these guys write crap code. They need to fix their crap code.” I’m like, “No, no, no, no. This is a guy’s hobby project. You’ve had a security bot that’s gone and found some AI-generated stuff. That guy didn’t write crap code. It’s just an edge case to the 99.99999th percentile he didn’t think about because it’s his hobby project decoding Star Wars games.”
Forget the hobby project aspect of it. It’s just hard work, and it’s beautiful, and the right approach there is to celebrate people for doing incredible, incredible work. It’s just incredible that humans step up, not getting really paid at first or maybe ever, and they’re doing it out of the love of it. Human civilization runs on people like that. We need to celebrate them.
To give you an idea, I received death threats on VideoLAN, right?
You mentioned that to me. What is behind that?
That must have been 2009 or 2010, right? Apple was moving from PowerPC to Core Duo, probably in 2006, and by 2009 or 2010, I decided that we were not going to do new versions of VLC for PowerPC. At that time, VLC—we were close to the 1.0 release. There were 4 of us, right? We were just like, “No, this is not possible.”
So I received a death threat with some powder in it, right? Remember, there were some anthrax threats at that time? It was because I had made the decision not to maintain the PowerPC port anymore. Of course, it wasn’t anthrax. It was some type of flour and so on. But I received that with a letter saying, “You piece of shit, you should die. PowerPC forever,” and so on. It was 2009 or 2010, right? I was young. I was just like, “Why? What did I do?”
Yeah, that can break your spirit.
My mother freaked out, right? We had to go see the police and so on. Now, I’m going to say that I’m quite happy that this happened at that time. It forged me a lot, right? I can take a lot of hate on me. I’m okay with it, right?
It sucks that that’s part of reality, because all the people that love VLC, all the people that love FFmpeg—like me, I have legitimately had a smile on my face hundreds, probably thousands, of times in my life because FFmpeg made me happy, period. How many times did I get a chance to say that? Zero, until I realized there was a Twitter account. Every once in a while, I’m messaging it.
One of the things I like about the Reddit meme about me—which I don’t like for a lot of reasons—is when someone says, “Oh, JB is on Reddit,” which I am, right? I say hello, right? Then I get so many people who say, “Oh, thank you for VLC.” I take pictures, and then I share that on Signal and IRC. Yes, we use IRC.
I saw, as a quick tangent, you mentioned IRC is like Slack for old people. So you still use IRC?
Of course.
Yeah, I have it on my phone as well.
Of course.
Every day.
Works fine.
Wow. It works fine, huh?
Works fine, yes.
You have to power it with a crank, I guess.
No, but there’s no—
There’s AOL. There’s AOL as your social media.
There are no ads, no tracking, nothing.
The biggest issue, to be honest, compared to Slack is that it doesn’t have threads. That’s annoying. It doesn’t have emoji reactions. Sometimes it would be nice.
IRCv3 has them.
Yes, v3, but no one uses it, and you cannot edit your messages, right? The rest works perfectly fine forever.
But how do you communicate without emojis?
Well, that’s why I said it’s for old people.
Old people.
And we do emojis with the colons, dashes, and parentheses, right?
Old school. So anyway, you communicate on IRC. What were you even talking about?
Yeah, we were talking about death threats—
Oh, damn.
—but having people thank you. Sometimes I get people who send me a message saying, “Oh, thank you for VLC.” I always answer because I want to validate the fact that you need to thank the open source community.
Yeah, please, everybody listening to this: celebrate, celebrate FFmpeg, celebrate VLC, celebrate all the incredible open source projects—Linux, everything. There are so many. And you know what? Even outside of open source, just celebrate companies that create software that you use a lot and love.
Celebrate human endeavor. Celebrate the human effort to not just build something that’s okay, but build something that’s damn good.
Yes, this is important, right? As we said, we work for technology. We do something very complex for normal people. We want our excellence in tech to be useful for everyone. This is why we work, right? This is why I wake up in the morning: because I want people to use our stuff, because it’s making everyone’s life easier.
You want to solve hard problems, work on something interesting, work on some interesting technical challenges.
As engineers, we love to build things, right? When I was young—very early—I knew I wanted to be an engineer. I wanted to do cars, right? Maybe at some point I will go back to cars, right? But we want to build things that are cool and useful. And they need to be challenging, right? Because you want your brain to turn on.
When did the 2 of you first fall in love with programming, with building, with engineering? When is the first time you programmed, Kieran?
Microsoft QBasic, when I was using Windows 3.1 and Windows 95.
Oh, wow. What did you build?
A multiplication table, just counting loops: 10, 20, 30, 40.
Nice.
Then I thought I could do everything after that. I wanted to create a football video game. I drew everything out. I was like, “I’m going to do it.” I didn’t quite grasp that it was actually a massive piece of work to jump from BASIC and drawing some pictures to a video game, but there we go.
Yeah, I think I did BASIC and then Turbo Pascal when I was at the end of elementary school. But mostly, the first time I actually did some serious programming was the first year of what you call middle school, when you’re 11. I lived in Italy for a year, in Florence, and it was an amazing year. The math teacher told us to work in a programming language called Logo, where you had a turtle that was designing things.
On the screen, you would turn left and right. In the end, we used that to do very complex programming because, of course, you could do things. This changed things, as I knew I wanted to do things with computers and program.
I don't think we quite talked about H.264 properly. We talked about David. Can we backtrack a little bit to H.264, this thing that powers basically all of the video on the internet?
Sure.
Can you tell me the story of H.264? And Kieran, you're actually a contributor to x264.
Yeah. So H.264 is a video encoder for the H.264 video standard. It dominates internet video, but also other areas such as Blu-ray discs. Blu-ray discs are interesting because the people who make them really want the highest quality, and there are some really cool high-end films that have been encoded, as well as content for broadcasting and all sorts of other areas.
H.264 was a big step change because it happened at the right time as well. A lot of the development took place when HD video was coming out. Intel Core 2 and Nehalem CPUs were getting fast. You could do real-time video. But the most important thing was a key focus on visual metrics.
For 20 years before that, industry and academia had been obsessed with mathematical metrics, or what’s known as peak signal-to-noise ratio: mean squared error, the logarithm of mean squared error, and so on. That led to tons of issues because mean squared error leads to blurring. You want to add a little bit of error to everything to reduce the mean squared error, as opposed to having a big error, and that led to loads and loads of blurring.
Hobbyists bucked that trend for their own personal videos, mostly anime. There were 2 things they did differently, and there was a big iterative feedback loop with the community. The 2 big things were psychovisual rate-distortion—using block energy and trying to compensate for human perception when making decisions—
So the psychovisual distortion, that’s the critical thing. That’s the thing. It’s kind of revolutionary that we can rethink this. Don’t make it this kind of theoretical thing about compression. Make it all about—
Being pleasing visually to the eye.
Yeah, yeah. So compressing in a way that loses the least amount of information for the stuff that matters to us humans.
Yes, exactly, as opposed to what some parts of industry are still obsessed by, which is mathematical numbers that don’t look good in reality.
Adaptive quantization was the other big one, where it was biasing bits against complex areas and redistributing them to less complex areas, like grass. Grass has some high frequencies, but it’s less complex overall compared to more complicated things. This came around because of Park Joy.
So Park Joy was really the canonical sample, the one with people running around in the park.
Yeah.
This one.
Yeah. This was created by Swedish television at the beginning of HD. It was done on film, no expense was spared in terms of production quality, and it was given away for free. This was really the sample that sorts the men from the boys in terms of having so many challenges: the trees, the water, the grass, the motion. I don’t think there’s ever been any public test sequence as good as that these days.
For people who are just listening, we’re looking at a bunch of humans running along a river, with the reflection, a lot of really high-information textures everywhere, the leaves and the lighting playing with the leaves and all of this.
You could clearly show that encoders with high PSNR—
Will blur everything.
You could see that I could turn on psychovisual stuff, turn on adaptive quantization, and it would just look so much better. But your metrics—these metrics at the time were considered so holy. These were the holy metrics that were untouchable. PSNR was the most important thing.
Can you speak to how you measure psychovisual stuff? How do you turn how pleasing a compression is for the human eye into a number? Is that even possible?
That’s what Netflix has been trying to do with VMAF. They’ve used a machine-learning model.
That’s a more recent thing. But back when x264 was being developed, that was by eye. You were basically—
It was by eye. It was developers on their laptops. So it’s not like even with big companies with professional screens or anything, it’s—
That was actually one of the goals. The developers at the time, Loren Merritt in particular, said, “I don’t want to test this on a $30,000 screen. I want this to look good on someone’s laptop at home.”
Brilliant. There is another sample, a sample from Planet Earth that is a killer sample that I absolutely love. You’re going to see why.
Yeah, you’re going to love this.
It’s a ton of birds flying, and the more it goes on, the more birds there are. At the end, it’s almost like you have millions of birds. It’s the most complex thing ever to encode. You’re watching it on YouTube, and you see how bad the YouTube encoding actually is. It’s phenomenal to optimize and get perfect quality at a constant bit rate.
There was a lot of optimization, mostly by Loren as well, on anime. For a long time, anime was very badly encoded because there was a ton of banding. You saw those issues, and there were a ton of other things. So x264 is—today, it’s still the reference for any new encoder: AV1, AV2, VVC, HEVC. Everyone compares to x264.
One of my favorite films, Cinema Paradiso—I know the engineer who created the Blu-ray, and he showed me the comparisons of x264 versus others. It’s completely different. I think a bunch of guys in the Blu-ray world started using x264. I think the big one was Chris Henderson from Warner Bros. He did the whole Fringe box set with it.
It’s quite a thing that a person on the street actually watches and wants to look good. They took a risk in their jobs doing that because they’re in a big company. That big company can buy whatever they want. They said, “No, no, no, I want to use this free and open-source thing so that things look good for my customers and build the best.”
To this day, I personally still try to avoid watching the most cinematic films on streaming services and buy the physical discs because they look good without even having to buy an expensive TV. I think that’s the key thing.
x264 is yet another example of an open-source project. It was started by Laurent Ehrsam when he was at École Centrale Paris, where VLC was born. Then you got a generation of people like Loren Merritt, like Jason Garrett-Glaser, like Måns Rullgård, and so many others.
Henrik Gramner, Anton Mitrofanov—
And this is where the assembly thing that we use now in FFmpeg, dav1d, and so on was born. x264 is an amazing project with people who were really all over the world, and I think most of them never met each other.
But all of them, according to Kieran—or a large percentage—love anime. There are several things I’ve never got into, and one of them is anime, and I need to—
I watched anime so much, especially at the time. A lot of anime content didn’t exist commercially. We were before Crunchyroll. Usually, what happened was that people who loved anime took some DVDs from Japan and ripped them because there was no commercial offering.
Some of the people who were what we call fansubbers were basically translating them themselves to make subtitles. At that time, you downloaded completely illegally. It was the only way to do that. All of that was handcrafted, and it fits the open-source community because they needed tools to encode and to do fansubbing.
One of the most amazing open-source projects for subtitles is called Aegisub. It’s a subtitling tool made for anime and for East Asian and Japanese languages.
There are weird textures in anime that I don’t think you get in real-life content. I think that was a key one, optimizing these weird textures that you get because anime is not done in a normal fashion.
Yeah. The way you produce it is not— You mostly produce it on screens these days, right? You have all those gradients in colors because they’re very easy to produce digitally and very complex to produce in real life.
The subtitles are also very complex because you often need to have the Japanese, and then you need to have the diacritics—what we call the ruby—which is the hiragana and katakana for the kanji. Because, of course, you have the official subtitling, but you also need to have the English subtitles or the French subtitles because you want to learn that.
There are so many crazy things in subtitles, and we’ve had crazy samples on subtitles that we’ve seen all around. This is an important part of the culture, but also because there was no official offering. There was no way of doing that.
Can you speak to the difference between H.264 and AV1, and then x264 and dav1d? This is a big step. Can you help people understand whether some of the streaming sites are moving more in the direction of AV1?
Let’s be honest: all of those codecs, since MPEG-2 Video, are based on the same concepts: inverse transform, intra prediction, motion compensation, entropy coding—all of them. However, each generation gives you a bump of between 25% and 50% more compression for the same quality.
And so you had MPEG-2, the DivX era, and H.264, which was changing. H.264 improved so much. Then you had more: HEVC and VP9 at the same time as HEVC. VP9 is a bit similar to HEVC in terms of compression quality, but it's royalty-free.
Because in multimedia, there are a ton of patents, and the licensing after H.264 became out of hand. It could cost hundreds of millions of dollars per year, so it made no sense. Google did VP9, and the Alliance for Open Media did this new codec called AV1. You can imagine that AV1 uses 40 to 60 percent less bandwidth than H.264 for the same visual quality.
At a given bitrate.
At a given bitrate, right? So either you set the bitrate and increase the quality, or you set the quality and decrease your bitrate. But now you've moved from SD to HD, HD to 4K, and 4K to 4K HDR. You're increasing the size by a factor of 2, 3, or 4, right? So you need better compression to keep it manageable.
It's more coding tools, bigger blocks, and lots more subpartitions in each block. It's just exponentially more complex.
It's more complex because the encoder needs to search more possibilities. For example, one of the things that's easy to understand is predicting one color block from another. You have directions, right? You can go left, right, bottom, or up, and then, in terms of the other quadrants, what I call north, northeast, northwest, and so on. But that's 8 directions.
Then you can do more divisions. You can do 16, 64, or 128, right? Every time, your encoder is going to spend more time checking, “Oh, well, this block is exactly this one.” The encoder needs to check which of those tools is going to compress better. So I guess that AV1 encoding is 2 orders of magnitude more than H.264 in terms of CPU cycles.
Yeah. And as we discussed, CPUs aren't getting faster. You're just throwing more cores at the problem.
But it's also a fact that you encode once and have hundreds of millions of users. For example, YouTube is a very good example. YouTube encodes almost everything in H.264, but popular videos get re-encoded in AV1 because it costs more, of course, to encode. But you encode once and send that to millions of people.
It's a trade-off between encoding time and complexity, and CPU usage on the server side and on the client side. At the end, if you're distributing a video to hundreds of thousands of people and the size is half of the other, then it's better. It's better for your battery, your modem, and so on.
So we can lay out, let's say, the top 5 codec-container combinations: H.264 inside MP4 containers, AV1 inside MP4 or WebM containers, and ProRes for nonlinear editing inside MOV containers. For people who don't know, I guess ProRes is—
It's Apple's codec for editing, originally for Final Cut Pro, and it's designed to be fast to decode and fast to seek, because an editor needs to move very quickly. So it's a different use case from the distribution element.
There's no, or very minimal, temporal compression in the—
There's none, yeah. There's none in ProRes, so you can do cuts. This is what we call intra-only codecs, right? I'm going to explain quickly what I, P, and B frames are.
Yes, please.
I-frames, often called keyframes, are complete frames. It's like an image. It's a JPEG, right? You start, and you see everything. The next image can be a P-frame, which is a predicted frame. You take some part of the previous image, saying, “Well, I need blocks 5, 7, and 42,” and you replace them, and then you just give the extra information.
That means that in order to decode this P-frame, you need to have access to a previous I-frame. Then, of course, you have more complex ones, which are B-frames, or B-predicted frames. They can depend on different types of frames, some in the past and some in the future. ProRes is an intra-only codec.
For the people who can see, this is—
Yeah, that's a good one.
—a very good one, right? So I-frames are complete frames. P-frames basically depend only on I-frames, and B-frames can depend on frames in front.
And this GOP, or group of pictures—I think the default for FFmpeg for H.264 is 250 frames, something like this.
Yes.
And to me, it's magic that you could predict all this, that you could have a complete frame every several seconds, and then still have this chain of predictions. The fact that somebody like me can use FFmpeg to compress something and not notice that the result still plays back smoothly is like magic.
You can even have—and we use that a ton on Kyber—is what we call intra-refresh, where basically there are no I-frames present.
You have no I-frame? You have one at the beginning, and you never send an I-frame. How does that work? What is it?
You build up an I-frame gradually across the stream as it continues.
Ah, so you refresh certain parts—
—of the image. But you never have an I-frame. This is intra-refresh that we use, right?
That's even smarter.
For me, the biggest mind-blowing thing when I started was the B-frames. B-frames—B-predicted frames—can depend on frames that are coming in the future. In order to decode this B-frame, you need to wait for the next frame that it's dependent on, buffer that, and decode it so that you can decode the B-frame.
The decoding order is not the same as the display order. That means the encoder needs to be very clever and decide, “Well, you know, I'm going to depend on things in the future.” This is—
It's incredible.
—mind-blowing.
The fact that it works so smoothly every day is kind of miraculous in some ways. You can have a stream that works across the world on one decoder versus one in the US versus one here, from different manufacturers, and they produce bit-for-bit exactly the same material. That's quite remarkable. They do quite complex things, getting more and more complex and still being bit-exact. There's a lot of work that goes into that.
There's a lot of knobs you can control in this whole process. There are a lot of fascinating parameters that I've gotten to know more and more over the years, and FFmpeg gives you complete access to them. Maybe you could speak to some of them.
First of all, obviously, we can lower the resolution, lower the frame rate, or use different kinds of codecs, as we mentioned, from H.264 to AV1. There are ways to tune the trade-off between bitrate and quality, as we've discussed. You can do constant bitrate or constant quality—say, CRF, CQ, or QP. You can use longer or shorter groups of pictures, or GOPs, as we mentioned. All that kind of stuff is crazy. The number of B-frames, for example.
What is crazy is that a ton of people's jobs are to optimize those parameters. A ton of people that you see at YouTube, Netflix, Meta, and so on aren't writing codecs. They're just finding the right parameters for the file and format they have.
Something for a movie, user-generated content from your phone, a screen recording, or something that you're going to video-edit—you don't want the same things. There are thousands of people whose job is just to optimize all that.
Yeah, they're wizards. Hats off to them. YouTube likes to deliver—all the streaming sites, actually—to deliver at scale. YouTube is really magical because it's not just doing what Netflix does, which is one-way broadcasting. It also has to upload videos from all over the place.
So they're also encoding at scale for videos that are going to be watched by 5 people, and it still has to deliver them in a moment's notice. No delay, nothing—very minimal latency. It also serves them in all different resolutions. YouTube is basically the web version of VLC.
Actually, it's funny because Google Video, which was something they did before they acquired YouTube, was using the VLC plugin so that you could run VLC inside the web browser using the ActiveX plugin. It worked in Internet Explorer, and you were actually running VLC inside your browser.
It's funny because today we have the opposite: VLC WebAssembly, where we compile all of VLC and FFmpeg for decoding, so VLC can run inside the JavaScript virtual machine with WebAssembly.
Okay, there's this legendary story that you pointed me to. It was discovered via a WikiLeaks release of Vault 7 documents. The CIA was using a modified version of VLC to basically try to trick people—what, to steal their data?
Yes, exactly.
So can you explain what the heck happened? What—
This was a surprise, right? At some point, WikiLeaks mentioned some documents. There were a few that had something related to Blu-rays and VLC, but the most interesting one was the CIA's Vault 7, which, if I understand correctly, was that the CIA had a custom version of VLC with a specific plugin. We had to write a press release about that.
VideoLAN wrote a press release saying the only safe source for getting VLC media player is the official VideoLAN website. I mean, I suppose that's a security vulnerability for basically any piece of open-source software.
Somebody can trick you.
To download it from a fake website or through a targeted advertisement, right?
That was a targeted advertisement to watch a specific file that you needed to watch with this custom version of VLC. It was the normal binaries of VLC, except they added 1 DLL. I think it was psapi.dll, which was basically reading your document folder, encrypting that, and sending that.
The thing is, this is very clever, to be honest, because once you're watching a movie, you're going to do that for 2 hours, and you're not going to touch your computer. Sometimes it's normal because it's HD, your fans are going up and saying, “Vroom,” and there is a ton of CPU usage because you're using VLC. That's normal. But what you don't see is that it's actually a modified version of VLC used by the CIA.
We had exactly the same problem with Chinese hackers targeting Indian people, and that got VLC banned from India until I had to fight in the courts in India against the Indian government to unban VLC. They didn't use VLC. They took just 1 DLL because we signed the DLL correctly, and they used that DLL in another program. So you had the vlc.exe, and it was calling libVLC, but it was calling into a fake one. They used that to target people.
There is not much we can do, actually, to block those types of hacks.
Yeah, and I think people should, for all open-source software—for all software in general—pay attention to where they download the thing.
Yes, because that means that they were not downloading it from our website.
Do the search engines help you?
No, they don't.
Just to clarify, can they prevent threats from people manipulating SEO to get to the top of the links?
Absolutely not. We have had a big issue for more than 10 years: there has been a fake version of VLC in Germany that has been reported for 12 years now. Google basically decides not to do anything. They know what's in it, but the binary is too big for their virus analyzer to analyze.
So if you're in Germany, you can go to a website that is a fake version of VLC with a custom installer. It's very popular in Germany because the website is in German, and Google mentions it before VideoLAN.
The weirdest thing is that it doesn't do anything on your machine for 3 weeks. That's how they do the detection. After 3 weeks, there is a small program—a service—that is installed at the same time, which wakes up after 3 weeks and starts downloading spyware and adware. Google knows about it. They've decided not to do anything. The guys used black-hat SEO in Germany to do that at some point.
This is very damaging because one of the things they are downloading is something that is replacing your ads inside your machine.
It's actually quite surprisingly effective, whoever is doing it with Twitter and X. With X, I'll get emails saying, “Your X account has been hacked.” However they phrase it, it gets me to at least click on the email—not to follow the thing—and then you're like, “Man, whatever they're doing with the psychology to try to trick you, they're quite good.”
There is a security update for VLC, right? You receive an email saying, “Hey, there is a security update for VLC. Think about updating right now because it can hack your computer.” You go to it. It's a website that looks decent, and you download a new version of VLC. Great. You don't know. A month later, you're hacked. You have no idea. You're part of a botnet.
Yeah. So make sure wherever you're downloading stuff, it's legitimate. I'm part of the botnet.
Speaking of which, you've mentioned that VLC sandboxing is something you're working on, and it's actually quite challenging. Why is it important? Why is it hard?
VLC is a core with around 500 plugins. One of them is FFmpeg, but we support so many other formats. We support new protocols, new filters, and weird architectures. In this release of VLC, you have modules that are going to call your drivers, mostly the hardware decoders, which are going to call your Intel, NVIDIA, or AMD driver. They're all calling FFmpeg.
There might be a security issue in the shader. There might be a security issue in VLC or in FFmpeg that is going to basically crash. The issue is that you're running VLC like every other program, like Adobe. You're running it on your machine, and it has access to all your documents.
The idea is to be sure that you have a sandbox so that we can protect ourselves, because inside the VLC process, some code is running that isn't even ours. Either it's open source from other projects that we integrate into VLC, or it's your GPU driver or something else that is provided by someone else inside.
When we crash, we don't want to allow people to do bad things. One of the common ways of hacking people is to crash a program, very often done with a web browser, very often done with PDF files, and less often with multimedia, but that could happen. When you crash, you launch something on the machine of the person. It could be ransomware; it could be a botnet. So the security of desktop applications is important.
On mobile, it's a bit different because most mobile applications are running inside their own sandbox. For VLC, we could run it inside 1 sandbox, but the problem is that we need access to so many things that basically we would have all the permissions. If you have a sandbox and you put holes everywhere, it defeats the purpose.
What we are trying to do, and we're actually doing, is splitting VLC into several processes. One is decoding, one is demuxing, and one is filters, and all of them run in their own sandbox. That way, if the whole VLC or a part of VLC crashes, like when Chrome crashes on some tab, it crashes, but it does not crash the whole program. This is what we're trying to do.
It's difficult because it's a sandbox that needs to sustain gigabits per second of memory copies. It's not a website that is 5 megabytes or 10 megabytes. We're talking about hundreds of megabits per second. So this is why it is quite challenging. This is a research topic that we are working on in order to have a multimedia player that is secure.
This is all the kind of stuff you have to think about when millions of people are using it. You've mentioned somewhere that with all the different features of VLC, when you have that many people using it, somebody will use every single feature, and they will tell you about it.
The best feature in VLC is called the puzzle filter. You click the puzzle filter, and it transforms your video into a jigsaw puzzle. You can click and move the pieces.
It's very, very useful when you're watching a French movie. You're bored because it's very long or it's a love triangle. We've seen that so many times. But you need to watch it because someone—your wife or your boyfriend—told you to do that. So you're doing that, and you can click and move the pieces around. It's absolutely useless. Who cares about that?
First, it was done by a math teacher at a high school in the south of France to teach his students about Bézier curves, which is something that everyone should know about. It's very useful. But the code was clean, so it got into VLC. It was merged in 2010.
5 years later, I received an email saying, “Hello, JB. I have a problem with VLC. The puzzle is too simple.” I was just like, “What?”
The puzzle in the UI was limited to 16 by 16—only 256 pieces. He said, “I'm sorry, but I love puzzles. This is too simple.”
There is a commit by me, which you can check online, where JB changes the dimensions to 256 by 256.
Right.
But my point is that so many unused features are used by a few people. There is a way to watch VLC movies in the command line without any UI.
I saw that. You can do ASCII.
ASCII art. Is it useful? Very useful. Imagine you're debugging a multicast network. You have thousands of very complex networking stacks. You can SSH to all of the routers and put VLC on them with no UI, and you're going to see whether it's black or it's not black, or whether it's all green or not all green. So you can see—
Amazing.
Yeah, right.
This is fun.
People don't realize there are so many things in VLC that are useful, and they have users, because once you have hundreds of millions of users, you have people who use every feature.
I would love to zoom in and talk a little more about the distinction between downloading a file and watching it offline versus streaming—the complexities and the challenges of streaming. Is there something we could say about what it takes to stream files?
We've been talking about codecs, and I think a lot of that implies encoding and decoding without having to communicate over the network.
Sure.
Sure. So can you elaborate? What's required to do things over the network?
Yeah, but it is less complex than it seems compared to everything we've talked about. Especially because the most complex thing is not about streaming in terms of streaming services, but about what was done to actually broadcast through satellites.
In most modern broadcasting services, you can pause and you can go on. But when you're sending live streaming, whether it's broadcast or live streaming for services that are live, this is much more difficult because you need to encode in real time. When you go on a satellite, you have a specific size of the link. You cannot have a burst—
of bandwidth even for a second, right? Because you don't have the space for that in your total file. However, there are different types of challenges, which are interesting challenges, but I think they are less complex than the ones we've seen in the late '90s and early 2000s about broadcasting and streaming through satellite.
They're different. They are control-systems challenges, whereas some are more mathematical. I think that's the difference.
In the streaming world, what you have is called adaptive streaming. The difficulty—and it's not really a video problem; it's mostly a CDN problem—is that you might have too many people watching the same thing at the same time, and there's congestion in the network, right? So your player has difficulty downloading things fast enough to play them. What happens is that locally, the player is going to read a lower resolution of it. There are some very clever algorithms to do that, but most of it is quite basic, to be honest.
Even on the buffering side, it's pretty basic.
Yeah, you start to download a segment, what we call a segment, and then you time it, right? If it takes more than 50% of the time to download a segment, you go down in quality. The difficulty is more about when you go up in bandwidth, in quality. When you encode, you're going to encode 7 resolutions, right? And you're going to give the bitrate. The difficulty is to have your encoder give the same bitrate, but it's not as strict as it used to be.
Probably YouTube has to figure out the human-psychology side of that: How pissed off do you get when it's at a very low bitrate, and how long should it wait before it increases the bitrate even though the connection is better? Because maybe the changes in the bitrate are what affect you psychologically.
No, I think actually the interesting one is the audio.
That's true.
You can notice when they move from full-fat AAC to the compressed versions of AAC that use Spectral Band Replication. You can hear it go a bit tinny, and that up and down is very jarring. The video side is a lot smoother, and there's less to notice. It's really the audio—you can definitely feel it when it moves from a different audio profile to one or the other.
I don't know. We're surprisingly tolerant of skipping audio glitches. I'm surprised by how tolerant people I know who are not video engineers are to watching sports at 30 FPS, for example, whereas it should really be 60. The world is a lot more tolerant of that, but audio people are very—there's an immediate feedback mechanism of, “Oh, something's changed.”
If you hear a glitch, you realize it directly. I get to fully realize that, I suppose. One of the things I'm afraid of when I listen to audio more and more is that I get to notice every single tiny detail, and that you can over-obsess, when people in general are able to blur their consumption. They can look past certain imperfections.
But then when you combine an event that is, for example, a sports event that is probably going through satellite or somewhere else and goes to a central place for encoding, and then you need to encode all those resolutions in real time, it is complex. You don't have time for QA. You need to push that to CDNs. You need to add probably DRM for protection. You need to have that over a ton of different devices.
But you're in a web browser or on very different devices that you use for television, where you had a defined set-top box or cable box that you knew, where you controlled end to end. So it's a challenge, but it's less complex. I think the networking part, while you agree to have 10 or 20 seconds of latency, is not very difficult.
Speaking of networking and latency, your new effort, as we mentioned, is Kyber, which is aimed at ultra-low latency. As you say, every millisecond counts, and you're applying that to remote-controlled machines like robots, drones, and computers. Can you tell me about it?
Sure. If you start from where we used to be, right? You used to use FFmpeg to encode files, right? And then we used FFmpeg and VLC to encode in streaming services, right? And then you need to go lower and lower. The question was: How far can we go? This question is very important because there are many use cases where you need to be fast, and it's when you have feedback interaction, right? We're not just listening to something; you're actually controlling it.
That's the biggest difference compared to what we've done so far: I need video to have feedback on something that is happening live, whether it's a drone flying, whether it's controlling a humanoid robot from a distance, whether it's controlling a rover, or whether it's playing a video game in cloud gaming. This is what I did in a previous job, right? I was CTO of a cloud-gaming startup.
This is a very interesting topic because you push the network to the limit. You need to care not about the quality, like we've done on video and talked about with H.264. You care about latency, because a millisecond is meaningful when you're controlling a car, right? Well, you've seen and used Waymos, right? When Waymos don't work—and that happens even if 1% of the time—there is someone who is basically remote-controlling that. This is exactly the stuff that we're building. It's really an SDK platform to do end-to-end control of machines.
This comes up quite a lot in a lot of different contexts in robotics. Obviously, teleoperation, or teleop, is becoming more and more important, including for training robots via machine learning.
Yes. What we do that's a bit different from everyone else is that we take only 1 socket, 1 connection, which is a QUIC protocol based on UDP, which is interesting because it's designed for low latency. It doesn't have 2 of the problems that we call the TCP head-of-line problem and the HTTP head-of-line problem. It's encrypted by default, but on the same wire, we send multiple streams, like multiple tracks. We send audio, we send video, but we also send the commands: mouse, keyboard, gamepad, and so on.
We do that while maintaining coherence—synchronization—because what people don't realize is that all the clocks actually drift. When you're controlling a robot, a robot is going to have 2 cameras, 5 cameras, 10 cameras, a ton of sensors, GPS, and so on. If you want to train your robotic AI model correctly, you need to have all those in sync and coherent.
What we've done—and it's all the stuff that we learned on VLC, in broadcast, in real time, and MPEG-TS, which Kieran knows well—is that we account for clock drift. When I record a Kyber stream from a robot, I am sure that it's going to be predictive in the way you play it back. When you're going to do recording and training of your AI model, you need to be sure that every time you retrain based on the data, the data is going to stay coherent. Clocks actually drift. The existing solution works with 1 camera. Once you go to 5 or 6, it's more complex.
You want to make sure that the visual snapshot perfectly matches the time it actually happened.
Exactly. And also, if you're going to control—if I do something on a robot—I need to be sure that it is actually happening at that precise time. So we have on the server, which would be a robot, a sort of retimestamping mechanism accounting for clock drift.
That's one of the use cases of Kyber: to control robots. I also think about remote drones, whether for defense or non-defense, remote cars, and remote submarines. There are many places in industry, or in remote surgery, where the expert cannot go everywhere the machine is because it's either dangerous or too costly. So you allow people to have machines next to you. The goal of Kyber is to make distance disappear because it's either the projection of skills or the projection of power.
Imagine we are all like—you've seen the Ray-Ban Meta and everyone else, right? You need to stream there, because you're not going to run anything over there, right? So you need GPU power, whether it's on a cloud or on a phone, to stream that. All of these use cases need to be not about extremely low latency, but real-time latency for video.
That means you need—we're toying with the encoders so that the encoders encode a frame in 4 milliseconds. Kieran, with his company, also goes under those types of latencies, because you need to optimize local latency as much as possible. This time is going to be added to your networking time.
It's not just about low latency; it's also about reliability. We do clever things like forward error correction. Forward error correction is when you over-transmit a bit of data, a few percent, and while over-transmitting, you're allowed to lose some packets. All of that is very difficult over an internet network where you're going to do things very far away.
If you check that all packets are delivered, you add a ton of latency. If you don't want latency, what we do is over-transmit some data that you can reconstruct on the client side when there are things that are broken.
A few days or weeks ago, we were doing a demo around Las Vegas for CES. We had a rover that was fully 3D-printed. It's very simple. It's a car, right? It's a small car with a telescopic arm, and it was actually controlled from France.
The video was from a webcam and a very small server. A small PCB was basically running and sending that to someone who was on the other side of the planet.
And so there are so many use cases. You can also think about having AI control many drones and so on. Technically, we need to be amazing at video and networking. We need to care about every millisecond in networking, encoding time, and decoding time, and we also need to integrate at a very low level.
So synchronize everything together well. But how? What kind of latency can you get to? When you say milliseconds, what’s the goal?
My goal is 4 milliseconds of glass-to-glass latency.
What does glass-to-glass mean?
It’s easy, right? You have a computer running a program, probably a video game, and this one is actually running. It could be an example of a robot. Then you have the replica that is created through the network, and you want, if you take a 1,000-hertz camera, to have that picture there within 4 milliseconds. Four milliseconds means 240 hertz.
Yes. Nuts.
So far, we’ve achieved 7 milliseconds from Windows to Windows or Windows to Mac. If you look at the timing, most of that is around 3.5 milliseconds inside the NVIDIA hardware encoder and around 2 milliseconds on the Intel decoder. The encoder plus the decoder is already 6 milliseconds. In order to go down, we need either some other type of codec or a better encoder that is faster. But 4 milliseconds would be the grail.
That’s pretty nuts. I love it, though. I don’t think anyone’s ever achieved that, right? That’s fast.
You can achieve that with custom hardware, with SDI and professional hardware. But I want that to work over the internet. I want that to work with any robot where you’re going to have a small Jetson Nano in it or an N150, right?
I want that because there are going to be millions of robots. Drones are just rolling robots, flying robots, or swimming robots. It’s just a machine that you control. In order to do that, either you need to teleoperate them, or when everything is fully autonomous, you need to teleobserve them. You need to check what’s happening.
In my view, in the future, all those remote cars will be teleobserved by an AI model, which is just going to say, “Well, everything is good.” When it’s not good, it will say, “Hey, there is a problem,” and then you have an operator. This is going to be about safety.
When you have your humanoid taking care of your grandma or my grandma, I want to be sure that everything goes well, and I don’t want to be in those horrible scenarios where the robot is dangerous. When I’m driving, I want the car to stop when it should stop, and if needed, someone takes care of that.
There are so many use cases and scenarios around real time, and the goal of Kyber is to make real-time control of machines. Distance disappears.
It’s incredible. Some of the same technology and ideas that we’re talking about are connected to what you’re doing.
For me, it’s amazingly challenging. I would say that I’m doing okay with video, but with networking, I have so much more to learn. It’s about congestion protocols and bitrate adaptation in real time. It’s quite funny.
I created this project, and we’ve fundraised in the U.S., of course. But it’s open source. This is important. Everything on Kyber is open source.
So how do you make money?
It’s a dual license: commercial and AGPL. You remember what you said about licenses. Basically, if you want to use Kyber in your product, you must have your full product open source. If you want to use this amazing technology but not open source, you pay for the commercial license.
The small teams, hobbyists, and very small companies that want to do that can use the technology. They build something that is open source and cool.
That’s awesome.
If you’re a large company, you’re going to have support, all the IP, the right to make modifications, and so on. It’s really cool.
I’m also building robots, and I love that. The rover we have is 3D-printed. We’re finishing a demo where it’s an actual wing, a type of drone wing that is also fully 3D-printed. We’re trying to make a sailboat that is 3D-printed, and we’ll work on some humanoids.
Of course, they’re not going to be very good robots. It’s not our job, but we’re here for everyone to make robots.
Ah, you’re talking to the right guy. I love robots. There’s a bunch of them upstairs. Teleoperation is going to be really, really important, especially as the number of robots scales across the world. So, 100%.
Let’s talk about the future of multimedia. We talked about FFmpeg and VLC, but we didn’t really mention some of the codecs, like AV2. Can we just lay out what AV2 is? What’s the hope for it? What are H.265 and H.266?
AV1 is this codec created by the Alliance for Open Media, where there are Google, Netflix, Amazon, Apple, and VideoLAN. We try to make a very good, royalty-free codec. Now it’s being deployed, but the codec was finished in 2018, and a codec takes years to be used in a wide range of scenarios.
AV2 is the next generation of this codec. It’s 30% better, so if you keep the same quality, you get a 30% reduction in bandwidth compared to AV1.
What’s the connection between dav1d and AV2?
We’re going to do dav2d, which I call “Devid,” because “deux” is 2 in French.
Ah, well done.
You have to know that dav1d is an actual recursive acronym. It means “dav1d is an AV1 decoder.”
Oh, nice. Nice. I didn’t even think of that. People should know that dav1d is spelled with a 1.
Yes. And so dav1d 2—
It’s going to be spelled with a 2. Please tell us.
It’s going to be D-A-V-2-D. I don’t know how you pronounce that. Again, we did a demo at CES of VLC running the first demo of AV2.
Can you clarify the specification of AV2, and then the encoding and the decoding?
Sure. The specification is the document that explains how the codec is supposed to work.
And that’s really AV2?
That is AV2, like H.264. Then you have an encoder. The current encoder is called AVM, and there will probably be other encoders, probably one called SVT-AV2. Those are the encoders, in the same way that x264 is an encoder for H.264 and x265 is an encoder for H.265.
The decoder for AV1 is dav1d. The decoder for AV2 is dav1d 2. The decoder for H.264 is FFH264 inside FFmpeg. The decoder for HEVC is FFHEVC inside FFmpeg.
There is a next-generation codec from the MPEG world after H.264 and H.265. There is one called H.266, also known as VVC.
So HEVC is H.265, and VVC is H.266. Why is H.266 so sexy and so much better?
The question we often have is: Why are there 2 names? Most of the time, it’s a joint effort from the ISO world and the ITU, which is the International Telecommunication Union.
These are the 2 regulatory bodies.
No, one is a private entity and one is the United Nations.
Which one is private?
ISO is private.
In theory, H.264 is MPEG-4 Part 10, H.264/AVC. That’s the full name.
It’s the concatenation of the ISO name and the ITU name, even though they work together. This is politics—historical politics.
For HEVC, it’s MPEG-H, H.265, HEVC. And there is H.266, which is also named VVC.
Is there a high-level thing to say about the improvement of—
Thirty percent each generation is the best summary.
This is true both for the AV codecs and the H.264, H.265, and H.266 codecs.
The professionals who are listening to us are going to kill us because they’ll say, “No, it’s 35%, 25%—”
“No, it’s 50% or 60%.”
But globally, you need to know that HEVC is 30% better than H.264, and H.266 is 30% better than H.265, because there are so many cases and scenarios.
For example, there are cases, especially for screen recording, where the gains are humongous because you have the right tool designed for that. For a specific video, a new generation is going to give you a 70% or 80% gain.
There used to be a ton more codecs, but now the 2 main codec families for transmission are the H.264, H.265, and H.266 family, and the other is AV1 and AV2.
I guess the major difference would be the cost of encoding.
Yes, and the royalties for the patents. This is the reason you see the AV versions of codecs: They try to be as royalty-free as possible, which means no cost for the patents.
What you need to know—and we haven’t talked about that so far—is that multimedia is what we call a patent minefield. There are 2 places where you have the most patents: everything related to 3G, 4G, 5G, and radio frequency, and multimedia. It’s because it’s very mathematical, and you can get great gains and so on.
Google, Meta, and Netflix wanted something that was royalty-free. There are people who say that they have patents outside, but they are fringe patents. It’s mostly true that it’s patent-free.
Oh, you should explain. Patent checking was done as part of the standardization process in AV1 and AV2, whereas patents aren’t even discussed in the MPEG world. Patents are completely off-topic.
Can you educate me on the patents side?
Usually, MPEG creates a format, and then everyone comes around and says, “Well, I have all these patents for the format.” They usually form a pool called MPEG LA, the MPEG Licensing Administration. You put all your patents in, and then you ask everyone who’s using the format to pay for it.
What does it mean to have a patent on a codec? Why are there so many patents?
Imagine I’m doing something where, instead of using blocks that are square, I’m going to use rectangles.
Somebody patents it.
Oh, man. How many lawyers are involved?
It pays for a lot of lawyers. The biggest issue is that, at the time of H.264, the patents were, let's call it, sane. But there was so much money in it that, for HEVC, a lot of things were pushed into the specification—there were a ton of things that are not useful 99.9% of the time, just so someone could add a patent on them.
And so it became that, for HEVC licensing, there was MPEG LA plus another patent pool called HEVC Advance.
That one?
I think Nokia was outside of the patent pool.
Yeah, a few of them are outside.
And so it was impossible to license, right? I think that several months ago, HP decided that they were going to remove support for HEVC from their Windows laptops because the cost of those patents was increasing. It arrived at a point where there were uncapped patent fees.
For YouTube or Netflix, we could talk about hundreds of millions of dollars in patent licensing per year. They said, “You know what? At $100 million per year, I could create my own codec,” and this is what they did.
And so that’s why we have the Alliance for Open Media, of which we are part, which created AV1 and is creating AV2. We also create audio codecs. The main difference is that you need to work around the patents or do things that are not patented, so a lot of things are different.
The basic things that were done in MPEG-2 30 years ago are, of course, out of patent. But, for example, there are things like a golden frame, an S-frame, or different types of frames.
These are all patented ideas.
I can’t believe it’s not butter. I can’t believe it’s not a B-frame. It’s kind of what it is.
Yeah, that’s to try and sidestep things like that. You need to have double creativity: creativity in terms of being more efficient, but also creativity to be sure that you don’t infringe existing patents.
For example, VVC has all the patents of HEVC plus new ones. That’s why AV2 tries to be as royalty-free as possible.
To what degree do FFmpeg and VLC have to think about this kind of stuff?
We don’t. One of the reasons why VideoLAN was in France is that France rejects software patents. Most of those patents are invalid in France because they’re basically mathematical patents or idea patents, and they’re not valid in Europe.
I once calculated that if I had to pay all the licensing fees for VLC, I would need to pay more than 200 euros per user. It’s the same in dollars.
Let me just ask, at a high level, out of curiosity: The meme online, on the interwebs, on X and Twitter, and so on—and my own sense, having friends in Europe—is that Europe is not friendly to entrepreneurship. They overregulate, there’s too much bureaucracy, and so on. Is there anything positive to say? Is there hope for entrepreneurship in the future of Europe? Is Europe over from a tech perspective?
Yes.
—in the future of Europe? Is Europe over from a tech perspective?
What you probably don’t see yet is that there’s a new generation of entrepreneurs in Europe, mostly in France. The UK has done it for a long time because it’s more of an Anglo-Saxon type of business outlook.
But especially what happened in France—and, of course, sometimes it’s a bit overdone with everything called French Tech—is that today, most of the people who come onto the market want to create startups. 15 years ago, that wasn’t the case. Everyone wanted to work at big companies.
When you failed in France, for example, 20 years ago or 15 years ago, and destroyed your company—which is normal for a startup—you were not allowed to create a new company. There was a lot of stigma. The stigma is gone.
There are so many things happening in AI in France and so on. There is overregulation, sure. I know that; I’m an entrepreneur. But it has some good things also.
I mean, is there a paralyzing aspect? If I look at the case of somebody I’ve become close with, Pavel Durov, he was blamed directly by the French government for the kind of things his “platform” was hosting. I could see the same kind of stuff, just as an example, with VLC being blamed for the kind of videos that people are watching.
But they tried, right? We had issues.
I mean, that’s the pressure that people worry about, because if you have to think about that kind of stuff when you’re just obsessed about—
No, you don’t think about it.
And that’s okay, right? But what if they come in? What if they show up and—
There is no office. VideoLAN doesn’t have an office.
I mean, this is what happened with Pavel. They arrested him, right? They arrested him for particular videos or particular content being shared on the platform.
Sure, but I don’t have any platform. Everything is on the client side.
Yeah, but they can still arrest you.
On what grounds? I’m not sharing anything. The content doesn’t go through my systems.
For sure, but it’s still lawyer fees. That’s the problem.
Yes, that’s correct.
It’s paperwork. If you had infinite trillions of dollars, you would win easily because you’re on the right side. But there’s a degree to which they suffocate you with paperwork, through process. That’s the downside of bureaucracy. It’s the Kafkaesque thing.
You have to realize that one of the good things, for example, in France or most of Europe, is that answering to a court order does not make you bankrupt. It’s not like in the US, where it can actually bankrupt you.
The way the legal system works is that I receive lawyers’ letters every week. I can tell you that the cost of lawyer fees for VideoLAN is less than $10,000 per year. That’s not really scary.
It’s similar with Pavel. The intelligence agencies tried to say, “Can you put a backdoor in VLC?”
Yes. Two of them.
What do you say?
No. I was a lot less polite.
I see. You’re basically saying, “Hell no.”
If we had to compromise our software, we would shut it down. This is clear.
What’s the definition of compromise? Allowing a government to put in a backdoor?
There is no code that gets into VLC that we don’t control. The way we compile VLC, you would call me completely paranoid. We compile on boxes that are offline, where we start by compiling the compiler. We do everything offline in places that have never been connected to the internet.
The way we do signing, there is a double signature. We’ve seen—and we believe it was a governmental agency that is not from the Western world—try to push a fake binary into our own servers, and that scared us a lot.
VideoLAN is open source. How can you kill it? Where would I move? I move to Malta. I move to, I don’t know, the Cayman Islands, change the domain name, and start again. VLC is a tool. It’s a tool that is going to help people do things. We are not a platform.
And for patents, I’m sorry, but most of the patents shouldn’t exist. You shouldn’t be able to patent math and matrices. This is wrong.
Does VLC ever censor the kind of videos it can play, not based on the content of the video?
No, never. VLC is completely offline. It doesn’t talk to any server, so we don’t know anything about what you’re using the software for.
So there’s no government that can say, “The French government doesn’t want anime. Anime is destructive to society. We don’t want any anime to be played.”
No, they cannot do that. What they tried to do is say, “I want to know if that person watched that type of video.” The answer is, “No idea.”
So no on that, too. For surveillance, no.
No. The only infrastructure we have is a downloading infrastructure. There is no telemetry in VLC.
It would be difficult because of the international nature. It would be difficult for you to incorporate that code because there would be someone in the UK, someone in Germany, and someone in the US as part of VideoLAN who would be able to see that. It would be extremely difficult.
The only thing that we can do, which happened, is that we had a case with some police in the US who said, “We have a murder case. The file is corrupted or doesn’t play in that version of VLC. Could you help us?”
We never have access to the video. It’s like normal support.
Oh, it’s really about playing the file?
Yes. I remember, in the middle of the Afghan War, I received an email from someone in the army. I don’t remember the rank. It was just like, “We have a big issue with the latest version of VLC because it doesn’t play the file correctly on an RTSP server that we have, where there are all the movies.”
He said VLC was very important for the morale of the troops on the ground. At night, I think it might be boring, so they had a collection of videos or movies to watch over there.
Of course, I did an update and broke some support for RTSP. I gave them another version just for them because it was important.
And because VLC is completely open source, I think it is allowed on US Army laptops, right? Because I guess someone in the US military actually looked at it and said, “Well, okay, this is okay,” right? And the way we document how we process, that was okay, right? So the only way we work with authorities is to help them with support.
That’s amazing. That’s an amazing story. Yeah.
We don’t see anything happening in how people use VLC, and this is strong.
Do you feel the stress of this? First of all, millions of people are using it. Second of all, the military is using it. Maybe sometimes there’s pressure from governments. That’s a small team, right?
Yes.
How big is VLC—like, the core contributors? How many?
6 or 8. But everything legally is only me. Everything that is legal is only me.
You’re not stressed about this?
I used to be in a lot of stress. But the thing is, we’re doing what we can for everyone, for the greater good. We make extremely complex technology easy for everyone. We’re a tool, and every tool is going to be used for great things and for bad things, right? You cannot blame a tool, I think. And this is very important for us.
I used to be in a lot of stress. I’m not anymore, right? I have a way of thinking about what is the worst-case scenario, always, right? And the answer is, at the end, if I take it like a chess player, in the end, am I dead? Yes or no?
And I do that nonstop, right? And that’s also how I do my startups, right? I’m here to get something right. What is the worst case? It goes bankrupt. That’s life. A company lives, a company dies. That’s okay, right?
My moral way is always, “Am I dying in the end? Am I hurting someone?” If the answer is no, then too bad, right? Some lawyers are going to be unhappy. What are they going to do? Take all the money from VideoLAN? Wow. They’re going to have 50 grand. Amazing, right? What are they going to do with that?
The source code is out there. It’s not stoppable, also because what we do is good and it’s done for everyone.
What’s the secret to your zen? Over and over in the chats I’ve had with you and in the conversation today, about every even tense topic, you’re very zen. What’s the source of zen?
One of the coolest communities in open-source multimedia, mainly led by someone called Dave Rice—I’ll give him a shout-out, I think from City University of New York—is the archiving community. They’ve done so much stuff.
They value open source, first, because they lack budgets, but second, they see the fact that archiving video is important for the world, and being able to play that is a big problem. Famously, in the UK, there was something called the New Domesday Book, and they archived lots of stuff on BBC Microcomputers. Within 10 to 15 years, no one had the right software to play that. I think it was 20 years or something like that, and someone had to go and reverse-engineer this. Imagine that in a thousand years.
I think one of the great things about FFmpeg is that it’s written in C. C is the closest to mathematics you’re probably going to get. It’s the closest to logic—
Do you think in 1,000 years we’d still have C compilers?
Yes. We have languages that exist that haven’t changed too much. We have mathematical notation that exists. It will be like Latin. C will be like Latin. It will be a thing that you learn from the past, but it will still be usable in certain contexts.
The archiving community is really great, practically. They have limited funds. They funded the development of the FFV1 codec, so that’s a lossless codec. The archiving community is really scared about the act of compression losing things, and they have a fair point in this. If they compress too hard, it could change the view of the material. There could be something slightly different here and there, so they’re really concerned that things need to be not just compressed well, but lossless and fast.
They worked with FFmpeg to develop a whole new codec designed for fast, software-based encoding. They’re really concerned about resilience, so if they’re storing on tapes or other hard disks and they lose some bits, they need to recover quickly. They can’t lose a whole GOP because they’ve lost a bit—something like that.
So they’re a really great bunch of people. They funded GPU encoding in FFmpeg to make FFV1 encode faster. It’s really about preserving the world’s multimedia heritage in a way that’s usable, and there are a lot of great teams and archival groups across the world who’ve chosen FFmpeg and FFV1 as their archiving solution.
They can really provide us with super-specialist advice. They can explain, “Ah, in the 1950s, colorimetry was done like this on this certain type of tape, and so there is this special case that you need to handle, and you’ll never get this anywhere else.”
You see, they know things about video that we don’t. Every time I talk to Dave Rice or the people from the British Film Institute, I just learn something new, and I’ve been doing video for 20 years. They have a lot of expertise, especially on colorimetry and colors.
Storage, these other things.
I mean, they have a deep, deep appreciation of the content itself, of the video itself. Especially when you’re thinking of lossless, they’re terrified of losing something essential about the thing, and in so doing, they’re deeply understanding the thing that is to be preserved, which you sometimes might not be thinking about when you’re obsessing about the actual technology of the encoding and so on.
And when you enter the rabbit hole of film scanners, you take those things to make them digital, and it’s a huge topic that would take another 5 hours of the podcast just on that topic.
On film, and there’s a lot of film that needs to be archived. Film is degrading. It’s maybe not stored in the right environment.
The other thing they can do is, because it’s open source, they give away their workflows to countries that can’t afford archiving institutions, where archiving is done by volunteers or other groups. They go and teach—in India, they teach children to do FFmpeg commands.
They’re really great. They’re really the model community, the model ethos of what we’re trying to achieve. They are such a great bunch of people, so interested in participating and being part of something much bigger because they realize the work they’re doing, in a thousand years, is going to tell a lot. In a thousand years we may be drowning in AI slop. This stuff needs to be important and archived well. What was life like?
Yeah, it feels like capturing the 20th century and the 21st century is essential because it feels like a transition point, where we went from scarcity of data to oceans of slop, and that transition point is good to archive.
It’s important, yeah. But people don’t realize we are losing a ton of films today. There are a ton of things from the ’30s, ’40s, and ’50s where there is no value—
And tape. In the ’70s and ’80s, there was tape, and there aren’t enough tape heads in the world to read all the tapes left to be read, so they have to decide what they want to archive and throw away the rest of the tapes.
There’s a huge moral hazard, I guess, for want of a better phrase, around this topic, because this is a digital record of human history and they have to make decisions. And there’s digital stewardship, I suppose, for want of— I made that phrase up. That’s not a real phrase.
And then there’s, realistically speaking, a needle in a haystack, where there’s a lot of value in archiving all that footage and then, over time, finding the gems that we don’t know are there.
Hey, there was something in that corner that we just didn’t—
Yeah. Uh-huh.
And that would’ve been compressed away because it was some little thing. Oh, wow, there’s something there.
That’s it.
And that’s it. They’ve made sure that it’s lossless. They can prove mathematically that it’s lossless. They can run different trade-offs: if there’s a bit flip, if they lose a bit, if a single bit flips, I can make sure that I only lose a portion of a given frame. They can do error recovery on previous frames. They can do all sorts of different things.
Do you think VLC and FFmpeg will be here 100 years from now?
FFmpeg, yes.
Yep, FFmpeg, yes.
VLC, maybe.
What’s the future of—where is FFmpeg going? Where is VLC going? In the next 5 years, 10 years, 20 years.
5 years, 10 years is easy. The question is after that, right? The question is, do we arrive at something called holograms, right?
Yeah, so will VLC and FFmpeg expand to whatever—
Multimedia.
So multimedia might become—I’m sorry for the pothead expansion of the topic—but if you look at something like Neuralink with brain-computer interfaces, it’s very possible that we start to consume whatever multimedia means: whatever codec, whatever data our brain wants to consume through brain-computer interfaces. That’s one.
Then virtual reality, of course.
You will have VLC for Neuralink.
Yep, and you'll have FFmpeg -i input format human brain.
Yeah. There are going to be codecs for the brain.
Sure, 100%.
Of course.
Yeah, to compress neural information.
Today, there are new codecs for—
Whoa.
For example, what we call point clouds, or volumetric videos. There is a ton of research on what we call RGB-D, so codecs for depth that are useful for robotics and for 3D things.
Nice.
There are a ton of codecs for compression of 3D elements.
Compression for astronomy.
On VLC, for example, we already have a VR and XR version of VLC. Also, on Kyber—we talk about Kyber—we do streaming of XR content for glasses that don't have enough power, or inside the Apple Vision Pro or the Quest. We already work on streaming 3D, XR, interactive, low-latency content.
There is something called volumetric video, or point-cloud video, so it's not stopping. At some point, VLC and FFmpeg will manage 3D data. It's obvious.
So that's where it's moving. The community is open.
Not everyone in the community sees that, but Kieran and I are entrepreneurs. We know where it's going. We see that, right?
I suppose there is a tension, probably inside FFmpeg. It's like, “Hey, listen, folks, we're really good at doing video and audio, so why expand? Let's do the thing we're really good at doing.”
In order to answer that question, we need to answer what multimedia is. Multimedia is a digital representation of several streams for the human senses, and we will do that.
Imagine there is now a way to not have a microphone, but to have an odor sensor and a diffuser of odors. It will get into FFmpeg.
So your demuxer is coming up.
Yes. Of course, your demuxer has a new track type that is basically odors.
Smell, touch.
It's like audio. You'll have a left and right nose track, just as you have a left and right audio pair. It's easy.
Yes, of course.
Stereo smell.
Stereo smell, yeah.
In VLC, for example, we already have a plugin for haptics. It's mostly for what we call 4D cinemas, the ones on hydraulic arms. You know, where everything is moving, like in theme parks. There is a synchronized data feed that is basically transporting this information.
Is there yet a standard for that?
There are many standards.
This is—you make me so happy.
Of course, we have a plugin that is not in the normal version of VLC—
That's good.
—that is basically transporting those types of movements, which are physical movements, or haptic movements. It is a human sense, so it will get in.
That's such an exciting future. It's a small community of developers. How do you pull that off? If you're a contributor to FFmpeg or VLC, it feels stressful. Just looking on Twitter, it's a huge amount of work to make it work on all these different operating systems. It's an incredible effort.
No, see it in the other direction. We are not the contributors; we are the maintainers. We maintain for everyone. Every year, around 150 people contribute to VLC, and maybe 300 to FFmpeg.
Our goal as a small team is to get all the contributions in. If there is more usage, there will be more contributions, and those people will write the right module, the new format, and so on.
We care about the architecture of VLC and the architecture of FFmpeg. Now we're doing things in VLC like spatial audio. We did the demo not long ago. There were changes needed to the architecture, and we did the first spatial-audio module. When we're going to add the second one, it's going to be easy, and the third one is going to be easy, right?
Our goal—and it's going to be the same for others, or for haptics—is to work on the architecture so that modules can be added to provide future capabilities. We are a multimedia framework, so that's not just audio and video. It's everything that is timed and represents something that you can sense. If it's brain waves, it's going to be brain waves.
I think that's inevitable. Sorry. I love this on so many fronts because FFmpeg and VLC are pushing companies and pushing the world to standardize. For example, to standardize brain waves.
It would push—I hope Neuralink comes up with a standard for multimedia via brain-computer interfaces, or for robots with haptics.
By experience, what happens is always the same. You start with a new topic. There are 5 different standards because everyone starts to do this. The hype goes down, and every time the hype goes down, people start to say, “Well, you know what? We need to do a standard.”
Two or 3 companies, usually not the leader but 2 or 3 followers, do a standard. Then we implement the standard, and then it's the end of the curve. It starts to be more paper.
And then the leader is kind of pressured into it because it is better to do a standard. Yeah.
For example, 3D audio. 6 or 7 years ago, everything was about 3D. You had Cardboard on Android. You had 2 audio formats. They're all dead, right?
Now it's coming back with actual use cases, and we learn from the mistakes of the past standards. So it will be the same everywhere.
I saw somewhere you didn't have too many nice things to say about Dolby.
No, I don't.
It used to be an amazing company doing tons of great things with amazing engineers. They defined what sound was. Now it's mostly lawyers and licensing.
Oh, so they're closing stuff off. They're trying to make money on licensing.
No, it's just that they don't innovate as much as they did. It's a bit like—I'm sorry to say—HP.
Very true.
My 2 favorites are “Talk is cheap, send patches.” I think that embodies a lot of the stuff we've talked about: stuff doesn't get built unless someone does it. It doesn't just appear from the ether.
The other one that I like is “FFmpeg, nothing is beyond our reach.” I think that comes from a U.S. military satellite patch, where they invented some kind of monitoring system that could see the whole world, and this was released.
Wasn't there something where FFmpeg was running on a rover on Mars also?
Yeah, FFmpeg is used by the Mars 2020 rover to compress pictures. They wrote a paper about it, and they really wanted to use as much commercial off-the-shelf technology as possible.
Oh, that's cool.
FFmpeg runs on Mars, so we are a multiplanetary open-source library.
Nice.
We've often seen tweets from people using VLC in weird places. A lot of the people doing Formula 1 are in all the paddocks; they use VLC to play the live feed. We've seen the European Space Agency. We've seen SpaceX monitoring the launches with VLC, and it fills you with joy.
I've seen a particle accelerator.
Oh, yeah. One of the most amazing things I did was go to CERN, to the LHC, because they were using VLC to monitor all the sensors on the ring, which is 27 kilometers long.
They had some analog cameras, and they were using some of the capture cards to go from analog to VLC, so VLC could stream on their multicast network for everyone at CERN to access it.
I visited that in 2010 with Laurent, and we fixed their issue in an hour or something like that. It was some parameters that maybe weren't well documented at that time.
He said, “Okay, for the whole day, what do you want to do?” We visited everything—things with antimatter and colliders and so on. That was one of the most amazing days of my physics background.
Yeah, it's used everywhere. Any tweets, Kieran, that you regret?
Tweets I regret?
Or is it like that—how does the French song go? “Regret nothing.”
“Je ne regrette rien.” Yeah.
Yes.
That's very important for me. Don't regret anything. Regrets are a tax on your mind. Learn from your mistakes, but don't regret, because you've done it. Unless you have a time machine to go back in time, don't regret. It's just going to tax your brain. Learn from your mistake, sure. Don't regret.
It reminds me—it's beautiful. It's a tax on your brain. It reminds me of the Johnny Depp quote I saw, where he was saying, “Hate—you know, I don't hate. Hate is a very expensive emotion.”
Are you comparing me to Johnny Depp? Because that would be your first one.
Well, gentlemen, like I said, I'm eternally grateful for the software that the two of you and the larger community have been part of building with FFmpeg, VLC, and everything else. I'm eternally grateful for the spicy tweets. Never stop.
I'm grateful that you would talk with me today and give me this sexy hat. I feel like a wizard. I feel special, and I feel special to get a chance to talk and celebrate the piece of software that brought me so much joy over the years.
Thank you for everything, and thank you for talking today.
Thank you for having us.
Thank you so much.