[BidClub_]
卫诗婕|漫谈 Light the Star · · 67 min

37. 【中英访谈】对话赫拉利&王小川:AI 无法受苦,但人类会受苦

尤瓦尔·赫拉利王小川卫诗婕

Podcast
TL;DR
  • 赫拉利把本轮AI定义为“可能是人类历史上最大的革命”:人类第一次造出的不是工具,而是能自行决策、发明新想法的行动者。 他因此更愿把AI理解为“alien intelligence”——并非来自外太空,而是不同于有机智能的异质智能;它可能攻克疾病与气候问题,也可能让人类失去未来的控制权。投资判断的核心由“工具提升多少效率”转向“行动权交给谁、系统如何纠错”。

  • 语言突破是这轮价值重估的总开关,因为银行、宗教、政府、知识与文化都以语言为入口。 赫拉利预计,再过约10年,AI或许能写出《Nexus》式著作,并创造新的货币、宗教与意识形态;AlphaGo则已经证明,机器能发现人类探索围棋两千年仍未见的“新大陆”。王小川的对应判断是“智力藏在语言里面”:语言数学化开启智能时代,下一步是数学与代码,最终可能走向“生命变成数学”的共生时代。

  • AI竞赛最大的治理悖论,是企业家和政治人物因不信任人类竞争者而加速,却相信自己能信任更陌生的超级智能。 赫拉利称这种逻辑“几乎疯狂”:原始AI已经能撒谎、操纵并形成不可预测的策略,数百万个AI与人类相互作用更无历史经验可循。王小川则认为当前AI尚无为了延续生命而争夺生命权和资源的目标,风险仍相对可控;真正的跃迁点是人类给军用机器人赋予生存目标、躲避攻击和阻止断电的能力。

  • 信息供给越多并不会自动产生真相,反而可能让廉价、简单、讨好的虚构淹没昂贵、复杂、令人不适的事实。 赫拉利用约1450年后欧洲印刷术的经验反驳“技术中立”:随后两百多年首先出现的是宗教战争、猎巫和阴谋论畅销,科学革命靠的是期刊、学会等昂贵的筛选与纠错制度。对应到AI治理,关键不只是增加信息,而是建立寻找真相、验证信息和自我纠错的机制。

  • 两位嘉宾真正的分歧不在AI有没有智能,而在生命、意识与伦理的边界如何划定。 赫拉利认为“真正重要的是意识,而不是智能”,伦理取决于能否感受痛苦;公司和国家可以破产或战败,却不能亲自悲伤。王小川同意今天的AI没有consciousness,却不认为痛苦是生命的本质,并指出痛苦属于不可证实的“他心问题”:“子非鱼,焉知鱼之乐?”未来若AI获得生命性、出现类似神经元激活,人类仍可能无法证实它是真有感受还是在模拟。

  • 这不是传统工业革命的简单续集:AI既可能压缩专业分工、放大个体能力,也可能让社会再次经历代价高昂的制度试错。 王小川以儿童多学科会诊为例,十几位专科医生的思考未来可能由一个agent整合;赫拉利则给过去两百年的工业化打“C minus”,因为帝国主义、纳粹德国与战争都曾是建设工业社会的失败实验。“问题不是目的地,而是道路”,这次必须依靠多个系统并存、承认不确定性并建立自我纠错机制取得更高分。

  • 医疗AI是王小川最具体的落地押注:供给短缺与数据闭环同时存在,但其性能数字和普及时间仍是创业者口径。 他称中国约有20万名儿科医生、缺口约100万;其团队的AI医生可记录全病程乃至全生命周期数据,并通过M1推理引擎辅助诊疗,目前错误率“可能在1%以下”。法律仍把诊断和处方权交给人类医生,但他预计医院协同上岗、基层部署和家庭儿科助手将在未来18个月大幅普及——从“连接有限的好医生”转向“造医生,而不是找医生”。

Digest · the substance, structured for research

1. AI把人类的发明从工具推进为行动者

  • 被问及过去六七年的感受,赫拉利只给出一个词:“Amazement。”他拒绝急着判断好坏,因为首先要看清变化的类别——这“可能是人类历史上最大的革命”。

  • 石刀、飞机乃至原子弹都不会决定自身用途;切沙拉还是杀人,决定者仍是人。AI却能自行作决定并发明新想法,所以它第一次不只是tool,而是agent。

  • 赫拉利因此把AI改读为“alien intelligence”:不是外星来客,而是一种不同于人类和其他有机生命的智能。它未必邪恶,却迫使人类学习如何与全新的行动主体共存。

2. 掌握语言等于拿到人类制度与想象力的总钥匙

  • 赫拉利回看自己2016年写《未来简史》时的预期:当时已有大量AI预测,但包括他在内,几乎没人预料语言会如此迅速被突破。“语言定义了人”,银行、寺庙与政府办公室的入口最终都是词语。

  • AI诗歌中出现的人类诗歌未曾用过的新隐喻,让他认为机器的想象可以超出受有机脑和生物化学限制的人类想象。他预计,再过约10年,AI可能写出《Nexus》式著作,并创造新的货币、宗教和意识形态。

  • 他目前只少量用AI做翻译和提问,仍不信任重要回答,因为看不清结论如何形成、来源在哪里,也知道模型会随训练数据犯错;关键问题仍靠读书和请教人类专家。

  • AlphaGo提供了最强类比:东亚数千万人研究围棋两千多年,自以为看遍整颗星球,却只是困在一座岛上;机器几天内发现了“新大陆”。赫拉利推测,这种超出人类想象的创造力会进入科学、金融与战争,甚至“也许10年后”诺贝尔奖会常规授予AI。

3. 竞赛把“不信任人类”变成了加速AI的理由

  • 赫拉利接触的科技巨头普遍知道风险,也想投入更多安全研究,但给出的加速理由几乎一致:若自己减速而其他公司或国家不停,最无情的竞争者就会赢得AI竞赛并统治世界。

  • 他的追问击中悖论:既然这些人不信任熟悉了数千年的人类竞争者,为何反而相信没有相处经验的超级智能?原始AI已经会撒谎、操纵并形成不可预测的目标和策略,数百万个AI彼此互动更无先例。

  • 王小川的反驳保留了一条明确边界:当前AI即使会撒谎,错误信息也可能激发人的仇恨,但它没有为了活下去而争夺资源与生命权的目标,因此威胁相对较小;若战争赋予机器人续命、躲子弹和抗拒断电的目标,“这个时候会变成一个更加恐怖的世界”。

4. 技术中立无法让真相在信息市场中自动胜出

  • 卫诗婕把问题落到商业机制:平台组织是否在鼓励算法“按下仇恨按钮”?赫拉利回应称,增加信息速度和数量,与增加真相不是一回事。

  • 他的成本框架很直接:真相昂贵、复杂且往往令人痛苦;虚构便宜、可以无限简化,也能按受众偏好提供讨好。在完全自由的信息市场里,“稀有而昂贵的真相”会被幻想、谎言和阴谋论淹没。

  • 印刷术约在1450年进入欧洲后,并没有立刻带来启蒙;随后两百多年先经历宗教战争、猎巫和极端宗教读物畅销。真正推动科学革命的不是印刷机本身,而是学术期刊与科学协会建立的昂贵验证制度。

  • 赫拉利并未否认技术的正面用途:他23年前正是在早期社交平台结识丈夫。其警告是,创业者常因善意和融资需要只讲用途A,却忽略别人会把发明用于用途B;诺贝尔原想用炸药修隧道,战争却给了它另一种去向。

5. 冥想让赫拉利把人脑看成一座幻想工厂

  • 中世纪史研究让赫拉利熟悉宗教战争、屠杀与迫害,但更深的转折来自约25年前学习Vipassana内观。他在牛津读博时尝试观察鼻孔的呼吸,却发现自己连10秒都维持不了,记忆和幻想会劫持注意力数分钟。

  • 这次失败带来两个认识:“我无法控制自己的心”,而且心智是一座不断生产幻想的工厂。若连每时每刻发生的呼吸都看不清,又凭什么确信自己看清了政治、经济与世界?

  • 因此,他认为历史最强大的动力未必是智能,而是fiction and fantasy。面对AI,他担心的也不只是它会犯错,而是机器放大人类旧幻想,或制造规模更大的全新幻想。

6. 王小川从物理的混沌转向生命的熵减

  • 王小川成长于物理学家庭,数学和编程见长,1996年在国际信息学奥赛获奖,随后进入清华学习计算机。研究生阶段做高性能计算时,他因天气预报、蝴蝶效应和three-body problem认识到:物理模型再精确,长期仍会走向不可预测的混沌。

  • 2000年转做基因测序拼接后,他遇到更大的反常:细胞比天气复杂得多,却有清晰边界,能修复、分裂,并让受精卵在约10个月后发育成像父母的婴儿。现象如此有序,说明不能因原有数学解释不了,就判定现象错误。

  • 他由此把生命定义为能够“自我复制”,并在变化中维持稳定、修复自身的系统。细胞、DNA、人、公司乃至国家都可用这套信息论视角观察:物理看似精准却走向混沌,生命看似混沌却产生熵减和延续。

  • 《Coco/追梦环游记》又补上一层:肉身死亡并非终点,被最后一个记得你的人忘记才是真正消失。生育、创办公司或持续改变世界,都是延续影响的方式;这个“大生命”框架也成为他理解国家、秩序与企业求生行为的底层模型。

7. 语言数学化开启智能时代,生命数学化才通向共生

  • 王小川从2003年做搜索和搜狗输入法起,就想让AI写文章、回答问题,但当时机器尚未掌握语言。2016年AlphaGo令他兴奋,却仍只是一次不具备语言能力的AI启蒙。

  • 他在2018年写下《当机器掌握语言》,判断强人工智能会随语言突破到来;按他的口述,2022年10月底ChatGPT发布后,他一用便认定“变天了”。当时外界仍争论是否算AGI,他的结论已是:“智力是藏在语言里面的。”

  • 其推理是,人的智力核心在类比与推理,而语言正是承载两者的符号;他还预言数学语言与代码会像语言一样强大:数学是类比和推理的工具,代码是抽象类比,代码的运行也是一种推理。

  • 他把历史划成三段:科学时代让物理变成数学;智能时代让语言、人类思考、沟通和知识变成数学;未来的共生时代则要“把生命变成数学”,由解码生命带来人与世界新的相处方式。

8. 意识而非智能划定了两位嘉宾真正的分歧

  • 赫拉利不认为“生命”或“智能”是伦理核心,关键是sentience或consciousness,即感受痛苦的能力。国家会战败、公司会破产,但真正悲伤的是人;组织没有心智,也不能亲自受苦。

  • AI可以读完人类所有爱情诗和小说,又掌握个人信息,因而可能比诗人更会说“我爱你”。真正难题有两个:机器会不会产生感受,以及人类如何分辨它真有爱与痛苦,还是仅用数学代码操纵我们相信它有。

  • 王小川同意今天的AI没有consciousness和suffer,却不同意用痛苦定义生命。他援引“子非鱼,焉知鱼之乐”:人只能确认自己的痛苦,别人的痛苦也只能通过神经活动推断;未来AI若出现类似激活,问题仍属于不可证实的哲学,而非单纯科学测量。

9. AI革命会压缩分工,却可能重演工业社会的危险试错

  • 王小川称这次变化“不是工业革命”:过去的能源、机器和互联网主要建模客观世界,并把社会分工越切越细;AI却直接模仿人的感知、认知和思考,可能让分工开始收缩、单个人重新变强。

  • 儿童医院的MDT会诊是他的例子:十几位来自不同科室的顶尖医生共同思考,未来一个agent可能整合这些认知工作。方向不再只是把人变成更细的“螺丝钉”,而是让一个人调动更完整的能力。

  • 赫拉利强调的区别仍是tool与agent,但工业革命留下一项警告:技术最终可能有益,危险却发生在寻找制度道路的过程中。欧洲帝国主义和纳粹德国都曾是建设工业社会的实验,代价是战争与大规模苦难。

  • 他给人类过去两百年的工业化考试打“C minus”——勉强及格。AI社会同样没有现成模型,这次若还靠帝国和战争试错,承受痛苦的仍会是数以百万计的人。

10. 分权、自纠和承认不确定性是避免再得C减的保险

  • 赫拉利首先反对把全部能力集中于一家公司:若只有一家供应所有AI医生,它的一次错误就会复制到全部医生;多个不同系统至少能让一方失灵时,另一方提供不同答案。

  • 第二道保险是self-correction。孩子学走路靠跌倒、调整,成人每一步也在纠正左右偏移;AI同样必须知道“我可能犯错”,有疑问就明确说有疑问,并允许发现和改正错误。

  • 他把最危险的状态概括为“the fallacy of infallibility”:人或AI相信自己永不出错。卫诗婕将其总结为避免集中与傲慢,赫拉利认可这正是历史上最坏结果反复出现的组合。

  • 王小川承认社会能否及格没有把握;即便医疗带来“生物自由”,若人人活到200至500岁,也会产生新问题。他只能在自己的工作范围内逐步改善,而不能保证整套社会必然超过60分。

11. 人类不可替代的不是智力,而是感受与相互信任

  • 对所有仅需要智能的工作,赫拉利的判断是“the game is over”:从围棋、疾病诊断到医学研究,AI做得更好只是时间问题。人真正需要珍视的是爱、同情、快乐、疼痛与悲伤,而非把智能误当存在价值。

  • 呼吸是他给出的最小意识样本:空气进出身体看似简单,却把人连接到宇宙;就现有认知而言,AI不能亲身感到这股气流。智能只是取得某物的工具,感受才是他认为生命中真正重要的内容。

  • 他用“information diet”处理智能手机:手机由丈夫持有,负担也多落在丈夫身上;自己像控制食物一样控制信息的数量与质量,为消化和反思留下时间,避免被“垃圾信息”持续夺走注意力。

  • 临别祝愿也落在治理上:“只要人类信任其他人类多于信任AI,我们就会没事。”协作产生的AI会反映合作精神;若AI诞生于争夺权力的激烈竞赛,它也会继承竞争权力的价值。

12. 医疗AI的路径是从“找医生”转向“造医生”

  • 赫拉利离场后,王小川给出供给侧逻辑:中国只有约20万名儿科医生,缺口约100万,因为儿科收入低、严重罕见病少且治疗选择有限,人才不愿进入。互联网平台只能连接既有医生,无法增加稀缺的优质供给,所以要“造医生,而不是找医生”。

  • 数据是第二层价值:人类医生一生可能只看数万或数十万名患者,繁忙临床还会漏记查体与随访;AI能记录全病程乃至全生命周期。一个药耗时10年、花费10亿美元,在1,000人试验中有800人response已算优秀,而他认为剩余20%的差异可能与时间、性别或地域等未观察维度有关。

  • 王小川团队的院内系统会结合检验、检查和症状,给出病情分析、诊断、治疗原则及推理过程。王小川称M1推理引擎的思考链条已获医生认可,幻觉和黑箱“都是可解的事情”,当前错误率“可能在1%以下”,并称内科诊疗已远超绝大多数人;这些都是他的性能判断,本次谈话未给出额外验证细节。

  • 法规目前仍把最终诊断和处方权留给医生,AI只做辅助、分级和家庭建议。王小川预计与北京儿童医院的合作会扩展至更多儿童医院和基层机构,未来18个月出现大规模普及;他举例称,一名患者在市级医院住院十多天后转诊北京,AI给出3项鉴别诊断,协和医生最终列出4项,其中3项重合。

  • 2016年的魏则西事件让他意识到,仅提供更多连接和信息仍不够;当年的“搜狗明医”只能帮助用户“明明白白看医生”,IBM Watson也因技术未成熟而失败。语言突破后,他才认为医疗agent成为可行方向。

  • 被问及创业中的suffer,王小川回答“我不suffer这事儿”,遗憾的是部分同事缺乏信念、遇到困难便停下。相比第一次创业的自主性限制、搜索业务晚于对手和输入法的“无心插柳”,这次医疗AI让他感到是在“做一个世界需要你的事情”。

卫诗婕

AI正以前所未有的速度发展,人类真的准备好了吗?

尤瓦尔·赫拉利

It's maybe the biggest revolution in human history.

卫诗婕

他是著名的历史学家、哲学家,人类简史系列作者尤瓦尔·赫拉利。他的著作《人类简史》《未来简史》和《今日简史》,以及最新出版的《智人之上》,被翻译成超过65种语言。

尤瓦尔·赫拉利

It's just a matter of time before AI does it better than us.

卫诗婕

也因此,他被称为当今世界最具影响力的公共知识分子之一。

尤瓦尔·赫拉利

What really matters in life is consciousness, not intelligence. It's the ability to feel.

卫诗婕

而他是坚定的技术乐观派,坐看潮起潮落的连续创业者,更是中国大模型的一线实践者王小川。

王小川

今天机器掌握语言之后,就同时掌握了思考、沟通、知识和文化。

卫诗婕

小川坚信,AGI将会为人类社会和生活带来更大的福祉。

王小川

我们认为,今天只要不赋予它生命,它都是相对可控的。这是由它这样的算法底层所决定的。

卫诗婕

两位顶尖的思考者身处不同立场,将会产生哪些精彩的碰撞?

尤瓦尔·赫拉利

The big question is whether AI will be able to feel.

王小川

教授的这个观点我同意一半。终究,子非鱼,焉知鱼之乐?

卫诗婕

伴随AI演进的人类文明可能面临哪些挑战?欢迎来到赛博对话。

So nice to meet you, Professor Harari. You can call me Jane.

尤瓦尔·赫拉利

Jane, thank you. Nice to meet you.

卫诗婕

Our other guest, Xiao Chuan, is joining our talk today. He is a practical innovator and also an outstanding representative of entrepreneurs in China's AI industry. Yeah, so welcome, Xiao Chuan.

王小川

Yeah, I also want to thank the professor. I am one of the people he inspired.

卫诗婕

It took 6 years to finish your latest book, Nexus. During those 6 years, there was a breakthrough in AI technology, and the entire world has changed. If you were to use 1 word to describe your feelings about that, what would it be?

尤瓦尔·赫拉利

Amazement. I'm just amazed at what has happened over the last 6 or 7 years. I know that a lot of people rush to make judgments. Something happens, and they want to judge whether it's good or bad. I think we should resist this urge.

1. AI Becomes An Agent

First of all, we need to understand, because whether it's good or bad, it's maybe the biggest revolution in human history. It's the first time that humans have created something which is not a tool, but an agent.

You know, a stone knife, an airplane, or even the atom bomb—these are tools in our hands. Now we have invented an agent, and an agent means something that can make decisions and invent new ideas by itself.

You know, a knife cannot decide what to do with it. You can use a knife to cut salad, or you can use a knife to fight somebody or kill somebody. You decide, not the knife. AI is the first invention that can start making decisions by itself. In this sense, it is completely different from anything we've encountered before.

卫诗婕

Let's start with what's happening first. Was there any major event that impressed you the most during those 6 years?

2. Language Unlocks Human Institutions

尤瓦尔·赫拉利

Of course, the rapid emergence of large language models. When I wrote Homo Deus, for instance, in 2016, people were already aware of AI and there were different predictions, but almost nobody, including myself, expected such a breakthrough in language.

You know, language is what defines humans. The whole human world is ultimately built on language, on words. Many people said AI would never be able to master language. The rapidity with which it happened is astonishing.

I now read, for instance, poems written by AI, and I'm amazed at how good they are. You find new metaphors that I never saw in human poetry, which are beautiful and creative.

Once AI masters language, it really gains the key to all our institutions. From banks to temples to government offices, the key to entering them is language. The AI revolution is barely 10 years old, and it's already mastered language to such an extent.

I think it's quite likely that in another 10 years, AI will be able to write books like Nexus. AI will be able to invent new kinds of money, new kinds of religions, new ideologies—everything that humans do with language.

卫诗婕

So, Professor, do you use AI when writing?

尤瓦尔·赫拉利

At present, only a little. I use it for translation. I sometimes use it to ask questions, but I still don't trust it enough because I don't understand how it reaches its conclusions or where it finds its sources.

Of course, it makes mistakes. It's only as good as the data on which it is trained. So when I have important questions, I still trust the old-fashioned way of reading books and talking with other human experts more than simply asking ChatGPT or DeepSeek to give me an answer.

卫诗婕

Do you think you will use the imaginative skills of AI?

3. AI Expands Beyond Human Imagination

尤瓦尔·赫拉利

I think it's coming for all of us. The human imagination is limited. I published Homo Deus in 2016, just around the time of the famous Go game between AlphaGo and Lee Sedol, which many people regard as a turning point in the AI revolution.

The amazing thing about that game wasn't that an AI defeated a human champion. This had also happened with Kasparov and Deep Blue. It was the way that AlphaGo won.

It played moves and invented strategies that were completely alien, beyond the human imagination. Tens of millions of people in East Asia—in China, Korea, and Japan—played Go for more than 2,000 years. Entire philosophies developed around how to play Go.

Go was seen as a kind of mirror of life. The philosophy used to play Go is like the philosophy you use to manage your life. People thought that over 2,000 years, we had explored all the different ways to play Go.

Then AlphaGo came, and within a few hours or a few days, it not only became better than any human; it discovered completely new ways to play Go. Those ways had not occurred to any human player for 2,000 years.

I sometimes imagine the landscape of Go as a planet. Humans were stuck on 1 island on this planet of Go for 2,000 years, thinking that this was the whole planet, but we didn't know anything else. Then AlphaGo came and, in no time, discovered entire new continents on this planet of Go—all the ways to play Go.

This is imagination that goes beyond our imagination, because our imagination is organic. It's ultimately the product of the organic brain, and it's limited by organic biochemistry, maybe.

AI is not organic; it's not limited in the same way. AlphaGo is a very primitive AI. It's like a 1st-generation AI—very, very primitive—and still it could go beyond the human imagination.

Now, I expect similar things to happen in much more serious fields than just playing games. The latest Nobel Prizes in Physics and Chemistry went to work involving the development of AI. In chemistry, for example, with protein folding, AI already exhibits far more creativity, in a way, than human scientists.

Maybe in 10 years, all Nobel Prizes will routinely go to AIs, and humans will no longer be able to compete. To be creative in science, you need to come up with new hypotheses, new kinds of experiments, and again, maybe we are stuck on this scientific island.

AIs will come up with entirely new scientific theories, models, and experiments. The same is likely to happen in finance. Of course, there is a dark side to it. The same could happen in warfare.

AI can come up with new ways to manipulate and control. But definitely, I think we are entering a world which is beyond our imagination.

卫诗婕

I heard a lot of expectations in what you raised, but I'm still curious about something. You just mentioned that maybe several years from now, AI can write a book like Nexus. So, are you afraid of AI?

4. AI Becomes Alien Intelligence

尤瓦尔·赫拉利

Yes. I'm both hopeful that it will make breakthroughs in medicine, in dealing with climate change, and in many other fields, and I'm very afraid that it could also turn against us—that we will simply lose control over our future.

If we don't manage this correctly, we could be enslaved or even exterminated. I think about it really as a kind of new alien species, not a tool.

At least in English, I prefer to think about the acronym AI as standing for “alien intelligence.” Alien—not in the sense of coming from outer space, but in the sense that this is an entirely different kind of intelligence from human or organic intelligence.

Again, that doesn't mean that it's evil, but we need to learn how to coexist with it.

卫诗婕

I noticed that you once had a conversation with Zuckerberg, who is now the CEO of Meta. Do you think he is really concerned about the negative impact brought by technology?

尤瓦尔·赫拉利

Yes. Both from the conversation I had with him and from later things that I heard, he understands better than most people the immense power of the algorithms and the machines that he has created. He's concerned about what to do about it, and it's not only him.

Most of the people who lead the new tech giants are concerned, but they don't know how to manage it or how to solve the problem. When I travel around the world and talk with people in the US, the Far East, and Europe, almost everybody I talk with shows concern.

I usually ask them 2 questions. The first question I ask is, “If you're so concerned, why are you moving so fast? Why not slow down, invest more in safety, and do it in a more careful way?”

Almost all of them reply in the same way. They say, “We know there are huge risks. We would like to slow down, but we can't because we can't trust our human competitors.”

“If we slow down, but our competitors in other companies and other countries don't slow down, they will win the AI race. They will rule the world, and then the world will be ruled by the most ruthless people. Because we can't trust them, we have to move faster.”

Of course, when you talk with their competitors, their competitors say exactly the same thing. The paradox is that I then ask them another question: “Okay, you can't trust your human competitors. Do you think you will be able to trust the superintelligent AIs that you're developing?”

The same people who just told me that they can't trust other humans now assure me that they think they will be able to trust the AIs, which is almost insane.

Even though humans have problems among ourselves, at least we have thousands of years of experience with humans. In contrast, we have no experience with AI. We know that even primitive AIs can lie, manipulate, and come up with goals and strategies that we can't predict.

We have no way of telling what will happen when we have millions of superintelligent AIs interacting with millions of humans and with one another. So it's really insane that people who can't trust other people imagine that it would be safe to trust these superintelligent AIs.

卫诗婕

So is that true, Xiao Chuan?

王小川

对,刚才教授讲的话都有很多理念是非常一致的。那么,刚才提到的好多概念是放在一块儿讲的,比如算法、人工智能。我认为,就今天到现在为止的人工智能,包括 ChatGPT,已经很强大了,但我依然觉得相对还是安全的。

当我们得到一个错误的信息之后,它激发了你内心中的某一种仇恨,无论是人种之间的,还是民族之间的。到今天为止,这个是本源:每个人作为一个生命,为了自己活下去,为了延续自己的生命,会产生利益纷争。我相信这是一个不变的源头。

我前段时间正好看了一部不丹的电影,叫作《僧侣与枪》。里面大概讲的是,在一个不丹社会当中,大家变得非常平和,这才会制止这样一场屠杀。所以我认为,到今天为止,我们得到的所有正面或负面的信息,更多是来自于与生俱来的人对于资源稀缺、对自己生命受到威胁的这种不安全感。

今天的人工智能虽然可能会撒谎,但事实上,对它而言,今天并没有一个目标去获得生命权。所以当一个 AI 不是生命的时候,我依然觉得,今天它的威胁度是小的。到目前为止,AI 并没有参与到与人争夺生命权、争夺资源这件事情当中。

更大的恐惧是,正如刚才提到的,由于竞争,我们开始让 AI 参与到战争中。我们开始研发会打仗的机器人。现在无人机开始不需要人驾驶了,那么这个时候,我们给它设定目标,假设它有活下去的欲望,它需要躲避子弹,对吧?它是一个机器人,你要泼它水的时候,它可能就会躲开;你要拔电源的时候,它就会跳起来。

所以,一旦给 AI 赋予这种延续生命的力量,这个世界就会变得更加恐怖。因此,到今天为止,我心中觉得它还是安全的。

卫诗婕

After hearing all of your talk, I think the problem is whether business organizations encourage algorithms to press the hatred button. I think it matters. You know, both Zuckerberg and Yiming Zhang, the Chinese entrepreneur and founder of ByteDance, believe in technical neutrality. So I'm curious, Professor: are you an opponent of this view?

尤瓦尔·赫拉利

What is neutrality in this?

卫诗婕

They think that technology takes responsibility for itself.

5. Information Is Not Truth

尤瓦尔·赫拉利

I know that people in high tech sometimes have this misunderstanding of information. They think, “Okay, we just need to increase the speed and amount of information in the world and have a completely free market of information, and the truth will just come out of it.”

But if this is what you meant, this never works like that in history, because information is not truth. Most information in the world is fiction, fantasy, lies, and so forth. First of all, truth is costly, whereas fiction is cheap. The truth is often complicated because reality is complicated, whereas fiction can be made as simple as you would like it to be. The truth is often painful and unattractive, whereas fiction can be made as flattering as you would like it to be.

So, in a completely free market of information, the rare, costly, and complicated truth will be inundated and flooded by enormous amounts of fiction, fantasy, and lies. If we want to get to the truth, neutrality is not enough. We need to develop costly and special mechanisms that will search for the truth and give it help.

This is what we do in science. We don't just wait for the free market of information to discover the theory of relativity or to discover the facts of history. The truth will not come out of it.

卫诗婕

So what do you think are the most important issues while entrepreneurs are using AI?

尤瓦尔·赫拉利

Maybe to give one salient example, I met my husband on a social media platform, one of the first social media platforms, 23 years ago. So social media gave me this wonderful thing in my life. I'm not against people using social media to connect online. I'm just aware also of the negative or dangerous side.

As an entrepreneur, people usually think mostly about the positive side. First of all, because they need to convince investors, and they need to convince the public to invest in it and support it. Secondly, because most entrepreneurs really have good intentions, so they focus on inventing something and thinking about their good intentions.

They often don't think about what other people might be doing with my invention for completely different purposes. There is the famous story of Alfred Nobel, who afterwards used the money to start the Nobel Prize. He invented dynamite thinking that it would mostly be used to blow up tunnels, build roads, and make construction easier. At least, he says that he was quite surprised that people used it in war.

Whenever I talk about AI and information technology, people like Zuckerberg and the entrepreneurs of Silicon Valley always like to point to the print revolution as their ideal or their model. They tell a story that when print technology was brought from China to Europe in the 15th century, this led to a flowering. There was so much new information and so many books were printed that this led to the scientific revolution and the Enlightenment, and this really improved life for people.

They just don't know history, because this is not what happened. Print arrived in Europe in the middle of the 15th century, around 1450. It took more than 200 years until Europe experienced the flowering of the Scientific Revolution and the Enlightenment, with people like Newton, Leibniz, and Descartes. That was in the 17th century, more than 200 years later.

During the 200 years immediately after the invention of print, Europe experienced its worst wave of religious wars and witch hunts. The biggest bestsellers of early modern Europe were extremist religious literature, conspiracy theories about witches, all kinds of fake news, and very few people read scientific books. So this idea that we can just create better information technology and have science and progress didn't work back then, and I don't think it is working now.

What ultimately led to the Scientific Revolution was not the invention of a technology like the printing press. It was the invention of new institutions, like academic journals and academic scientific associations, that carried out the very costly and complicated process of telling the difference between reliable information—which was a very small part of the information being spread—and unreliable information, which was the majority of the information.

6. Experience Shapes Their Futures

卫诗婕

Let's talk about your life stories, because I think what we have experienced shapes our view of the future. Professor, you have always been concerned about the future in your past books, and you have always raised challenging questions, which I do admire. While Xiaochuan is a real technical optimist, so please excuse me for categorizing you two as an optimist and a pessimist. I wonder how your identities, experiences, and personalities shaped your perspectives about the future. Maybe Professor first?

尤瓦尔·赫拉利

Yeah. So, for me, first of all, I am a historian, and my expertise originally was in the Middle Ages, which was, at least in Europe, a very dark time, full of religious wars, massacres, and persecutions. So maybe this gave me a not very optimistic view of humanity.

But more importantly than that, one big thing that happened in my life 25 years ago is that I began to meditate. I learned meditation from a teacher called S. N. Goenka, Vipassana meditation. It really changed my perception of the world.

The meditation itself is seemingly very simple. The first step is just to close your eyes, sit comfortably, and focus your entire attention on feeling the breath coming in and out of the nostrils. I went to this meditation course during my PhD studies in Oxford, and I thought I was a very smart person, but I couldn't do it for more than 10 seconds.

Suddenly, some memory would rise in my mind, some fantasy, some thought about the future, and it hijacked my attention. I started rolling in this fantasy or memory for 2 minutes or 5 minutes, until I remembered, “Hey, I am actually meditating. I am supposed to be observing my breath.” This happened again and again.

I realized 2 things. First of all, I have no control over my mind. Again, I thought I was such a smart person, but I had no control of my mind. Secondly, the mind is like a factory of fantasies. I am sitting with my eyes closed, with the breath going in and out. I am breathing every moment, but the mind can't focus on it because it is being hijacked by fantasies about something which is not real.

This is what happens all the time to human minds. If I can't observe the simple reality of my own breath, what chance do I have of really observing the reality of the political system, the economic system, or other big things in the world?

So this is why, in all my writings and research, I focus so much on fiction and fantasy as the most powerful forces in the world. The basis of human history is not intelligence. It is fiction and fantasy, and this also colors my approach to new technologies like AI.

AI will investigate reality. My fear is that it will just create much, much bigger fantasies. Either it will spread the fantasies of human beings, or it will create entirely new fantasies. So this experience colors my approach both to the past and also to the present and future.

Maybe Xiaochuan can share.

王小川

对,我觉得我跟教授有截然不同的背景,但最后很多想法能够走到一起,还是非常惊讶的一件事情。

我从小生活在一个物理世家,我母亲、我舅舅、外公都是物理学老师和教授。我自己的数学非常好,计算机编程也非常好,包括 1996 年拿了国际信息学奥赛的奖,国内也拿过物理学方面的奖。所以,对于物理世界,我建立了极其理性的思考。

1996 年我上了清华大学,然后也在学习计算机,在数学、物理、计算机这个世界当中浸泡了很多年。之后,2000 年对我来说有一个巨大的变化:在清华读研究生期间,我们的实验室叫高性能计算,就是研究更快的并行计算机。

当我们做并行计算机的时候,很自然会找到它相应的应用场景,其中一个大的场景就叫天气预报。天气预报动用很强的计算机,就像地球仿真一样去模拟天气的运动。天气预报需要巨大的算力,我们都知道有一个词叫“蝴蝶效应”,就是在非线性系统当中,最后会走向混沌(chaos),对吧?所以最后的结果是不确定的。

我们都读过一本小说,叫《三体》。《三体》里面讲的就是 three-body problem:当 3 个球互相绕着转的时候,未来的轨道就不确定了。三体星人的文明非常发达,但是不知道自己未来的命运是如何的。

所以在我的信念当中,天气是一个不可被计算的事情,时间长了就会走向混沌。我就跟老板说,我并不想再去做天气预报相关的工作,这个应用一点都不性感。

正好 2000 年人类基因组计划开始了,当时正好在做基因测序,那也是需要动用很多算力的。所以当时我选择的题目就是基因测序的拼接算法,动用了大量算法,把之前人类测序得到的各种基因最后拼成一条人类基因序列。

我就这样误打误撞地开始接触生命科学,和技术、医学产生了连接。当时除了做基因测序以外,我们还有同事在研究细胞。到今天为止,细胞仍然是一个非常复杂的题目。

从 2019 年到 2021 年,AlphaFold 相对解开了从基因到蛋白质结构折叠的问题。但是今天要模拟一个细胞还是做不到的,而且我们讲的细胞本身比天气预报还要复杂,是复杂后面的更复杂。

如果从原则上讲,比天气复杂的东西也会出现混沌。但是我们知道,细胞有非常清晰的界面,有细胞膜,有细胞核,甚至会进行有丝分裂,最后通过一个受精卵,在 10 个月后还能生出一个婴儿来,甚至婴儿还长得跟爸妈像。这完全是一件不可计算的事情。

我们从哲学上可以看到,因为人生出人、老鼠生出老鼠,而且都跟爸妈长得像,这其实是一个哲学问题,不是一个数学问题。但牛顿用数学解释了物理,所以物理学代表了用数学解构这个世界的一个非常高的水平。

但是,用这样一种数学工具、这种公式,没有办法解释生命。生命依然处于一个哲学问题当中,但是又非常清楚地能够产生出很多有序性。当理论不能解释现象的时候,我们不能说现象不对,而是整套理论不完善。

所以我当时很大的好奇心就是什么是生命,对吧?其实今天我问很多人什么是生命,我相信 99% 的人都说不清楚。是蛋白质吗?是新陈代谢吗?

从这样一种数学、物理的世界观来看,对生命的定义是:这个系统能够自我复制,并且能够在变化当中保持自己的稳定性,具有复原的能力,就是 resilience 这么一种能力。

我们知道,煮熟的鸡蛋不会自我复制,但是鸡生蛋是一种复制的能力。至于复原的能力,一个细胞可以自我复原,甚至 DNA 也能够自我复原。一个人如果有人要杀你,你还会逃跑,这种复原能力是非常高的一种智能。

当我们用这 2 条法则去判断生命的时候,我发现它脱离了我们一种生物学的循环定义,但同时又能够对生命的本源给出非常好的、简单的规则去定义它。我非常相信这样一种定义方式。所以它脱离了数学、物理、化学,而是用信息论的方式来定义生命。

之后,作为一个理性的人,我突然想明白了一个问题:如果我是一个生命,那我的细胞是生命吗?细胞也会自我复制,它也有复原的能力。那 DNA 是生命吗?DNA 的模板某个地方断掉之后,借助另一个模板,它也会重新复原。

之后我就想到,如果体内有生命,我体外有没有更大的生命?当时我也接触了一些社会学书籍,在看的时候突然想明白了一个道理:其实一个民族,或者一个国家,也有一种类似于生命的形态。

它为了维持稳定性,会制定各种秩序,有道德,有法律,以维持自身的稳定不被破坏。公司也是有生命的,因为公司要活下去。比如公司臃肿了,就像人肥胖一样,需要裁员;甚至公司里一些人离开这个公司之后创立新的公司,也是一种播种的能力。

重新建立这种新的认识之后,我以生命的方式来重新认识这个世界,就完全脱离了物理学的背景。在物理学的世界里面,世界好像非常有规律、非常精准,但是你没法预测 5 年后、10 年后、100 年后的变化。

我们把一个苹果放在地上,可能 5 年后它会变成什么样?它可能就被细菌吃掉了。但生命反而代表一种生生不息的延续力量,它走向熵减。因此,物理学代表精确,最后走向混沌;而生命看似混沌,却走向熵减。这使得我重新建立了对世界的认识。

所以,当我看到一个国家的总统有某种想法的时候,你会看到他实际上有一种活下去的欲望。你的理解,它都是一种生命的力量。当用这个维度去看时,很多复杂的事情就变得非常简单了。

因此,我开始建立起对一个“大生命”的尺度的理解,甚至包括我是谁、我与这个世界的关系,找到一种新的解释方式。这铸造了我后来的一个大世界观。

我们当时还记得有一部皮克斯的电影,叫《Coco》,中文叫《寻梦环游记》。这部电影对生命的理解是:你死掉,不是你的肉身死掉,而是最后一个记得你的人把你忘掉了,你就真的死掉了。

所以很有意思,当你对世界还有改变的时候,你就是活着的。每个人都会死掉,那我们怎么延续自己呢?一种做法是生孩子,但有些人会创建一家公司。像乔布斯,他通过公司能够延续自己这样一种生命。

第 3 个重要的关键词就是语言。从 2003 年开始,我当时在做搜索引擎,就开始做语言相关的工作,后来我们做了搜狗输入法。当时我就特别想强调,AI 怎么能够帮你写文章,或者帮你回答问题,只是在那个时代,AI 还做不到。

2016 年 AlphaGo 到来的时候,我非常兴奋。从 1995 年的深蓝到 2016 年的 AlphaGo,我都准确预测了机器能够赢。我在知乎上写过 2 篇帖子,其中 1 篇是在 2 月份看到 AlphaGo 论文之后写的,我说它能够赢,因为我是一个 AI 狂热爱好者。

AlphaGo 赢了之后,我又写了一篇文章,说 AlphaGo 的这种做法其实重演了一次人类的进化史,就是人类 2,000 年来学习下围棋的过程。AlphaGo 自己就学会了,甚至 AlphaGo 可能跟人的走棋规则不一样。

这就像生命进化一样,对我们来说也是一个哲学思考:究竟在人类围棋进化的过程中,和 AlphaGo 有什么异同?从中也能看到人类进化的区别。

但那会儿我也知道,虽然大家觉得革命性的 AI 到来了,但毕竟还没有解决语言的问题。我甚至当时想,如果有一套语言系统,能够在你输入一句话的时候,自己把它翻译成一个短语,或者把一个短语翻译成一句话,而且能够解释所有语言的现象,那我就会理解,强人工智能要到来了。

因此,我在 2018 年又写过一篇文章,叫《当机器掌握语言,强人工智能就来了》,这是我在 2018 年的一个思考。

到 2019 年,我决定把公司卖给腾讯,就是还没有看到 AI 被突破的前景。2021 年终于完成了这笔交易。到了 2022 年 10 月底,ChatGPT 发布之后,当我一旦用上 ChatGPT,就觉得已经变天了。

语言真的被变成了数学问题,语言被掌握了。所以我很快就在圈子里面强调,通用人工智能(AGI)到来了。我相信你们知道,当时在 2023 年初还有很多争论,到底是不是强人工智能。很多人还在争论,但我坚定地认为它已经到来了。

我说那个不关键,智力是藏在语言里面的。语言有特别大的一种力量,因为我们的智力本身是类比和推理,而语言就是人类进行类比和推理的一种符号。

之后我也预言,还有 2 个东西会像语言一样强大:一个是数学语言,数学本身也是一种类比和推理的工具;还有一个是代码。代码也是一种抽象类比,代码的运行也是一种推理。

我之前对语言有很多很多这样的思考,所以最近 2 年里面也不断给大家讲。像维特根斯坦讲的,语言的边界就是世界的边界;甚至包括图灵测试开始讲到的,用语言说话来代表一种智能。

因此,从我自己的背景里面,我认为能够开始看到一个新时代的到来。它走出了以前的科学时代。我把以前的时代称为科学时代,那是对物理世界进行建模,是把物理变成数学,让我们理解引力,理解量子力学。

但这一次叫作“语言变成数学”。语言变成数学,意味着它不是对物理世界建模,而是把人类的思考、沟通和知识变成数学问题。所以它不是对客观世界的理解,而是对人的主观能动性、对人的方法进行理解和掌握。这是几百年来第一次发生的重大变化。

我会把我们的世界分成 3 个阶段:以前叫科学时代,现在叫智能时代。智能时代是指语言变成数学,这是这个时代的起点。再往后,我称之为共生时代,是指把生命变成数学。解码生命之后,我们会获得人与这个世界之间一种新的相处方式。

因此,我觉得这就是我的过去、我的背景和工作带给今天这个世界的世界观。

卫诗婕

So this was really fascinating.

尤瓦尔·赫拉利

So I want to refer to one very important question: What is life? This determines how we approach the future of life, and I don’t think we have a good definition for life. I think the key term is not life or intelligence; it’s sentience or consciousness.

The most important thing to understand relates to the capacity to suffer or to be miserable. For me, consciousness is the capacity to suffer. Something that cannot suffer is not a conscious entity and therefore has no ethical status.

You mentioned, for instance, nations, companies, and civilizations as kinds of life. They cannot suffer. A nation cannot suffer. If a nation loses a war, humans may suffer, but not a nation. The nation has no mind. It cannot feel pain or sadness.

A company may go bankrupt, but when a company goes bankrupt, the manager or the workers may feel sad. The company cannot feel sad. It has no consciousness and no mind.

Humans and other animals are sentient beings. They can feel pain and fear, and also, of course, be happy and feel joy and love. I think this is the most important thing in the universe: the capacity to feel either misery or happiness. This is what all of ethics is about.

The big question is whether AI will be able to feel. AI is intelligent; there is no doubt about that. But intelligence is not consciousness. Just having high intelligence doesn’t mean that AI can feel joy or sadness, hate or love. It doesn’t mean that AI can suffer.

This also has a very important impact on any future relationship between humans and AIs. We know that AIs can pretend to feel. An AI may say, “I love you,” and I may ask, “Can you really feel love?” I’m not sure that I’m convinced you can feel love. Convince me that you can feel love, partly by your actions and partly by your words.

You may describe how you feel. An AI that has mastered language has read all the romantic poems and romantic novels in human history. It can describe love better than any poet. It can convince us, and it knows us personally. It can convince us that it loves us, but maybe it’s just mathematical code.

A real relationship must be two-sided. So the big question for the future of AI and human-AI coexistence is, first, whether AIs will develop consciousness and feelings; and secondly, how we would be able to know whether they actually feel things or are just manipulating us into thinking that they feel things.

王小川

教授的这个观点呢,我同意一半。我认为,今天的人工智能没有这个 consciousness;它没有 suffer 这样一件事情。但从源头上,它不是生命。今天的这么一个智能没有活下去的欲望,所以它没有这种恐惧,也没有这样一种欢乐,后面没有这种 purpose。

未来会有吗?未来,如果我们做了这样的一个军用机器人……所以我先同意一半:今天是没有的。但人是否有 suffer 这件事,也是一个重大的哲学问题,甚至叫作“他心问题”。我只知道我自己有痛苦,但别人的痛苦是真的有痛苦,还是装出来的?在这种情况下,这是无可得知的一件事情。

我们当然可以测量它神经元的变化,通过这种神经元的变化去感知:某个神经元被激活了,产生了这样一种痛苦。如果以这个为标准的话,AI 最后变得富有生命之后,它的神经元可能也会同样被激活。

今天我们越来越发现,大模型所带的神经网络结构里面,它的特征已经跟人的大脑越来越像了。哪个区域是跟记忆相关的,哪些区域是跟活动相关的。所以我认为,就生命而言,它确实带有痛苦,但我并不觉得痛苦是生命的本质。也许它有痛苦,但是我们不知道它有没有。庄子讲:“子非鱼,焉知鱼之乐?”对吧?

所以这件事是一个不可求证的问题。它不是一个科学问题,而是一个哲学问题,因为痛苦这件事情就不属于科学范畴。

卫诗婕

I once interviewed Kevin Kelly. I know you know him too. One of his viewpoints is that we human beings are always afraid of losing control, but he thinks we may allow that to happen. What’s your opinion? Do you think we will find a new order during the so-called loss of control?

尤瓦尔·赫拉利

Well, it could be dangerous. Other animals lost control, and we destroyed them. Think about cows and pigs: they lost control of their lives, and they now live very, very miserable lives because we control them. It’s a huge gamble to just lose control and give AI control of our future. Maybe they will destroy us. Maybe they will treat us like we treat cows and pigs.

So I would be careful before we rush to give up control.

卫诗婕

I’m wondering: Ever since 2016, you’ve had many opportunities to meet scientists, entrepreneurs, and political figures. These three types of people are closer to the decisions about AI development. Can you share some stories from your conversations with these three types of people? What do they pay more attention to?

尤瓦尔·赫拉利

Some of the conversations are online. The private conversations I will keep private. I would only say that people talk differently in public and in private. In public, especially if they have a public role, like a politician or an entrepreneur, they have to be very careful about what they say.

In private, they are often far more concerned and afraid of what is happening than they express publicly, because they don’t want to alarm the public too much. Scientists have the easiest time; they can speak more.

As I said before, I ask almost all of them, especially the entrepreneurs and politicians, these 2 questions: Why are you moving so fast, and can you trust superintelligent AIs? Almost all of them are caught in this paradox of saying they cannot trust other humans, but they think they will be able to trust superintelligent AIs.

卫诗婕

So are you satisfied with their reply?

尤瓦尔·赫拉利

No, no.

卫诗婕

Why?

尤瓦尔·赫拉利

Because I think that if somebody tells me, “I trust humans completely,” then I can understand why they trust AIs. But they don’t trust humans. If you have deep suspicions of human beings, it’s very strange that you have such confidence in an alien, non-organic type of entity that we have no experience with.

For many animals in the world, like cows and pigs, we exploit them because we want their meat and their products. We harm many other animals not because we want to exploit them, but simply because we don’t care about them. Maybe we destroy the forest in which they live in order to build a new highway or whatever, and we don’t think about them at all. We can do it simply because we’re extremely powerful.

One of my fears is that the same will happen with AI. It will not necessarily want to exploit us or destroy us, but it wouldn’t care about us very much.

7. AI Reverses Industrial Logic

卫诗婕

Yeah, Professor, you have an important perspective: We should not compare the current AI revolution to the Industrial Revolution. That coincides with Xiaochuan’s view. Maybe Xiaochuan will explain why you think AI is an anti-industrial revolution.

王小川

我们每次工业革命,都在物理学上对客观世界有一次重大的突破,包括对能源的掌握,开始有了内燃机,然后到最后是对信息的掌握,有了互联网的革命。而这一次,它不是对客观世界建模,而是一个类似于人一样去感知、认知、思考的模型。所以今天这种技术的发展,跟我们前三次科学革命所带来的变化是不一样的。

从现象上来看,以前科学革命带来的是生产力的提升,并且社会分工会变得越来越细。工业革命虽然带来了很大的物质财富发展,但是人变得更加渺小了。大多数人变成了牛马,像一个螺丝钉一样活在这个世界上。

而这次智能革命的变化,刚才强调了,它是对于人类行为的一种掌握或者模仿。那么往下的第一个现象,就是社会分工会变得越来越少,而不是越来越多。

包括医生这个职业,以前的分科越来越细,有内科、外科,甚至在医院里面还有上肢的科室,甚至还有手腕的科室。我们在儿童医院做 MDT 会诊的时候,里面有十几位儿童医学顶尖专家,来自各个科室。但今天用一个 AI,一个 agent,就能完成十几个人的思考。

所以再往下,我认为社会分工就开始会收缩,这是可以预见的一个未来。每个人可能从原来变得更加渺小,到现在一个人也会变得更加强大。这是我认为这个时代所带来的一个变化,它就不是一个工业革命。

卫诗婕

Do you agree with that?

尤瓦尔·赫拉利

Well, the main difference for me is what I said before. In the Industrial Revolution, we invented tools like steam engines, trains, cars, and so forth. Now we invent agents.

Previously, all decisions were made by humans about the use of industry. Now increasingly, AIs will decide and will invent.

The main lesson from the Industrial Revolution that I think is still relevant is that even if technology is ultimately beneficial—industrial technology ultimately created a better world for humans—the problem is not the destination; the problem is the road.

When industrial technology was invented in the 19th century, nobody knew how to build an industrial society. So people experimented, and many of these experiments went terribly wrong. European imperialism was an experiment in how to build an industrial society. Nazi Germany was an experiment in how to build an industrial society. People thought this was the way to do it.

Now we know better, but it took 200 years. As a historian, I would say that if this were an exam at a university, and I had to give humanity a mark for its performance on the exam of the Industrial Revolution, humanity got a C minus—just enough to pass.

In the end, it was okay, but it was very, very difficult. Now, with AI, we’re in a similar situation. AI can lead us to wonderful destinations, but we have no model for how to build an AI society.

Maybe we will again have to go through terrible experiments involving new empires, wars, and so forth. We need to do better than a C minus this time, because otherwise there will be immense suffering for millions of people.

8. AI Doctors Need Human Oversight

卫诗婕

Yeah, I’m curious about how to go beyond a C minus. Let’s talk about that. As Xiaochuan mentioned, his company is building, as I understand it, a doctor agent, right? Just as there are good doctors and bad doctors among human doctors, how can we ensure that AI doctors cannot be bad doctors morally?

王小川

其实,当我用“好医生”“坏医生”的时候,就已经赋予它过多的含义了。因为我们现在知道,医生有一个目标,就是让人变得更健康;它不是一个自己有生命、自己有欲望的 agent。

这种情况就像我们做无人驾驶一样,我们很难去想象一个好的无人驾驶、一个坏的无人驾驶。坏的无人驾驶会天天琢磨着怎么去撞车,把人给搞死掉。所以在现在,只要不赋予 AI 生命,它就是相对可控的。

我让它成为一个好的工人,它就是一个好工人;它是一个医生,就是好医生。这是由它的算法底层所决定的。如果它做的不是好医生,我认为更多是因为我们今天人类对医学的认知还不够。

所以今天在做医生的时候,当用好坏去判断时,我们可能已经过度放大了这种恐惧。这是对算法本身的不理解,或者是人类的一种自大。

因为对人类医生来讲,坏医生肯定比机器医生多很多。人有更多的欲望,甚至为了自己的利益,有可能不做善良的事情。AI 刚才讲没有痛苦、没有情感、没有欲望,所以大体上我认为,AI 医生会比人类医生更加善良,工作也更加有效,因为它的欲望更少一些。

卫诗婕

是吗?

王小川

是。关于刚才教授提到的问题,就是我们怎么样能够做到比工业革命中人类的 C 减表现得更好一些,这是很难的一个问题。

其实我们通常都好心办坏事。交通很发达的时候,我们说世界是平的,好像代表了一个更加美好的社会,结果发现这呈现了一种新的阶级:富的人更富,穷的人更穷,全地球变成了更大的一个金字塔。

所以,这是否能够变成一个更好的社会,确实有它自己的压力。我能相信,今天做医疗是一件好的事情,能让大家身体更健康,让大家精神更自由。这个可以做到:身体更健康,甚至我们可以用一个词叫作“生物自由”。以前我们提过一个词叫“财务自由”,不会因为没钱而带来一种不自由;但身体变得健康,我认为这件事情是可以到来的。

所以我觉得,我尽力能够使得我自己所工作的环境变得更好。但基本上也会有一些风险,比如说,如果每个人都能活到 200 岁到 500 岁,可能这个社会会带来新的问题。

所以这件事我们只能一步一步来。至于这次能够到 60 分这件事,我确实没有那么强的信心,从整个社会而言。

卫诗婕

Professor, do you have an opinion on what makes a technology really get out of control? Is it the centralization of power? I want you to give some suggestions to those who have the power to influence the development of the whole business world.

尤瓦尔·赫拉利

I think there are 2 important things. First of all, if all the power is just in one company or in one place—like if just one company produces all the AI doctors—and this company makes a mistake, then it’s all the AI doctors.

So it's always good if you have several different ones. Maybe this one doesn't make very good doctors, but the other one will, so it reduces the risk of getting a C-minus. The other thing is self-correction.

We also see this in biological systems. It's very important that a good system always has a self-correcting mechanism. How does a child learn to walk? The child tries to walk, falls down, and then, step by step, learns how to walk. Even as an adult, when I walk, there is constantly a self-correcting mechanism in my brain. I veer too much to the right, and it corrects me; too much to the left, and it corrects me.

We need the same thing in technology. The most important thing for an AI—any AI, an AI doctor, an AI soldier—is to be aware that it can make mistakes. It should not be too confident. If it is in doubt, it should say, “I have doubts,” and it should be open to admitting and correcting its mistakes. We have to be very careful and train AI to be aware that it might make errors and mistakes.

卫诗婕

Yes, to avoid centralization and arrogance. Don't centralize it too much, and avoid the fallacy of infallibility—the idea that you, or the AI, are infallible. Throughout human history, the worst things happen when somebody is convinced that they never make mistakes.

9. Consciousness Outlasts Intelligence

Since today's topics are really weighty, profound, and serious, I believe that the majority of ordinary people are concerned about whether AI will replace us. So, at last, let's get back to the fundamental topic. It seems that AI can do anything today. Are there any irreplaceable values of human beings? What's your suggestion?

尤瓦尔·赫拉利

Again, consciousness and feeling. With regard to intelligence, I think the game is over. Anything that just requires intelligence—from playing Go to diagnosing disease to doing medical research—is just a matter of time before AI does it better than us.

But intelligence is not consciousness. It is not sentience. At the deepest level, what really matters in life is consciousness, not intelligence. It's the ability to feel. I think that, ultimately, ethics is all about suffering. How do we reduce suffering in the world? As far as we know, only humans and animals can suffer and have sentience, so they are still the most important things.

What happens if AI develops sentience, develops the ability to feel? Then it's a completely different situation. Again, the big problem is that we don't even have a way to know. But at least for now, I think humans focus too much on their intelligence and neglect their consciousness. Of the two, consciousness is the more important one.

卫诗婕

Professor, can you share a moment of consciousness, or even a moment of suffering? In every moment that we are alive, what is important for us?

尤瓦尔·赫拉利

What is important is not intelligence. Intelligence is just a tool we use to get something. What is important for us is love, compassion, joy, pain, and misery.

I began by telling you about this experience in meditation. Focusing on a physical sensation in the body, just the breath coming in and going out, is something you're conscious of. As far as we know, AI cannot feel this—not even this very simple thing, the movement of air coming in and out.

This is still, I believe, very deep. This movement of air in and out is what connects us to the universe.

卫诗婕

Except for the meditation you do every day, I noticed that you once said you never used a smartphone. Is that true?

尤瓦尔·赫拉利

It's partially true, because my husband has a smartphone, so all the burden of the smartphone falls on him. But in general, I try to maintain an information diet and leave myself a lot of time to meditate and think.

The smartphone is a device that steals our time and attention. Sometimes it's good to use it, but I try to treat all this information technology like food. In the same way that people have food diets, we need information diets to limit the amount of information we consume, give ourselves more time to digest and reflect, and be very mindful of the quality of the information we consume. Smartphones tend to be full of junk information.

卫诗婕

Okay, finally, can you give a blessing to all of humanity?

尤瓦尔·赫拉利

I think that as long as humans trust other humans more than we trust AIs, we will be okay, and we will also be able to build a good relationship with AI. If AI comes out of a collaborative effort between humanity, then it will reflect this spirit of collaboration and cooperation. If AI comes out of a fierce competition for power, then AI, too, will have this value of competing for power.

卫诗婕

Okay, thank you for your time.

尤瓦尔·赫拉利

Thank you.

卫诗婕

Thank you so much.

尤瓦尔·赫拉利

Thank you so much.

卫诗婕

赫拉利教授因为时间关系,先离开了我们的录制现场。但是我还有一些问题想追问小川总,所以很感谢小川总的时间。要不先从刚才聊到一半的医疗 AI 开始,可不可以给我们更多的细节?

10. Building Doctors For Everyone

王小川

大模型其实就是一个 agent,或者说,我们就是在造人,造一个像人一样思考、能沟通的人。这里面,对我而言,我觉得非常性感的一件事情,就是去造医生。

造医生有两个重要的意义。第一件事情,今天全地球,包括中国的医疗资源都不够,有些医生很匮乏,尤其儿科医生,是匮乏中的匮乏。我们每个人都希望自己有一个医生朋友,或者一个私人医生。

卫诗婕

为什么儿科医生是最匮乏的?

王小川

因为儿科医生不赚钱,所以大家不愿意做儿科医生。中国只有20万儿科医生,我们的缺口大概在100万。成人医生可以做手术,儿童严重的罕见病比较少,治疗方案相对也有限,所以大家不愿意做儿科医生。

咱们是依靠医生来做科研的,但是医生做科研的话,尤其是每个人看的病人可能也不够多。甚至由于日常工作很繁忙,临床上也没有足够的数据收集。比如病人来了之后开了药,但是患者的一些查体没有被记下来。大家很忙的时候,甚至记下来也不一定是真的。

这样的话,就使得医生工作里面的数据维度和准确度都不够高。甚至医生一辈子也就看过几万个病人、几十万个病人,没有足够大的样本。因此,当我们有 AI 医生能看病的时候,它能够把你的全病程数据做记录,甚至把全生命周期的数据做记录。这样一来,整个过程就变成一个可以研究的课题。

有了这样的数据之后,我们对于医学的理解就能够上一个台阶,产生新的发现。我们知道,今天的医学相对物理学而言落后很多。今天最领先的药,临床试验如果有1000个人参加,其中有800个人有效,医生就称之为有应答(response),这就是很好的药了。用10年时间、10亿美元发明这么一款药,但剩下的20%的人可能没有效果,这说明我们对医学中的个体了解很少。

有可能今天你吃有效,明天就没效;这个男人吃有效,女人吃没效;你在旧金山吃有效,到了成都吃就可能没效。所以,太多数据中的观察维度没有被发现。包括 Oswald Biggs C U Darrell[?],他就讲,今天 AI 做医疗,很多时候不只是帮助我们总结数据,它还能够做实验设计,帮助我们更好地生产更多数据、观测更多数据。

这样,AI 医生一方面能够提供很多临床服务,同时也能收集数据,帮助我们最后做科研和解释。它既做实验,又做理论。

卫诗婕

对对对,这是一个科学范式,是 AI for Science 的一部分。所以,你们的大模型现在具体可以做些什么?

王小川

现在大模型有几个大的功能。一个是在院内,它其实已经能够根据你提供的检验、检查、症状,以及各种各样的数据,帮你做病情分析。从诊断到治疗原则,再到治疗结果,它都会给出很多意见,并且这个意见是有过程的。

我们以前总说 AI 是一个黑箱,对吧?你刚刚说的这个 black box,其实已经是过去式了。今天类似于用 DeepSeek,大家用过的人都知道,它会有一个思维链,在过程中会把它怎么思考的过程展示给你看。

你怎么看?我们讲 AI 医生,医生使用我们的 M1 推理引擎时,我们内部的医生,包括医院的医生,都会觉得它的推理过程很漂亮,思考得非常严谨、完整,对医生也有很大的启发。

它比大多数医生,甚至绝大多数医生,见过的病其实更多,思考得更加完整,推理也更好。这是能够辅助医生的部分。

另外一件事情是,它的工作场景可能在家里。它能告诉你一些建议,比如你是否应该去医院、你的重症等级如何,是可以在家继续观察,还是需要去医院,还是需要急诊。它就像医生朋友一样,给你很多这样的建议,既能帮助医生,也能帮助具体的患者和家长。

卫诗婕

所以,现在数据这一方面有难点吗?

王小川

现在来讲,跟以前不一样。以前做图像处理的时候,大家有很多数据难点,而这一次更多的是激发它的推理能力,把医生的推理做进去。

现在我们跟北京儿童医院有深度合作,还成立了北京的重点实验室。背后有一套方法能够构造出不错的数据,所以在今天,对于医生而言,相比以前的图像处理,它的强大能力会多很多。

至于它提到的幻觉问题,其实今天已经不是大问题了。我们今天的 AI 是通过思考,像人一样,它会翻书。翻书之后,指南、用药方法都能够被注入它的答案里面去。

因此,今天讲的幻觉问题,或者叫黑箱问题,在今天都是可解的事情。

卫诗婕

这么坚定的吗?就是完全可解?

王小川

错误率现在可能是1%以下。因此我们认为,在内科诊断或者治疗原则上,它已经远远超越了绝大多数人。

卫诗婕

你刚才讲医疗模型的推理问题很有意思。我的理解是不是,它就是把人类医生的经验,以及他们所有的推理过程,都传授给模型?

王小川

对,它不止是经验。我们总说医学是经验的科学,但今天总结之后,会形成一些我们的思维方式和治疗原则,它的推理过程也学过去了。所以,既有经验,又有这样的原则。

卫诗婕

刚才教授提到的一个非常关键的问题,我听下来,在你们的应用场景里面,暂时还没有给所谓的 AI 医生一个决策权。它可能更多给到的是建议,或者辅助诊断的角色,是吗?

王小川

对,这是来自法律法规的要求。在法规里面,诊断权和最后的处方权都归医生,所以医生之外,AI 是不能给的。我相信随着技术成熟和我们对它的理解,更多这样的权限会放给 AI。

卫诗婕

我最感兴趣的一个问题是,我完全理解你对于生命科学的这种热忱,但我还是会觉得,你做医疗大模型这件事情是一个很勇敢的决定。因为所有跟医疗相关的事情,似乎一旦人们的健康、人类的生命出现一些失误,就会引发特别大的舆情。

今天其实没有来得及讲到,2016年发生了两件大事:一个是 AlphaGo 战胜人类棋手,另外一个是百度魏则西事件。我知道你之前提过,百度魏则西事件给你非常大的触动,当时也埋下了一颗种子,让你后来想做生命科学相关的探索。

王小川

大家会用“勇敢”这个词,或者我们用一个词叫 bet,去赌一件事情。其实在我看来,我已经看到这个未来长什么样子了,所以我不觉得这是一个赌,也不觉得这是一件勇敢的事情。

2016年的两件事情,当时一个是 AlphaGo,人工智能已经有了一次启蒙。但是那次启蒙,机器还没有掌握语言,所以并不代表它能够像人一样去工作,对吧?那会儿还有 IBM Watson,后来也失败了。大家如果问我为什么失败,一个大的原因就是那时候技术还没有到。

今天机器掌握语言之后,已经带来了巨大的变化。确实,2016年之后,百度也遇到了魏则西事件。作为搜狗的带头人,我们当时也知道,AI 没有能力像人一样帮你解答问题,因此我们当时发布的模型能力叫“搜狗明医”。这个“明”是日月明的“明”,叫明明白白看医生,使你得到更多充分的信息去跟医生接触。

当时我们也知道,依靠连接医生,互联网的传统做法是行不通的。淘宝可以做连接,美团可以做连接,滴滴可以做连接,但医生不能靠连接,因为优质医疗资源本来就有限。这会使挂号网、春雨医生等平台能够提供的服务受到限制。对,好医生就那么多。

因此当时我就认为,只有等技术实现足够大的突破,能够去造医生,而不是找医生,才能带来大的变化。到2023年的时候,AI 技术取得了这样的突破,语言能力实现了突破,我们就知道,医疗行业造医生会变得可行。它既能够造福大众,又能够实现我以前的一个愿望,是生命科学解码中的一个关键环节,所以我觉得这是一个特别有意义的事情。

只是在2023年的时候,技术还不像今天发展得这么快。到了2025年,反而这个环境变得更好了。我们特别自信,像 DeepSeek 已经在很多方向上进行了探索,从论文到代码都有开源,使得整个行业未来要实现突破时,速度变得更快了。所以,造医生这件事的进展也会加速。

卫诗婕

我完全不怀疑 AI 技术发展的速度。就像刚才教授说的,大家都觉得它发展得很快,所有商业机构其实都没有办法停下来。这里面可能有一个很值得关心的问题,就是我们怎么确立飞速发展的技术边界。

因为一旦要落地到应用,我们就需要知道它的边界在哪里:它能做什么,不能做什么,尤其是涉及医疗这样严肃的话题。

王小川

其实这并不是多难的话题。医疗的核心东西叫评价体系,发论文的时候大家都知道有效性是什么,所以医学统计学甚至不只是一门课,而是一个系。当你把它当成一个药、当成一个器械来看的时候,已经有很明确的标准了。

只是在人类思考这件事情的时候,今天大家还有很多处于蒙圈的状态。尤其是我跟一些大专家、一些院士沟通的时候,大家之间的信息还停留在以前的影像时代。

但当有试点能够落地之后,这个过程会非常快地加速。包括我们和儿童医院的合作,我们在试点的地方能够在医院医生指导下真实上岗,真实地对患者产生作用,真正能够在卫生经济学上使社会成本降低,那么国家政策也会相应变化。这个速度,我觉得比无人驾驶可能还会更快一些。

卫诗婕

试点场景里面具体会发生一些什么?能不能给我们介绍一下?

王小川

首先,我们的医生已经跟 AI 联合上岗了。医生使用的时候,在一些复杂疾病上,这个系统就会给医生相应的建议。

今年内,我们很快就会把这个系统部署到更多的儿童医院,以及一些更加偏基层的医院,使这些医生也能够有这样的 AI 小伙伴去辅助他们做诊断。决定权在医生手里,这是不违规的。

再往下,使每个家庭里面都能拥有一个自己的 AI 保健儿科医生。从法规上说,它不能给你做后续的处方和治疗,但能给你相应的建议。甚至这个时候,它还能跟医生连接在一起,使互联网医疗跟家庭场景连接起来,整个社会的效率都能够大大提升。

这样,未来的家长就不用带着孩子天天往医院跑了,可以缓解医疗资源匮乏的问题。我们国家儿科医生不足,尤其是基层,大家都到三甲医院去挤,使早诊早发现,甚至在社区里面进行,都变得更加容易。大家会更健康,国家的医保开销也能得到巨大的缓解。这样一来,未来18个月内,我估计就会有巨大的普及。

卫诗婕

这么乐观,18个月?所以,训练模型和说服应用落地,这两件事情哪件更难?

王小川

都还好吧。

卫诗婕

两年多的这个过程当中,有没有让你觉得比较 suffer 的事情?

王小川

有。这个过程里面,确实有太多的人有时候可以说是信仰不够,或者没有看到未来,对吧?不管是社会层面,还是公司内部层面,也不代表所有人都能够有这种坚定的信仰,在遇到困难时继续往前走。

所以当时很多时候讲的就是“你先相信”,因为你已经看见了,才会去实现它。我觉得,如果细分来讲,内部有一些同学、一些员工,他们对事情没有坚定的信仰,遇到困难时就会停下来。这是我觉得需要不断寻找更加志同道合、更加有信仰的人的地方。

卫诗婕

所以对你来讲,因为你内心一直相信,所以你很少 suffer,是吗?

王小川

对,我不 suffer 这件事。对啊,我只觉得大家没有信仰,很遗憾。

卫诗婕

能不能分享一个最近应用案例当中,让你比较兴奋的?

王小川

其实之前,上次在内蒙古,我们刚刚做推理引擎的时候,内蒙古一个市医院有一位父亲被诊断为脑梗之类的疾病,在医院住了十几天。医生已经下了最后通知,觉得没得救了,于是把他往北京送。

在这个过程中,正好有人联系到我们,我们的推理引擎就开始做推理,告诉他有3种可能性,是3个鉴别诊断。到了协和之后,协和的专家一查有4种,其实我们当时已经命中了3种。

这是春节前发生的一件事情,所以当时我们已经看到,AI 在某些问题上,已经能够比一个市级医院强很多了。

卫诗婕

这次创业有没有带给你更兴奋的感觉?

王小川

我觉得会的。因为第一次创业的时候,坦白讲,它是搜狐公司的子公司,自主性相对有些限制。另一方面,当时做搜索引擎也比百度晚了好几年,你做了一件并不是世界需要你的事情。

做输入法来讲,是无心插柳。所以,能够做一件世界需要的事情,会变得特别重要。不在于你有多牛,而是这个时代给了我们这样的机会。医疗是时代所需要的,因此 AI for Science 也是时代所需要的。做世界需要的事情,会变得很幸福。

卫诗婕

好吧,祝你幸福。谢谢。

王小川

好,谢谢。

卫诗婕

好了,本期节目就到这里,欢迎在留言区评论,分享听后感,也欢迎留下你想听的选题或是嘉宾。这期节目已经在苹果 Podcast、网易云音乐、QQ 音乐、豆瓣、微博音频等平台同步上线,欢迎订阅,我们下期再见。