从笔记应用到 AI 工作空间：Simon Last 专访

Welcome, Simon. Thanks for doing this.

Of course. It’s really fun to be here.

Notion is at scale—an amazing platform with lots of users. You did start quite a while ago. I think of Notion as one of the companies that has really raced AI quite aggressively. I was told you first got your hands on GPT-4 at a company offsite in Mexico. Is that true? What’s the origin story of starting to work on this stuff?

That year was 2022. I’d been watching what was going on. In general, I’ve just been super curious about the technology and fascinated to try everything and think about how we can apply it. It wasn’t until I played with GPT-4 that it became really, really real.

When we got access to it, it was sort of a proto-ChatGPT-like interface. My co-founder Ivan and I both got access, and it was immediately clear to both of us—I’d say there were 2 big things. One is that it was pretty smart. It could follow reasonably complicated instructions, write things for you, and edit things.

The second big thing was that the scope of its knowledge was extremely interesting: super, super deep and broad world knowledge. When we played with it, it became instantly clear to both of us: okay, the time is now to start thinking about how to apply this. It’s only going to get better.

We’re talking about Mexico and GPT-4. You guys saw that it was clearly the time. Did you start with a particular vision of what you should obviously be able to do with AI in Notion, or did you start pulling people from different teams or recruiting people and saying, “Let’s experiment”? How did you begin?

I think we immediately had a long-term and a short-term vision. I’ll start with the short-term one. The thing that was immediately obvious was that it could be a writing assistant. It could be in your document: you could select some text and have it rewrite it. You could have it write text for you, maybe look something up, and then give you sources or more information.

That was the thing that we immediately got to work on. We started a tiger team around it, and then we were able to launch it in 2 or 3 months. The long-term vision that we immediately had was that it might be possible to build more of a general assistant.

What if you could just give it all the tools inside Notion that a human would be able to use? It could create its own databases, query and manipulate them, create documents, edit them, and weave all of these things together to do a longer-range task. We immediately started on both. The short-term one we were able to ship very quickly, and the long-term one didn’t really work yet. That took much longer to get working.

Was the first launch of the AI-specific Notion features and products last year?

No, it was February 2023 when we launched.

My timelines are wrong. Are there a few specific learnings or breakthrough moments you think have been interesting since you began releasing this stuff?

It’s been a slog over many years—or over all the years at this point—with many, many learnings. To give you a timeline of the arc of what we shipped, the first thing was our writing system. We called it AI Writer. That was the first thing we launched.

It was easiest to get working because it’s a single-step task: rewriting and editing text. There’s no retrieval aspect. It was just raw access to the model to write text.

The next big thing that we immediately started working on was Q&A: doing a semantic index of the entire workspace and then letting you ask a question, and it can give you an answer grounded in the sources. That was also immediately obvious to us as being super useful, so we started to work on it.

We launched that one in October 2023. We started a beta before then, but our general availability was in October. That was a much bigger effort to get working, obviously. We weren’t just plugging in the LLM. It was actually doing this real-time-updating index. We had to get much more serious about the evals and the quality there as well.

Q&A has been a multi-year journey. Basically, what we did was, as soon as we got the Notion index working, it was obvious that we should index everything else as well. We index Slack and Google Drive, and we’re launching new ones on a regular cadence. Now we have, I would say, a fairly complete index.

Those are very difficult problems that those products natively haven’t solved perfectly yet. I don’t know if that’s an offensive thing to say about other product teams, but it’s not working yet.

It’s kind of true. This is something we talk about a lot because it’s almost like, what right do we even have to do this? But it turns out that most companies are pretty bad at making their indexes. Honestly, it’s kind of baffled us a little bit.

My take after dealing with all of this and working with the teams trying to get it working is that there’s a little bit of AI-pilled savviness that’s pretty important. Most of it is honestly just a bit of craft and attention to detail.

In particular, with this indexing and retrieval stuff, in order to really get it working, you have to be quite empirical and iterative and actually try queries. Each data source is a little bit special. You can’t just apply a one-size-fits-all approach to querying Slack versus querying Google Drive, for example. They’re completely different kinds of information.

We found that there’s just a little bit of craft and love going into it in terms of actually trying a bunch of different queries, using it every day, and constantly iterating, rethinking, and tuning how the retrieval works.

How did you think about the diversity of how people organize their workspaces? Even Notion isn’t used homogeneously, right? I’m probably part of 15 workspaces as an investor. I look at them, and I’m like, “Well, mine’s a mess, and these people are really organized,” and the workflows are reflected in how their Notion works.

Totally. The interesting thing is that with embeddings, it almost doesn’t matter as much anymore. The AI doesn’t really care what the tree structure is. All the AI cares about is that there’s a snippet of text that has the context you need and that it can retrieve it.

We advise people now: don’t worry as much about organization. Just find a way to get it all piped in and throw it in there.

You still make decisions that could change performance quite a bit, like chunking strategy or whatever, right?

Yeah, that’s super important. But that’s sort of transparent to the user and independent of their particular method of organizing things.

It still seems like a difficult technical challenge, given how different the content bases are.

Yeah, I think that took a lot of iteration. Chunk sizing, how retrieval works, and the different steps in the retrieval pipeline—all of that involved a lot of iteration.

Ivan said I should ask you how many times you’ve rebuilt Notion and rebuilt your harnesses.

It’s kind of a running joke, almost. We rewrite our AI harness probably every 6 months or so, and the time to rewrite it has been decreasing because products have been accelerating.

I think this is honestly a really key thing, and something that a lot of companies get wrong: doing one thing and then just sticking with it. You really do have to be keenly aware of what the current state of the model and the technology is, and then design the harness, system, and product deeply around that.

It basically means you have to rewrite it every 6 months. I find it pretty fun. It’s part of the process. You get to restart and rethink it.

We’re working on a new version of our harness that we’re about to release in the next week or 2, and we’re already thinking about the one after that as well.

I think that leads to a set of questions I have for you about how Notion’s engineering, product, and research organization works now that you have the power of coding agents as well. I imagine your willingness to rewrite the harness goes up dramatically: agents are going to help me do it.

That’s extremely true. It’s been really fun to use coding agents. I think the ambition of what I even consider building has gone up a lot.

What do you think has most dramatically changed in how you think about how engineering and product should work at Notion over the last 2 or 3 years?

Woof, yeah. It’s definitely changed multiple times.

I mean, in terms of the coding agents, we went through multiple eras. There was the tab autocomplete era, and then we got into inserting and rewriting some code. But it wasn't really until the agents started working. I would say early last year we started to adopt the agents. I started using Claude Code around April last year, and that was a huge unlock.

The big shift there is that you can really push on getting these agents to implement, verify, and maintain things end to end. But it requires significant thought in terms of how you architect things and what the verification loop is. The upshot is, I think if you do it well, you can be much more ambitious about what you're building and also make it much more robust than you could have done with humans writing it. The flip side is, if you do it badly, it's all slop.

Does that change your lens of what teams should look like at Notion—in terms of size, seniority, or anything like that?

I would say the fundamental effect is that everyone's individual impact in terms of their output can be much higher, and your output increasingly depends on your ability and willingness to use the tools. I think that's the fundamental thing that's happening.

As for how that plays out, I don't think we've seen that much impact on team size, really. We like to work in smallish tiger teams for the most part. If you can make a team small, it's almost always better. That was true before, and I think it's still true—maybe increasingly a little bit, but not that much. The main thing is to really harness the tools.

Do you think something different happens to the median engineering organization versus the 10x engineer, or the engineer who's 10x more willing to use the tools?

I think the gap is bigger. You can be a 100x or 1,000x engineer if you use the tools right now. I think the gap is much bigger. The minimum bar has not changed, but the maximum bar has increased dramatically.

One impact it has had internally, I would say, is that broadly, things feel a little bit messier and more chaotic. But I kind of love that. There are more prototypes—way more prototypes. For example, our design team made an entire Git repository. They call it the Design Playground, and it's essentially a simplified Notion with a bunch of UI primitives.

Okay. Got it.

They've made it really sophisticated. It has an agent in there, and it's pretty cool because it allows all the designers to spin up super-high-fidelity prototypes really quickly. It's no longer pointing at a mock and saying, "How would this look?" They'll give you a URL to a prototype that's been deployed.

That sort of thing is true all the way up and down the stack for all of engineering. It's just a little bit more chaotic, with more stuff happening. All the pull requests are more ambitious.

Do you draw a line somewhere about stuff that's more dangerous to touch or sensitive? Is there a risk of data loss over here, or do you look at it all as fair game?

We still do reviews on all the pull requests. All the pull requests are now written by agents, and they're often larger and more complex. That's the worst part. But the better part is that they're often much better tested, and we can demand much better testing for the things that merit it.

I never produce a pull request that hasn't been fully end-to-end tested anymore. You can get to a pretty high degree of confidence that it works. But it requires more than just vibe coding by saying the thing you want. You're thinking carefully about what the change is you're trying to make, how it can be verified, and how it can be deployed safely, and then enlisting the agent to help you with that process.

When you think about where you said the general assistant doesn't quite exist yet, what do you imagine Notion's agents being able to do over the next year or two that they're still blocked from doing, either by capability or by your harness work?

We struggled for a few years to build an agent. It always sort of worked, but then it wasn't that useful. Largely, it was too early. We tried to build an agent, I would say, three or four times. Then we finally launched it last fall, around August or September.

If you use Notion AI now, it's the full agent that has access to pretty much everything in Notion. That totally works. A lot of the original vision that we had works now, and it's fully shipped.

Last August or September, we shipped our personal agent. Pretty much every user in Notion has an agent, and it has access to all the things that the user has access to. It can create a database for you, update things, create documents, search the web, and do research.

The second big thing that we just launched last week was custom agents. You can create a new custom agent and give it a name. Unlike the personal agent, by default it doesn't have access to anything, so you have to grant it access. But once you do, it can actually run autonomously in the background.

For example, you can give it access to its own database to file tasks, and then attach it to a Slack channel. It will start responding to people on Slack and fulfilling tasks. That's one use case.

Another one is that you could give it access to a database of weekly reports, and then let it search the web or search your workspace. A custom agent represents some work or job—some knowledge-work task that you want to be done autonomously.

One thing I'm really excited about going forward is that we want it to be extremely good at bootstrapping its own capabilities. Basically, from an initial kernel, allowing it to bootstrap itself to do anything. For example, it might build an integration that we don't support yet, deploy it, and then use it.

So you imagine that Notion agents are actually the broader definition of an agent, where writing code is a tool that it has access to?

I think it's pretty key. I think of coding agents as the kernel of AGI. AGI will be a coding agent. Code is just a really, really useful primitive for representing deterministic logic.

The thing that's really exciting about applying it to a knowledge-work agent is that it can bootstrap a capability. If an integration doesn't exist, it can build it. If it needs to connect itself to a new data source, it can do that.

Given that you have Notion at that scale, you're operating in a landscape of productivity and platform players that are at even more scale, right? Many of these will end up with their own agents. Lots of people from the labs and the Microsoft world are trying to integrate other data sources. So you have this cross-attempt to integrate and index. How do you think that plays out? What do you imagine Notion agents are best at, or what do they have the right to go do?

If you look at the landscape, I would say there's the labs, then maybe the software platforms, and then maybe infrastructure.

In terms of the labs, we see ourselves as Switzerland for models. Our customers don't want to be locked into a certain lab's model. They're always releasing new versions, and in any given month, one is better than the other. We want to be a place where you can easily get access to all the best models at any time, and easily switch around.

Do you think open source plays into that as well?

Absolutely. I think the open-source models are actually getting really good. There are four different Chinese models now that are quite good.

Yeah.

We just released one of them in our agent last week, and we're going to do all four for sure. They're actually quite good, and they're way cheaper than the frontier models. So I think there are a lot of use cases where you'd want that, and we want to give people that option.

In terms of the other platforms, we think of our role as taking all the best models that we can, creating really high-quality, state-of-the-art agent implementations where people can easily and conveniently get access to them, and then making a collaborative workspace that's really good for humans and agents to coordinate in.

I think it's something that's very needed in the world, and we're just trying to do it in a really tasteful, well-executed way.

You were describing how you need the index to make the agents good. You give the agents access to the tools that we humans have in Notion. How do you think about the structure of Notion, and where is it useful—or not useful or relevant—for agents? Blocks, databases, and such.

It's all still pretty useful—extremely useful. There's been a challenge: we want to make it really convenient for the agent. I think that's a new thing that didn't exist. In the past, it was convenient for humans, and then we also made APIs convenient for humans writing code to our API. So we essentially have a new customer, which is the agent.

At first, that was definitely a problem. For example, our API uses this crazy JSON format for blocks that, by default, is crazy verbose and horrible for the agent. But we took on that challenge and designed really convenient APIs for the agent. We created a sort of Markdown dialect that looks like normal Markdown, but it's enhanced with all the Notion blocks. The models are really good at it; it works really well. That's how it reads and writes the pages. For databases, we use SQLite, so basically it speaks SQLite, which also works really well. The default thing did not work really well, but we took that on as an engineering challenge, and now we have extremely convenient APIs that agents are naturally good at.

How did you understand or figure out what would make the API better for agents?

That's a good question. I would say it's a combination of just trying things. It's very empirical. We're playing around and noticing, “It's not very good at that. That's way too many tokens. How can we make this smaller?”

Then there's a little bit of first-principles thinking: What are the models being trained on, and what's in their prior? What do they know, and what do we think they would naturally be good at? How does the agent loop work, and what would be the convenient, efficient pattern for accessing these things? Then just a lot of playing around.

I hear user research where the user is actually an agent.

Yeah, you just chat with it. The user's always there. It's ready to talk to you.

Actually, that is wonderful, where you have infinite access to it.

Infinite access to it. You can script and scale the access as well.

I assume you have—actually, I know you do, because you walked in and you were like, “Hey, I need to get access to Wi-Fi. I need power. We can't block the agents while we're doing this.” What do you have running right now? Tell me about your setup.

I'm working on a new prototype, and I have a couple of agents working on that. My setup these days is either Claude Code or Codex. I like the CLI tools. They're super simple and work pretty well. I'm pretty comfortable in the CLI.

You don't need a generated game to use the CLI. It's a very cool idea.

I would say my whole goal these days is essentially to have as many running as possible and to run them all the time. Every night before I go to bed, I'm like, “Okay, let's go, guys.”

Basically, what I have to do is make sure that I've given it enough stuff that by the time I wake up in the morning, it will still not be done. I've maximized that.

That's victory.

Yeah, that's victory. I've done that for the last 5 nights pretty well. My personal record is that I've had a coding agent running for, I think it was, 13 days straight without stopping, just basically working through tasks, well prompted.

I admit to having woken up in the middle of the night at least multiple times this week and just being like, “Are you still going?”

Yeah, I know. It's kind of nerve-wracking. I always check it one last time before I go to bed and really make sure that it's still spinning.

What about the Notion agents? Do you have a workflow there that's core to your daily work?

I use our personal agent all the time, so it has all the context about our company and everything that's going on. For example, last night I was asking it about how the custom agents launch was going and what signals we were getting from it. It was super useful for that.

I have many custom agents that are running. My personal favorite is an email triage agent. It has access to all of my work and personal emails, and it wakes up every day and archives all the stuff I don't need to see. I trained it over time to learn my preferences.

Do you actually label data for it?

It's pretty easy to do this, actually. All you have to do is make the agent, give it access to your email, and make a blank page that's its memory and let it edit that page. Then you say, “Okay, now go and look at my emails and interview me. Ask me which things…”

It will propose things that it thinks it should archive, and you can correct it. It uses that to essentially generate a list of rules about what it thinks is correct or not. For the first couple of days, I was correcting it on things. After a couple of weeks or so, I dropped the approval entirely, and it just automatically archives all the things I don't need to see now.

Wow, that's a lot of trust.

It completely solved my email problems because I don't use email that much for work stuff. It's mostly in Slack. 95% of the personal and work emails that I get, I don't need to see at all, so it's just a waste of time. It completely solved that. Now, whenever I look at my inbox, it's only stuff I need to see.

I've got lots of custom agents running. There's another one that I built that triages customer and internal feedback and bugs. We have a Slack channel where people just post random product feedback and bugs. In the past, it would sometimes get answered, but then sometimes it would get haphazardly ignored because there are so many teams running things.

Its entire job is just to route it to the right place. It uses a similar memory pattern where it learns on the fly where it's supposed to file bugs. Over time, it's built up hundreds of rules that it learned. For example, if there's a bug about the mobile app, it knows to route it to the mobile team and file a task in their database.

Do you look at the generated and updated memory because it's legible to you? Do you look at it and say, “Does that make sense to me?”

I think I did at first, but once you trust that it's working, you kind of ignore it. If it ever breaks, I'll go fix it. It'll break every now and then, but I'll go fix it.

But the benefit of not reading your email is here.

Yeah, you just don't read it. Generally, the pattern I follow is that I build it as a prototype and have it in approval mode, where I'm watching it closely. After it runs a bunch of times, you trust that it's working.

Is there anything you do internally at Notion to make sure nontechnical teams have the intuition for how to build agents or how to express that productivity, too?

It's a great question. We do workshops and hackathons pretty frequently. For example, a month ago, I did a hackathon with the People team and got them started. The People team's been amazing. They're actually one of the highest adopters of custom agents. They do all these workflows in Slack and Notion, manual work like that.

People are super excited to try it and maybe just need a little bit of a push in terms of intuition and getting them started. Honestly, I've been super impressed. I think the concept is intuitive once you get past a little bit of the technical barrier of what a prompt is, what the agent is, how it gets triggered and woken up, and how that even works. Once you get past that, I think it's actually a very human-like interface.

Maybe the biggest barrier is actually just getting people to try it and assuming it's going to work at all.

You and Ivan originally met on the internet, in the Tools for Thought community. It feels like the tools we have for thinking are very different now. Has your core conception of Notion changed over the last few years because of all the AI stuff? What thinking does the tool do for you? What should agents do for you? What do you get to do?

Yeah, it's changed quite a lot. Broadly speaking, before AI, our goal was to create the best tool for humans to directly perform their work. Now the goal is to create the best tool for humans to manage agents to do the work for them.

That's a big shift. That's a pretty big shift.

It's pretty fundamental. But it turns out that you need most of the same primitives. All the primitives that we built are still extremely useful.

It’s more that we needed some new primitives, like representing what an agent is and how to interact with your pages and databases. But you still need the same primitives. You still need a document. It’s an unstructured way to write things, and agents love to write Markdown documents, so it’s still very relevant.

You still need a database. You still need structured data. If you’re working with your swarm of 100 background coding agents, you don’t want to have 100 chat threads. You want a Kanban board. It’s the same as before. You still need the coordination structure.

Makes sense. You still need the coordination structure. What is one thing that, because you’re ahead of this and trying to figure out how to bring Notion and its users along with you, has really changed about how you personally build, even in the last 6 months?

It’s completely changed. I haven’t written code since last summer. I don’t type code anymore.

We went from humans typing all the code to still typing, but using tab completion. Then we started talking to the agent, and it would do little tasks for us, but we were still in the outer loop. Now it’s more like I design an end-to-end task that involves making some change and verifying it end to end. Then I’m the outer verifier, double-checking at the very end that it’s correct and monitoring it if it’s going off the rails.

It’s a complete shift. I’m now the agent manager instead of the coder.

Amazing. Thanks, Simon. This has been a great discussion about how we’re all going to become agent managers, hopefully in Notion.

Cool. Yeah.