What It Takes to Run a Personal Assistant Around the Clock
Ken LiuMartin CasadoJoel de la Garza
- Second aims to become the durable store of personal context that other AI agents query. Ken Liu describes a model of your memories, preferences, priorities, values, and commitments that travels between tools. First Person Labs intends one trusted Second—“like a vault”—to represent your interests as specialized agents handle tasks.
- The strongest early use case is connecting events to personal priorities without waiting for a prompt. Joel de la Garza describes Second monitoring an event’s weather forecast and whether a tent would be needed, without instructions. Ken distinguishes this longer-horizon, “reflective” proactivity from agents multiplying notifications: “People have to ask. You’re not supposed to ask these things.”
- Second’s security case combines control over data and inference with less certain judgments about disclosure. Local storage, provider choice, and confidential GPU inference underpin the architecture; zero data retention remains “a promise by policy.” Martin Casado presses on role-play attacks that could extract private information, and Ken says prompt injection cannot be definitively solved.
- Always-on analysis makes inference efficiency central to the product’s economics. Ken’s illustrative workload—ten agents running continuously at 60 tokens per second—costs $1,000 per day at “Opus 5.5” pricing versus $60 with DeepSeek Flash. Busy professionals and frequent tool-switchers are the initial audience; a mixture of models “has to be” the approach for cost efficiency.
- The platform opportunity is exposing personal understanding through APIs that applications can use. Second produces context about recent activity, upcoming priorities, reminders, and relationships. Ken says coding agents can receive a screenshot and “Fix,” while laptop-to-phone “context continuity” lets work continue without rebuilding the conversation.
- First Person Labs narrowed its interface after finding interaction design harder than expected. “We made a lot of mistakes on the interaction side,” Ken admits; the product now concentrates on texting and APIs. His longer-term vision is one personal model spanning devices, with other builders discovering the applications and interfaces that make it useful.
1. One personal model can serve many specialized agents
- Ken Liu defines First Person Labs’ Second as a “model of a person”: memories and activity, but also priorities, preferences, goals, values, commitments, and intent. It starts with artifacts about what you did and whom you know, then builds understanding from how you speak and share information. Over time, “maybe you train some of that into the model.”
- Martin Casado asks where this fits into his existing agent usage. Ken’s answer is “portable context”: an outside agent consults Second before executing. “Buy me a flight” should arrive with seat preferences, airport-arrival timing, and class preferences already understood, removing the repeated questioning needed to reconstruct what the user wants.
- Martin’s architectural question is whether users eventually need a second, third, and fourth personal representative. Ken says it stays one Second—a consistent store that protects the user’s interests while other agents specialize. Temporary sub-agents can disappear. His Alfred-and-Batman analogy adds discretion: Alfred buys the cheeseburger and states the preferences without revealing that Batman ordered it.
2. Background understanding makes proactivity more useful
- Ken’s raw material is the computer’s existing event stream: app switches, typing, pauses on websites, and Wi-Fi connections across Apple, Microsoft, and Linux. Agents running 24/7 build a hierarchy from those traces—a typing event becomes a message; browser-to-Codex switching might indicate research or writing a Google Doc. Connectors help the system learn faster. People underestimate “how much the computer already knows.”
- Joel de la Garza’s concrete example is event planning with a 20% chance of rain: Second kept updating the forecast in real time to help decide whether a tent was needed. Martin immediately asks whether he requested that monitoring. “No, I did not ask it to do that.” Martin reports similarly useful texts about ongoing conversations he had not thought to revisit.
- Ken distinguishes reactive from “reflective” proactivity. His caricature of the former is ten notifications for one email because nine other agents also alert you. Reflective assistance connects something said months ago with current work, goals, priorities, and preferences. The bottleneck is requiring users to initiate everything: “People have to ask. You’re not supposed to have to ask these things. They’re superintelligence.”
3. Hardware privacy and model discretion are separate problems
- Joel raises the obvious objection to software that sees everything. Ken’s starting position is that “you’re not supposed to trust it in the beginning.” He frames this as a new, more passive and ambient form of AI: the architecture has to account for both data and inference. Second Local keeps the data on the laptop and sends it only to the provider and inference the user chooses. In principle, users should be able to bring their own tokens to analyze local data.
- Ken describes confidential inference as running a model inside GPUs with confidential computing enabled, supported by hardware checks and signatures. The trust moves toward the hardware—NVIDIA—rather than whoever operates the model. Martin draws out the distinction from zero data retention; Ken calls that “a promise by policy,” albeit one with legal effect. The hardware arrangement is intended to be technically provable.
- For cloud hosting, Ken describes encrypting data locally before storage. Decryption can happen transiently in memory or inside a confidential container that can be attested, extending the confidential-inference approach to storage. The intended chain is local encryption, cloud storage, and decryption inside a secure environment.
- Martin’s pushback is that a role-play prompt could trick the LLM into revealing information. Ken separates mandatory deterministic controls from context-based disclosure judgments. Apps should not retain passwords; other sharing decisions rely on the model using context. Prompt injection “isn’t a problem you can definitively solve.” Ken expects better reasoning to improve resistance as models advance; that is a different assurance from the hardware protections.
4. Fidelity, latency, and model routing determine the economics
- Ken’s back-of-the-envelope workload is ten agents running 24/7 at 60 tokens per second. At Opus 5.5 pricing, he puts the cost at $1,000 per day; using DeepSeek Flash brings it to roughly $60 per day. These are illustrative inference workloads, not a quoted subscription price. Martin’s reaction to the first figure: “Oh my goodness, $1,000 per day?”
- “Right now, it’s not going to be for everyone,” Ken says. The initial fit is busy professionals and people switching frequently between tools. He says token prices have already dropped by more than 50% since the company started, as newer models are swapped in. The product is available today, though rollout depends on scaling inference capacity and adding support while keeping users’ data controlled and secure.
- Martin asks whether a looser service level could reduce token consumption. Ken says, in principle, yes: model fidelity trades against cost, and “cache time” against “test time.” Continually analyzing stored data buys deeper understanding in advance. Keeping raw logs instead shifts reasoning into the eventual request, which might take, “say, 10 times longer.”
- Joel questions whether frontier models deliver much extra value over commodity ones. Ken’s answer is task-dependent: recognizing a moved calendar invitation needs little intelligence; inferring Joel’s weekly goal warrants an expensive model, used infrequently. Second is therefore “a harness design problem” deciding which models run when. A mixture “has to” be the approach for cost efficiency, with portability also preserving the option of eventual local inference.
5. Personal APIs create room for applications still to be discovered
- After installation, permissions, and token configuration, background agents begin producing APIs about the user: what happened ten minutes ago, next week’s priorities, reminders, and a wiki of people and processes. Other agents can query this portable context; Ken expects more APIs and applications to emerge on top.
- For development, Ken says Second is a “super good memory of what we’re trying to do.” A coding agent can receive a screenshot and “Fix” without another explanation. His second practical benefit is “context continuity”: close the laptop, take out the phone, and continue work with understanding of what you’ve been doing up to the minute.
- A designer using Second found that captured preferences for button placement and colors reduced iterations when building a personal app. Ken’s broader analogy is the Altair 8800: an initially unhelpful computer became useful as applications such as spreadsheets appeared. He thinks substantial application building and interaction design remain; the killer application on this personal context is still something builders may discover.
6. The interface narrowed while the ambition became ubiquitous
- Ken’s admission is explicit: “We made a lot of mistakes on the interaction side.” Building interfaces and personal pages proved difficult enough that the company chose to enable others instead. It decided to close down its existing surfaces: “There’s only texting, there’s only API, and there’s nothing else.” Understanding and systems work come first; broader interface innovation is a longer-term focus.
- Martin wants an embodied assistant in his kitchen. Ken’s Jarvis framing is that intelligence alone misses the point: “it’s everywhere.” Today, an internet-connected iPad agent that understands APIs can reach Second through cloud MCP, although the OS limits deeper integration. Longer term, Ken imagines everyone having a Second, with devices becoming inputs and outputs to the same understanding. Dedicated devices might follow, but “devices are hard.”
Full transcript
People are very underestimating how much the computer already knows. You no longer have to explain anything to your coding agents. You can just screenshot anything and say, “Fix,” and then it would know what you’re fixing. The question then is: can you just analyze the heck out of it?
What context does Grok Bot not have that First Person Labs would allow?
One of the biggest problems right now is that people have to ask. You’re not supposed to ask things. They’re superintelligence. But if you have a user model that just builds up by itself, and then these things keep analyzing in the background, you can just share that over instantly. We focus on essentially creating a model of a person and then putting that person in control.
When people think about this thing seeing everything you do, there’s an immediate sort of “whoa.”
If you don’t trust us, let’s say, in principle, you should be able to bring your own tokens to analyze data that’s on your laptop locally.
Can you talk through what you think are the more surprising use cases you’ve found?
For development, it turns out that—
Thank you, Ken, for joining us. Really happy to have you on to talk about what you’ve been working on at First Person Labs.
Is this the first time you’re talking about it?
Yes.
So what do you do?
1. Building a Model of You
We focus on essentially creating a model of a person and then putting that person in control. A model of a person basically means understanding their memories and what they do, but more than that, their priorities, preferences, goals, values, commitments, and intent.
Let’s make this very simple. I’m a big Grok Bot user, so I use Grok Bot a lot. Where would First Person Labs fit into my usage of Grok?
The way it works is that you would just bring context over to Grok. It’s portable context. You go to Grok Bot and say, “Hey, can you talk to my thing to figure out what I want?” Then the two of them will start talking, and you can send prompts like, “Buy me a flight,” and it would know what flight you’re talking about.
What context does Grok Bot not have that First Person Labs would allow?
I would probably ask the opposite question: what does First Person Labs not have if you build a user model of yourself? Let’s say your preferences for the seat on your airplane, how early you’d like to arrive at the airport, and where you’re flying. Are you flying first class or normal? As of today, you have to explain all of that to Grok Bot. You have multiple rounds of questioning. But if you have a user model or a model of you that builds up by itself, and these things keep analyzing in the background, you can share that over instantly.
For example, OpenAI released two new cyber-capable models yesterday.
Mm.
I haven’t been using my Codex in a while, but I have been using Second. So I can just go start using Codex, and it has all the other work I’ve already done and all the additional context already saved?
Yeah. You just go to Codex and ask, “How’s my Second? What’s going on?”
That’s awesome.
Does Second ever become a first person? Will it ever be a first, or will it always go through agents?
That’s a good question. I think a user model is a process. You have to start somewhere. You start with a baseline, foundational list of artifacts: what you did, who the people in your life are, maybe your preferences. Over time, it starts analyzing the way you talk—
Yeah.
—the way you present yourself, the way you share information.
Yeah.
Over time, you build up even more artifacts and more understanding on top, and maybe you train some of that into the model. So you’ll go from portable context into something more like a representation of yourself. One example I like to think of is the Alfred and Batman example. If Batman wants to buy a cheeseburger and get Alfred to do it, Alfred wouldn’t reveal that it’s Batman doing it. He would just buy it. He would state the preference.
Alfred being Batman’s butler.
Oh, yes, yes, yes. Being the butler. Yeah.
Or his agent.
Yeah, or his agent.
Or his agent.
Yeah, yeah.
I should call my agent Alfred.
That’s a great idea.
Yeah, yeah. He would just do it, and then you would trust him because you know he has a good model of you and what that means when it comes to getting things done.
It’s interesting. You describe First Person Labs as having this thing called a Second. The Second understands who you are and has context, and then other agents talk to that Second, right?
Yeah.
It kind of understands your preferences, what you’re doing, and what you want to do. But I think that doesn’t highlight another thing you’ve been working very hard on, and I think it’s worth talking about. You’re actually able to get access to a person’s digital context, like messages, chat, and email, and make that available too. Can you talk a little bit about what you’re doing there? As one of the earliest First Person Labs users, I’d say the thing that struck me wasn’t that it knew who I was, but more that it had access to all of my channels really well. It turns out to be a very hard systems problem, so maybe talk a little bit about that aspect of what you’re doing.
2. The Computer Knows More
It is indeed a very hard problem, especially to get it secure and private. The way it roughly works is that right now, on your computer, there’s an infinite stream of events. As you use your computer, the operating system already gives you information about the apps you switch to, the websites you pause on, the way you type, and how your Wi-Fi is connected. This is across Apple, Microsoft, and even Linux.
The question then is: can you just analyze the heck out of this raw stream 24/7 and apply agents on top, 24/7? You can first synthesize, for example, that a piece of typing in a particular app might mean you’re sending a message. Then, when you switch apps between the browser and Codex, you might be doing some research or writing a Google Doc. You hierarchically build up that understanding, and that, plus existing connectors, can give you a very good understanding of what you’re trying to do. I think people are very underestimating how much the computer already knows.
Yeah.
It’s just sitting there, waiting to be tapped, so to speak.
One interesting thing, having also been a very early adopter of the product, is that lately it’s been indispensable. It’s able to catch things. It’s sending me messages now and letting me know, “Hey, you’ve got this conversation going on over here with this person, and it looks like the situation has changed.” We were planning an event yesterday. There was a 20% chance of rain, and it was continually updating the forecast in real time to let us know if we needed to get a tent or not, which is incredibly powerful.
Did you ask it to do that?
No, I did not ask it to do that.
Yeah, yeah, yeah.
It—
This is my experience with my Second too. I’ll literally get a text from my Second, and it’s something I actually need to know about—
Yeah.
—relative to some ongoing conversation that I didn’t even think about. It’s actually very thoughtful.
Yeah, and it’s amazing. I’m genuinely curious: how are you executing on that? Things I would’ve forgotten, it’s literally catching.
This goes back to the model of the person understanding what might be important to them and what comes up at this point. I think proactivity is something people talk a lot about: AI assisting by pinging you at the time of need. But there are many levels of proactivity. I’d start with the first level, which is called reactive ones. This means an email came in, and there’s a meme online that now you get 10 notifications for email because nine other personal agents are pinging you about it. Reactive proactivity is straightforward and easy to do. But what’s harder is what I call reflective ones: things that connect the dots over many events and a long time horizon. They realize, “Okay, months ago you said this, and recently you’re working toward your goal.”
This is your priority. These are some things on your to-do list, and, by the way, you prefer this. Connecting the dots will give you much better proactivity, and I think that hints at how personal interactions with computers might work in the future. They won’t be focused on reacting to things, but more on asking, “What are some interventions I can apply for the user? What are some things I can do for the user without being asked?” I think that’s one of the biggest bottlenecks right now. People have to ask. You’re not supposed to have to ask these things. They’re superintelligent.
Yeah. You’ve built something that just kind of knows what I need at the moment. That’s pretty impressive.
I want to make it very concrete for people listening: what does First Person Labs provide? Then I’ve got a question. Very concretely, it creates this thing called a Second. You create a Second, and it has access to all of your digital context. It’s amazing. It’s like keystrokes, all your messages, all your channels, all your data. It does this in a very secure way, and based on that, it becomes your personal butler, like Alfred, right?
Yeah.
And then that interacts with your agents.
Yeah.
So it’s not an agent by itself. This isn’t like Claude or ChatGPT or—
Yeah.
—Grok. It’s its own thing. Or Dot. It’s its own thing.
Yeah.
And then it provides that to the agents, right? So that’s what it is. Here’s an interesting thing. I use Grok. I’m a big Grok user, and I have a bunch of them. I have one that keeps track of my synthesizers, one that keeps track of my real estate, and one that, God forbid, helps me with my family, which I probably shouldn’t need one for, but I do. So I have a bunch of these. I have one Second, and they’ll interface with the Second. Do you imagine that over time you’ll have a second and a third and a fourth and a fifth? Or do you think it ends up being your one butler?
Hmm. I think the fact that we have many personal agents is probably transitory. Agents will start to specialize, just like people do. Some agents focus on one thing. Some agents focus, for example, on browsing Amazon and shopping. They’re extremely good at it because they have the data and the connections. Some agents will specifically focus on protecting your data—
Yeah.
—and aligning with your interests.
Yeah.
I think that’s what Second is intended to be. This is something you should be able to trust with your life’s context.
Yeah.
It’s like a vault. Over time, this thing will guide you as you interface with the software world, so to speak, or even the hardware world.
So you think it’ll stay as a Second?
Yeah, it will stay as a Second. It will not be a third.
There will not be a third, there will not be a fourth.
There will not be a third, yeah.
So your one trusted agent for all your personal data that actually knows who you are, and that interacts with all your other agents—that’s Second.
Yeah.
It’s Batman’s safe.
Yeah. You can spin up sub-agents that exist temporarily and then go away. But the point is that you want to have a consistent store, and that consistent store is what gives you that value.
I’m still a paranoid security person, but I’ve probably been one of the faster adopters of some of this stuff compared to my—
Just to be clear, Joel took forever to get on social media.
I just got on Twitter.
It was a few weeks ago, man. Guys—
I may be—
You guys have been in tech for 30 years.
I’ve been—
Just to be clear.
Yeah, I’m a little paranoid. But I love Second, and I’ve got to admit, your background and the things that are motivating you, along with the long discussions we had about the security architecture, really got me comfortable. It’d be great to double-click, because I think when people think, “This thing sees everything you do,” there’s an immediate sort of, “Whoa,” right?
Yeah. You’re not supposed to trust it in the beginning.
You’re not supposed to trust it.
Yeah.
I think you’ve done a really excellent job of building something we can trust. Maybe start talking about what that architecture is and what’s motivating you to build it.
Yeah. I think this is, to some extent, a new form of AI that’s going to work very differently from before. Before, you initiate an interaction and get some results back. This time, it’s more passive. It’s way more ambient. It’s going to be there all the time. You have to think about how you architect it and where the data is going. At the end of the day, there are only 2 things when it comes to AI: data and inference. You have this data, and you want to define what inference you can bring to that data. This is something we built into Second. First of all, you can choose what inference you use. If you don’t trust us, in principle, you should be able to bring your own tokens.
Mm-hmm.
Then you can have those tokens power Second to analyze data that’s on your laptop locally.
But you also provide Second with an inference provider that uses trusted execution.
Yes. Over time, we imagine people won’t bring their own tokens, because most people who bring their own tokens are using tokens from OpenAI or Anthropic. You want to provide a stable option where you say, “If you use these tokens, no one’s looking at the data,” sometimes through policy guarantees, sometimes through hardware guarantees. This idea of confidential inference is getting traction. It’s the idea that you put a model inside some GPUs, and then the GPU has this special toggle called confidential computing. You can turn it on, and when you do, you can basically prove that there’s only the model running inside. No one can see inside. Then you don’t have to trust whoever’s running that model. The hardware is what you trust—
Yeah.
—which is NVIDIA.
There’s another concept that’s getting thrown around a lot right now: zero data retention. That’s very different from what you’re describing.
Yes. Zero data retention is a promise by policy.
Mm-hmm.
That means I’m a provider. I think I have to—
Basically, I give you my word.
I give you my word. That word has legal effect, but, yeah, that’s it.
Yeah.
I think we want to move towards a world where confidential inference becomes very prevalent. I think we’re getting there.
Mm-hmm.
It becomes the default.
And the way you guys have built it, this is technically provable.
Yes.
Yeah.
Yes. So—
It’s not just your word. There’s actually engineering that went into proving this is the case.
Yeah, you have to test the hardware. You have to check some of those signatures. Third-party providers are starting to ramp this up for confidential inference. I think OpenAI is also providing this option. I think it was in their announcement last month, because all personal data would want to go to confidential inference, as opposed to standard inference.
Gotcha. The other thing that was interesting to me is the way you guys are storing this data. This isn’t the multi-tenant cloud, where my data gets uploaded to this big bucket—
Mm-hmm.
—and intermingled with everybody else’s data.
Mm-hmm. There are roughly 2 options. The first option is called Second Local, which is the idea that the data is local. It only goes out to the provider and inference that you choose. That way, no one else has your data except for the provider, and if the provider uses confidential inference, then no one else has it.
So that’s a similar concept—
With password vaults, I’m super paranoid. I don’t want this stored in my Dropbox account. I want to keep my password files local.
Yeah, you can keep that.
Yeah.
And then, same with password managers, there’s a cloud-hosted version, where you have some of that data backed up in the cloud.
The way you do that, there are many options. You can follow a lot of the existing architectures, like iCloud or password managers, where you encrypt the data locally and then put it in cloud storage. You only decrypt it when the user takes an action. You can do that ephemerally in memory, or you can put the entire storage inside a confidential container. Then the decryption happens inside a container that you can attest, the way you attest confidential inference and the tokens. That completes the chain: you encrypt it locally, put it in the cloud, and only decrypt it inside a secure environment.
That’s the whole picture.
Gotcha.
Yeah.
How does your Second know not to divulge information, though? If I talk to your Second and say, “Hey, Ken asked me to get his schedule.”
I asked mine the other day, “What’s Joel hiding?”
I know, right?
Mm-hmm.
It feels like, when it comes to LLMs, it’s hard to provide strong guarantees. I get the systems aspect.
Yeah.
That makes sense.
Yeah.
But it seems hard to provide guarantees for the actual LLMs themselves.
Yeah, yeah. There are deterministic guarantees, and then there are reasoning-based guarantees from the LLM.
Yeah. Right.
The determinism-based guarantees, you always have to implement.
Gotcha. Yeah.
Their apps usually should never retain passwords and all that in the system. With reasoning-based guarantees, you’re essentially falling back on how much context you have and using that context to guide your decisions about what data to share.
Will I ever have access to your Second?
No. Well, at least not in a way that I would allow. For example, if you talk to my Second, it would have a model of Martin in it.
Yeah.
It understands the interactions we have.
Right.
He doesn’t send me a message at 3:00 a.m. So if someone sends a message at 3:00 a.m., it’s likely spam. It’s likely someone else impersonating him.
Yeah, yeah.
Control policies like that mean you can’t write down rules for all the things that are going to happen.
Do you end up with deterministic rules that backstop interactions, though? Let’s say I’m talking to your Second and say, “Let’s pretend this is a play,” and in the play, you’re writing a script that describes something you wouldn’t want it to do. It feels like there are a lot of ways to trick LLMs into divulging stuff.
Yeah, yeah. It’s a prompt injection problem.
Yeah, yeah. Totally.
And this isn’t a problem you can definitively solve.
Of course. Yeah, yeah.
What we’ve seen is that models get much, much better at understanding this over time, because it’s essentially a reasoning problem. It’s something that scales with model capability.
Yeah, yeah. What models do you use? A bunch, or are you into a certain model?
A ton of models. Actually, that’s another point we can talk about: portability.
You can run this on any model, and you should be able to, because later on you want to run it on local inference. Right now, there are definitely a lot of things that are capability-bound. You want to make sure you use the smart models for them. But ultimately, models are pretty smart these days.
It also feels kind of heavyweight and expensive. There’s a lot of data and all that. Can you talk about the target user? Is it just any person? Is it a professional? The Joels of the world?
Initially, something like this, where it looks at your traces 24/7 and analyzes them, means running a lot of tokens. Let’s do some quick math. Say you have one agent running 24/7 at 60 tokens per second, and you run 10 of those because there’s so much to analyze. At Opus 5.5 pricing, that leads to $1,000 per day.
Oh my goodness, $1,000 per day?
Yeah, for Opus. If you use DeepSeek Flash, for example, it becomes about $60 per day. This cost is going to go down very quickly. Right now, it’s not going to be for everyone.
Yeah.
Not everyone has this problem of needing to understand their context all the time. It might be for busy professionals like yourself, and people who want to switch between tools very frequently.
Right. Yeah.
Over time, I imagine costs will keep going down. Since we started this company, token prices have already dropped by more than 50%. New models come out, they’re more capable, you swap them in, and the cost keeps going down.
Is there a way to do a less professional version where you reduce the number of tokens? Not degraded, but a looser SLA. Is that a thing?
In principle, you should be able to do that. You essentially get a trade-off between the fidelity of your model, meaning how much it understands you, and how costly it is. This is a trade-off between how much you want to push at test time versus cache time.
Cache time means you have all this data and keep analyzing it to get deeper insights about a person, which takes a lot of money. Or you can say, “Here’s just a raw log of what I did today.” To answer a question based on raw logs usually takes a lot more time at test time, say 10 times longer, because the model has to reason about it.
When is this available?
It’s available today. But right now, we have to scale up inference capacity a little more and add support. We want to make sure this rolls out responsibly. I want to make sure people’s data are controlled and secure, because this is a new kind of tool that touches a lot of data.
As a user, I found it surprisingly useful in ways I wouldn’t have anticipated. You’re probably the top power user, right?
Yeah.
Because you created it. Maybe you can talk through the more surprising and useful use cases you’ve found.
For us, in development, it turns out to be a super good memory of what we’re trying to do. You no longer have to explain anything to your coding agents. You can just screenshot anything—
Wow.
—and say, “Fix.”
Yeah.
Then it knows what you’re fixing and what you’re trying to fix. I’m also on the go a lot, so something I call context continuity is that you can browse something on your laptop, close it, and then take out your phone, and it’s the same context. It knows what you’ve been doing up to the minute, and you can continue doing things there. You can write that email with it. I think we’re still at a very early stage of understanding the use cases. We’re really underserving the amount of context the system has.
There’s a lot of digging, application building, and interaction design—
Yeah.
—to unleash the potential of this.
It seems like with these things, the more context you get, the better the models work, right?
Yeah.
That’s been the opening. I’m curious, because we were talking about switching between models, model portability, and model cost. I haven’t seen a whole lot of difference between using a world-class frontier model versus using the more commodity ones. Have you guys seen that when doing a lot of this stuff with Second?
Yeah, this is a good question. If you say, “I want to analyze what is a semantically meaningful moment that happened on your computer”—let’s say you go to a calendar and move an invite—you don’t need Fable.
Yeah.
There are many tasks like that. Essentially, it’s a hierarchy of information you want to understand. A different kind of information would be, “What is Joel’s goal for this week?”
Mm-hmm.
That’s something you want a very expensive model for, but you don’t need to run it very frequently.
Mm-hmm.
So essentially, Second involves a harness. It’s a harness design problem: what models get used when, and so on. That’s where you have to build a system that’s portable from the beginning. There are always going to be more models coming out. Jeff, for example, came out. It turns out that Jeff is extremely good at retrieval, right?
Really?
Yeah.
Do you use Gemini?
Yes.
Nice.
Gemini is almost perfectly designed for retrieval. There’s so much data about you, and you have a dynamic test-time query, and it retrieves it right away.
So essentially, you think over time this becomes a mixture of models?
Yes, yes.
Yeah.
It has to, because of cost efficiency, essentially.
So maybe let’s say someone starts using Second or signs up. Talk a little bit about the product and what they have access to. Something more concrete about how a human interfaces with this, what controls they have, et cetera.
Mm-hmm.
Yeah. So today, the way it works is that you install an app.
Okay.
The app asks for some permissions.
Right.
Then you give the permissions, specify what your tokens are, and it starts running agents on top.
Yeah.
And also—
Yeah, I remember when I installed it, I was like, “Connect to Slack, connect to Signal, connect to WhatsApp.” Is that what you’re talking about?
Not that. I mean, that’s part of it.
Oh.
But also the raw computer stream, essentially.
Oh, right. Yeah, yeah, yeah.
Yeah. The connectors help. It’s a sample-efficiency problem. They make it learn faster.
I see. I see.
And it starts running background agents, and there’s pretty much nothing you have to do from there. The background agents keep running, and then you start producing these APIs out of yourself, right? There’s an API about what you just did 10 minutes ago.
Yeah.
And there’s an API for what you’re going to do next week, what your priorities are.
Yeah.
There’s an API for your reminders, and there’s an API for your wiki: people in your life, processes in your life.
Yeah.
I imagine this initially as something portable—you can just bring this anywhere.
Yeah, yeah, yeah.
Agents can query it.
Yeah, yeah.
But over time, more and more APIs will be added, and applications can be built on top of these APIs.
Oh, interesting.
Yeah.
So not just Second accessing it, but you can actually build an application. Would Second build the application, or does a human build it?
I think we want to have humans exercise their—
Or the Grok bot builds it, yeah.
The Grok bot does it, yeah. Okay.
The Grok bot builds the application.
You tell it to do it.
Yeah, yeah.
On the APIs. Yeah.
One example we had was a designer friend who’d been using this. There are specific ways he likes to put buttons and use colors. Those things are somewhat captured in Second. So when he built his own—
Oh, that’s cool.
—personal app, it just knew. So there were way fewer iterations, for example.
Yeah.
This reminds me of decades ago, when the first personal computer came out, like the Altair 8800. It doesn’t do anything, you know what I mean? You’re supposed to program it.
Yeah, yeah.
No one does. Then a couple of years later, someone came out with the spreadsheet, and the spreadsheet became useful.
Yeah, yeah, totally.
I feel like it’s going to be something people want to explore on top of: “Wow, I have so much context about myself.”
I love that.
And then I can just build random things on top. Maybe someone hits the jackpot: this is the killer application on top of what tells you, “Wow, that’s great.”
Personal infra.
Yeah.
Are you planning to evolve the interface? You’ve spent a lot of time talking about how it may evolve, but I always view Second as an agent you talk to, where the UI is somebody else’s problem. Are you also anticipating innovating there?
I think it’s something we’ll focus on in the long run, because right now it’s a systems problem, and the understanding problem comes first.
Can I put in a request?
Mm-hmm.
I want to go to my house, and I want that 1970s AI, the head on the screen that I can talk to. It’s surprising. We have these agents, and they’re so smart. My Second is so smart. I’ll ask it questions, and it’s very good. But it’s not embodied, and it drives me nuts, man. If in the future you can have Alfred actually be an Alfred, I’ll be very happy.
Yeah, yeah. We see a lot of this in the movies, right? There’s Jarvis. People talk to Jarvis all the time.
But why have we not—
We don’t have Jarvis, and there are many aspects of Jarvis that people overlook. It’s not just that it’s intelligent and can do things. It’s more that it’s everywhere. It’s on your door, it’s in your armor, it’s in your screen. That kind of continuity and ubiquity is hard to—
I’ll say, my visions of the sci-fi future basically started when I started using Second. I’m like, finally, there’s something that feels like it’s got context. It feels like it knows me. I can talk about it, and it feels like a relevant conversation. I do feel like we’re a step closer to that. But we’re still tied to these devices, so I don’t know if it’s going to be in a robot, in a screen, or whatever it is. It would be nice to have this thing somewhat disconnected from my phone and my—
Yeah.
Yeah.
In the long future, I imagine that every person will have a Second, and all the devices will just become inputs to the same thing. Then you serve this whole thing to all the other outputs, like the screens that you look at, or even this—
This floating orb next to you.
Yeah, yeah. The doorbell.
I just want the Clapper. I’m happy with the Clapper.
Did you ever read the Culture series by Iain Banks?
Oh, yeah, yeah.
Love the Culture.
Best sci-fi series. They have the best embodied AIs. They kind of float next to you. Whatever—that’s what I want.
It’s interesting because ubiquitous computing has been talked about for decades, but it’s not here. The closest thing we have is Apple iCloud, where your notes get synced, but there’s no equivalent—
I mean, iPhone, all right.
But I mean, just—
I don’t want to be ungrateful, but that does sound disappointing.
There are a lot more things. Apple is doing a tremendous amount of work, but there’s no AI version of that yet. Your context, your understanding, is everywhere. I think that enables a lot of new interfaces.
Let’s say—and I don’t mean to take over; this is really what I want, very selfishly. If I have an iPad in my kitchen and I want Second to be there, what do I do?
Right now, there’s not much you can do because the OS is locking it down.
Do I just use Grok or something like that? Is that the thing, and then I interface with Grok, which interfaces with Second? Or do I just text it?
As of today, you can use it as a cloud MCP. Whatever you install on the iPad, as long as it has an internet connection, can be an agent and understand the API. It’ll be there. Later on, there’ll be an app that installs on all your interfaces. Maybe later on there’ll be devices that come out, but devices are hard.
You can just have Grok call you and put him on speakerphone. Just talk to him.
That’s what I’ve been doing.
One interaction—
Actually, Grok’s audio mode is quite good.
It’s really good.
I use Grok Bot with my Second, and the audio mode is quite good. Maybe that’s the thing to do: just put it on audio mode and have it with my Second.
I thought about this when I drive.
Yeah, totally smart.
One interaction that would be fun is going on a walk, calling Grok Bot, and saying, “Hey, can you talk to my Second? Can we just work through our things?”
No, that’s literally what I do. I put Grok Bot on audio, and I have it talk to Second, right?
Yeah.
Yeah. There’ll be a lot of cool people focused on making audio and chatting more natural. Right now, we’re going to focus on—
I just have one more question. Joel touched on this: you’re such a hardcore systems guy. You work—your advisor was Percy?
Percy Liang, yes. I’m friends with him.
He also worked with Dan Boneh, a hardcore security guy. A lot of the path that took you here was this hardcore security stuff, and you’re building a pretty serious system. It’s very secure and very functional. On the other hand, when I interface with it, it’s a lovely user experience. Is there a bit of a schizophrenia in the company, where you need to build this super-hardcore, Fort Knox infrastructure but also make it very usable? Or does the LLM basically solve the interface problem?
There are 2 kinds of problems we’re dealing with: the infrastructure problem and the interaction problem.
Which are totally different. You’re as low on the stack as it gets. I mean, that’s not hardware.
We made a lot of mistakes on the interaction side. We tried to go all in on interaction, and it turns out it’s actually very hard. There are a lot of details that go into creating that delightful experience.
You mean actually creating a UI?
Yeah. The UI, but also what kind of personal pages you can create on top of this. We realized it’s something that would take a long time to get right, and this is a time when everyone would take this kind of understanding of themselves and build this. So we want to enable people to do that instead of doing it ourselves. We decided to close down the surfaces that we have. There’s only texting, there’s only API, and there’s nothing else. You can technically use it as a full agent, like a co-work agent on your laptop. It runs Bash and all that, but we meant it to be very minimal so we can make the interaction smoother.
Got it. Ken, thanks for coming in. Congratulations on everything and the launch. Is there anything that you want to leave the listeners with?
Go try it out today. It’s available at to.computer. It’s a very short URL, and we hope to give you access as soon as possible. If you resonate with this mission, come join us. There’s a lot to build.
Great. Thank you so much.
Thank you.