《No Priors》第10期｜与 Copilot 首席架构师、Minion.AI 创始人 Alex Graveley 对谈

So let’s start with some background. How did you end up working in tech or AI?

Alex Graveley

Tech was earlier. I started really young, and I got really into Linux when I was 14 or so. That was right around the time when the web was a new thing and you had to work to get on the web. Linux just sort of found me; it popped into my life, and I really liked it.

I read the GPL really early, and it struck me. The idea of helping in the open and making things freely available so other people could learn from them, like I was learning from things, seemed great. So I spent many years working on open-source stuff.

I don’t know, I guess I didn’t realize there was legacy technology. Windows was popular at that time, so people ran their data centers and networks on Windows. This Linux thing was very strange, and people didn’t really know what it was. I spent many hours compiling kernels and hacking on things.

What was the thought process behind starting Hackpad?

Alex Graveley

I had just finished 4 or 5 years at VMware, and I wanted to get into startups. I knew that. I left VMware and started working on an education startup, like many of us do.

It’s a rite of passage.

Many founders start with an education startup.

Most of us have done either that or consumer social.

Education is very hard. After 9 months, I was like, “All right, this isn’t going anywhere. I don’t know if there’s a value proposition here.” The value was that I learned you have to make something that is both achievable and that people want to pay for or spend their time on.

I was just fishing around. I was living in a warehouse in San Francisco with a bunch of Burning Man people, and we were having trouble organizing large-scale Burning Man projects. I forked Etherpad and started hacking on it. I recruited a friend in the community to start working on it with me, and it grew from there.

Before we did YC, many of the large Burning Man camps were using it to organize their builds.

Can you describe the product experience?

Alex Graveley

It was a real-time text editor, kind of like Google Docs. Google Docs was the only other product that did that at the time. It was nice because it highlighted who said what, so you could track down whether somebody had made a contribution and say, “What did you mean?” or “I heard you say something about this.”

That was very useful in large-scale, anonymous or pseudonymous groups where you don’t really know who has ownership over what, like Burning Man camps.

Then we did YC, and many of those people used it. My hack was to go to YC and try to get all the companies doing YC to use my product, which many of them did. I also took extensive notes on all the YC presentations using the product, and everyone would look at them.

We were able to get Stripe, and Stripe used us for many years. They built their first knowledge base with us, I think, and used it for a long time. A bunch of other big companies used it as well. We were very lucky.

After the Dropbox acquisition, you worked on what became Paper. How did you think about what you wanted to work on next after Paper?

Alex Graveley

I spent the next few years poking around at things. I knew that I wanted to make a robot that does things for you. There was this company called Magic, so I worked at Magic. They were doing text-based personal assistance.

Do you remember this one? Just like everybody starts an education startup, Magic is one of those names that keeps cycling back. There’s a really cool AI company right now called Magic as well, so I feel like there are names that persist from generation to generation, which is really cool.

Alex Graveley

I haven’t seen a demo yet, but it sounds like they’re doing the right thing. There are a few people doing that kind of whole-repository change, so it seems like a great direction.

What’s one learning from Magic? It was an early company trying to do sequence-to-sequence machine learning in general, right?

Alex Graveley

No, it wasn’t trying to do machine learning. It was all operations. It was a very operations-heavy business. They had teams of people working 24 hours a day, cycling in and out, and they would lose context. They were all busy because they were trying to deal with lots of people and lots of requests all the time.

It was really a crash course in human behavior: What do people do under stress? How do they act? What do they say? What can you train, and what can’t you train? Can you bucket things? The answer is no. Humans are complicated, especially in text form.

The beauty of the web and traditional UIs is that you fill them in, and if they don’t do what you want, then you decide whether to go forward or not. Text has this annoying property where you can be all the way at the 99% mark and then change the goal entirely.

It’s complicated and hard to understand what people want, especially when you’re dealing with the real world. Flights get delayed, passwords get lost, and all sorts of things happen.

Do you think we’re the last generation to deal with that? In other words, it feels like we’re about to hit a transition point where agents can actually start doing some of these things for us for real, whereas before, all these products started with operations-heavy approaches.

I remember there was a very early personalized search engine called Aardvark. If you looked behind the hood, there were a lot of operations people and a little bit of algorithmic routing. You would describe what you were good at, and they would try to send questions to you. I think people were doing some of the routing, at least.

A lot of people wanted to build complex bots or agents that did rich things, but the technology just wasn’t there. There were also some startups trying to do scheduling and assistance 6, 7, or 8 years ago, and it felt like they were a little early.

What were they called? Clara Labs—that’s the one. I remember that era, and then we had Operator. On the question-answering side, we had Jelly. There was a whole series of those companies.

Alex Graveley

I remember that era. What we were able to do with Magic, as an aside, was interesting. I started working there because I wanted to work on the AI part. Somewhere in there, Facebook M started as well, and it was a fun place to try to learn everything I could about solving those problems.

Before Transformers, sequence-to-sequence was the previous iteration. We were able to take all the histories of the chats between assistants and people and train a little model on them. By today’s metrics, it was a little model, and we ran it so that it would show gray text in the text bar. People could edit it and hit Enter.

We measured how much time the operators spent typing with it and without it. It saved about 30 minutes across 100 people per day during an 8-hour shift. That was my first shipped AI product, I guess.

How did you end up going to Microsoft?

Alex Graveley

There was a bunch of other stuff along the way. After that, I got into crypto. My friend was doing hCaptcha, which was a sort of CAPTCHA marketplace. It’s now something like the number 1 or number 2 CAPTCHA service in the world, which is crazy. We launched that, and it was fun. We annoyed people the world over for many, many man-hours in aggregate.

Then I left to work with Moxie on a cryptocurrency for Signal. That was really fun and complicated, and it all worked in a few seconds. We were shooting for Venmo quality, and I think we pretty much got there.

When you think about crypto in the context of AI, people talk about it in a few different contexts. One is programmatic money as code running, which could create interesting things from an agent-driven perspective. The other is identity. Worldcoin would be one example, but there are other efforts to secure identity cryptographically on a blockchain in an open way and then use that identity to differentiate between AI-driven agents and people.

Do you think that’s going to be important, or does that stuff not really matter in terms of the identity portion of crypto and how we think about the future of agents?

Alex Graveley

The honest answer is that I think we’re going to go through many years of extreme discomfort where AIs pretend to be things, confuse people, extract money from your grandparents, drain people’s life savings, and do things that are scary.

OpenAI is trying to do its best, but for some reason the focus has been on OpenAI doing everything. Instead, we should build systems that prevent that. We should pass legislation that drops a hammer on people doing that kind of stuff. Unfortunately, it seems like we’re going to need some really bad things to happen before we align correctly.

I’m not really scared about AI killing us, although I’m very grateful that there are people thinking about it. I’m more worried about bad people using new technology to hurt us.

Ilya Sutskever has some really interesting thoughts on this. He was one of the main authors—or the last author—on the Transformer paper before he started SSI, and he’s brought up the idea of how you stress-test society relative to the coming wave of AI. I think that’s an interesting concept.

Alex Graveley

It’s a great way to look at it. It’s not as bad as it could be. Most of the things you want to spam either have a spam blocker or are somewhat difficult to create an account on. Doing a better job of filtering sock-puppet accounts is going to be really important going forward.

I like what Cloudflare is doing with fingerprinting instead of visual CAPTCHAs, which aren’t good enough anymore. One saving grace here is that many of the things you would want to do cost money. Calling everybody costs money.

Sure. Texting everyone should hopefully be illegal, but it also costs money—maybe not enough money to prevent these things. Agents can make money, so you have to look at the trade-offs that cost.

That’s interesting. The other nice thing is that grandmothers don’t have crypto. They have bank accounts, and bank accounts can be traced.

I’d say it’s somewhere in the middle. Imagine that North Korea has been trying to do this to us for a long time, and now there’s a North Korea with more resources, or one that’s more distributed or whatever. We have some mitigations, but we need more, and we need to think about it a lot more.

How did you end up at GitHub, and how did you end up working on Copilot?

Alex Graveley

While I was working on MobileCoin, my dad’s kidneys failed. I tried to donate a kidney, and they found a lump in my chest as part of the scans they do. I had most of my right lung removed in 2018, so that was a big deal. Healing from internal injuries takes a lot longer than you think.

The happy story is that I’ve been cancer-free for over 4 years, and my dad got a kidney transplant as well. Things are good.

After I recovered for quite a while, I begged my friend for a job. I figured I should start working again. I worked on some random things at first. I converted GitHub to use its own product to build GitHub, which was kind of fun.

I think people still use Codespaces to build GitHub, which is pretty cool. Then the opportunity to work with OpenAI came up. Because I had been tracking AI and was pretty aware of what was going on, I jumped on it.

Was that proposed by OpenAI or by GitHub? Who initiated it?

Alex Graveley

I don’t know the exact beginnings. I know that OpenAI and Microsoft were working on a deal for supercomputers. They wanted to build a big cluster for training, and there was a big deal being worked out. Some software provisions were thrown in—I think Office and Bing, probably—and GitHub was like, “Maybe there’s something GitHub can do here.”

I think OpenAI threw over a small fine-tune and said, “Here’s a small model trained on some code. See if this is interesting.” We played around with it.

I have to remember now. This was before I knew very much, so it was definitely not a Da Vinci-sized model. I don’t know what size it was—probably less than 10, but I can’t remember. I learned later that it was basically a training artifact. They wanted to see what introducing code into their base models would do.

I think it had positive effects on chain-of-thought reasoning. Code is linear, so you can imagine that you do things one after another, and the things that came before have an impact. It was not that good. It was very bad. It was, as I said, just an artifact and a small sample of GitHub data that they had crawled.

We played around with it. The first 2 people after Uğur who got hold of the model and started playing with it were me and Albert Ziegler. Albert was able to say, “It doesn’t work most of the time, but here it is doing something. Here it is generating something useful.” It was only Python at that time.

We didn’t really understand anything. It was just lobbed over to us. That was enough to say, “Let’s fetch a couple of people and start working. Let’s see if there’s anything there.”

The first task was to test it and see what it did. We crowdsourced a bunch of Python problems and wrote in-house tests that we knew wouldn’t be in the training set. Then we started fetching repositories and finding the tests in them so we could generate functions that were being tested and see whether the tests still passed.

A new pytest feature had been introduced that let you see which functions were called by the test. You could find that function, zero out its body, ask the model to generate it, rerun the test, and see whether it passed. I think it was less than 10%, something like that.

The dimensions are: How many chances do you give it to solve something, and how do you test whether it worked? For the standalone tests, we had people write test functions, tried to generate the body, and if the test passed, then it worked.

For the in-the-wild test harness, we would download a repository, run all the tests, look at the tests that passed, find the functions they called, make sure they weren’t trivial, generate the bodies, rerun the tests, and calculate the percentage.

It was some very low percentage up front, but we knew there was a lot more juice to squeeze. We could get all of GitHub’s code into the model, along with a bunch of other tricks that we hadn’t even thought of at the time. Eventually it went from less than 10% on the in-the-wild tests to over 60%.

That means it could generate code for 1 out of every 2 tests, which is insane. Somewhere along the way, it went from 10% to 20% to 35% to 45%. We did more prompting work so the prompts got better. We used all the versions of the code instead of just the most recent version, and we used diffs so the model could understand small changes.

When we first started, we didn’t know what we had. We were just trying to figure it out.

Was the test-user group just 6 of you, or was it a larger group?

Alex Graveley

The first iteration was an internal tool that helped people write these tests. Then we wanted to see whether we could turn that into a UI that people would use—some way to cover up the fact that only 1 in 10 things passed. We tried a few UI approaches.

OpenAI said it would be nice if they could test these model fine-tunes more quickly. They asked whether we could do something like a VS Code extension and just use autocomplete. We did autocomplete at first, and that was a big jump.

They were still thinking in terms of Stack Overflow, but I didn’t have any ideas for how to beat Stack Overflow with this thing. We could play with things in VS Code that were closer to the code, so we tried that.

Autocomplete was useful. It would show a little pop-up box, like autocomplete does, and you could pick some strings. It wasn’t exactly the right metaphor, because you had generated code mixed in with the specific terms in the code.

We tried adding a button over empty functions so it would generate them. You could hit a Control key and create a big list on the side to choose from, or use a little pop-up. We tried every UI we could think of in VS Code, and none of them really worked.

I had an idea that it should work like Gmail’s gray-text autocomplete. I was enamored with that product. It was the first large-language-model deployment in the wild, quote unquote. It was fast and cool, and the paper was great. They gave you all sorts of details on how they did it and all the workarounds they had to use.

That was always in the back of my head. Gmail’s completions weren’t good either, but it seemed like the right thing. Somewhere along the way, I figured out a way to hack it into VS Code. I made a little demo video.

Was there support to build real support for it within the organization?

Alex Graveley

It was a little complicated. We were basically a skunkworks project, so no one really knew about us. We would go to the VS Code people and say, “We need you to implement this very complex feature,” and they would say, “I don’t even know who you are. What are you talking about?”

There was definitely some politicking to get the VS Code team to dedicate resources to it on a short time frame. We were moving really fast. It was less than a year from the beginning to the public launch.

Was there a particular metric where you thought, “This is good enough. We need to put it in the public product”?

Alex Graveley

We had a nice, long window of public access before general availability, when it was free and people could use it. We optimized for different groups of people: more experienced people versus newer people, and people from one area versus another.

That gave us good statistics. We learned that speed was the only thing that mattered. There was some crazy statistic that every 10 milliseconds meant 1% fewer completions that people would accept. That adds up.

Network latency to India?

Alex Graveley

In the first few months of public release, we noticed that completions in India were really low. For whatever reason, they were significantly lower than in Europe.

It turned out to be network latency. OpenAI only had 1 data center, and it was in Texas. If you can imagine, you’re typing in India, and the request goes through Europe, over the water, down to Texas, and back again and again.

If you’ve typed something that doesn’t match the thing you requested, the completion is useless. You don’t get a completion.

I know it’s obvious, but this has happened on every product I’ve ever worked on. When I was at Google, I worked on a variety of mobile products, and it was the same thing. Page-load times obviously matter. Search in general—100 milliseconds makes a huge difference in market share.

Alex Graveley

It’s crazy how much speed matters. Once we figured that out, we knew we had something awesome. People who were close to Texas thought it was fantastic. We had a Slack channel, and people were posting in it all the time.

The most fun thing was when people would pop up and say, “I don’t program, but I just learned how to write this 100-line script that does this thing I need.” It was amazing.

These models are really great at finding patterns. Once we had the UI mechanism that worked—which was ghost text; you hit Tab and it inserts—and we had done the work to get multiline working, we needed something that would show that off. The key insight was realizing that programmers think in blocks. If you can autocomplete the rest of the current block, that’s what people want. People are judging what this block is going to do, and if it does the right thing, they know quickly. If you can complete an if statement and it does the right thing, then it’s easy to judge. We knew we were onto something.

Then it was just squeezing as much performance as we could get out. We basically never found the bottom. We made it as fast as we could, and it was still improving on completions. That led to: “I know there’s this plan for Azure to run OpenAI in six months. We need you to do that in the next month. Let’s figure out how to make this happen,” because we wanted to run a bunch of GPUs in Europe so we could hit Asia. At the time, the other places we could run them were the West Coast and Texas. Microsoft stepped up, we got it running, and pretty much after that we launched.

Were you surprised by the uptake after launch?

Alex Graveley

No. Our retention rate was 50%, and it never went below that. Months later, it was still above 50% by weekly cohort, which is insane.

We didn’t know whether people would pay for it. I lobbied pretty hard for going cheap and capturing the market.

How did you think about inference costs for this thing at the beginning?

Alex Graveley

Our estimates were wildly off. We thought it would cost $30 per month per user, on average. Once Microsoft was able to get the Azure infrastructure working, we could fork off a little bit and make more accurate projections.

There were moments where we would wait for the results and ask, “How much is it going to cost?” The first big result was, “It’ll cost $10 per month.” I was like, “Oh my God, it’s so much cheaper than we expected.”

We hadn’t optimized for price yet. Then we optimized for price a lot, and now it costs less than that. It was very fortuitous. We thought it might be enterprise-only because that was the only market that would be willing to pay for it. $30 per month is a lot, especially with no margins.

For 40% of your code, it’s not a lot.

Alex Graveley

That’s the thing. We know that now. The crazy thing is that there are whales out there—people for whom it writes 80% of their code, which is insane.

If you extrapolate all of this 3 to 5 years out, are there basically going to be agents writing code for us? Is 95% of code going to be written by Copilots, with humans directing them? How do you see the world evolving over the next few years?

Alex Graveley

It’s hard for me to imagine what that world looks like because it’s such a shift from having my hands on something and knowing that it’s right to where we are now. Now, I mostly know it’s right, or I have a sense that it’s right, but I have to test it and see it run to know that it’s right. Trusting that the code will work without doing that is a pretty crazy transition.

You could imagine code review after some chunk of code, or some other kind of quality check.

You could review only the dangerous parts, or the scary or confusing parts.

Alex Graveley

If that’s the goal we want to reach as a people, I think every barrier to it is achievable. We can code-review only the dangerous, scary, or confusing parts. We can train a model on functions before and after changes and say, “This looks like a more polished version of the function.”

We can start with a basic main loop and add everything piece by piece, with tests, so we know what works and what doesn’t. We can have the model keep generating the logical next feature. All of these things will get figured out.

What’s the idea behind Minion?

Alex Graveley

I mentioned making bots that do things for you. It’s a broad topic, but I think that’s where we’re going. The next few years in AI will be about taking action—not just answering questions or writing copy, but actually helping us in our daily lives.

That could mean organizing your schedule, booking flights, finding a trip for you to take, doing your taxes, or telling you which contacts you haven’t talked to in a long time and should reach out to.

There’s a lot we can do by giving AIs access to information and letting them act on that information in a controlled way that checks to make sure we’re aligned. I think that’ll be a really fun future.

You can imagine Copilot applied to everyday activities. Copilot gives you a little bit of help in your code editor, and I want Minion to give you a little bit of help outside your code editor.

How did you decide to work on Minion specifically?

Alex Graveley

I quit Magic because I couldn’t figure out how to hook the AI up to data. I thought, “In order to improve the quality, I need a PhD in math. I don’t know what to do now.” The models got better, and that specific problem started to seem solvable.

That was where interacting with the real world had broken down in the past. Flights get delayed, prices change, and not just a little bit. They change all the time. The seat you want at the concert you want to go to might not be available.

AIs are a kind of compression of everything in their training set, but they aren’t a real-time mechanism yet. I thought, “Maybe we can work on this old problem of how to make a bot do things for people. That’s what we want, so let’s go make it.”

Maybe I could use the excitement from Copilot to launch into something incredibly hard, but something for which I believe the technology is now available. That’s the only reason I’ve ever gotten into startups. I want to do a startup so that I can do a harder startup, or a project so that I can do a harder project.

Is this one sufficiently hard?

Alex Graveley

This one is hard. It’s good and hard. There are fun things you learn along the way that keep you engaged.

With Copilot, it turns out that code is pretty special because you can run it. If an AI generates code and it runs, then you know something about that code that you wouldn’t necessarily know with text.

Doing agent-like work on the internet or in apps has many similar properties. You can learn from what people do, see whether it’s a success or failure, and figure out what to optimize based on what works and what doesn’t. You can figure out what’s annoying and try to improve it.

I think these things compound in some really interesting way. The goal is straight out of science fiction: You want to be able to say, “Hey, computer, file my taxes,” and have it do the right thing. I think we can get there in the next few years.

It’s also fun to think about how to break these things down. It turns out that breaking down tasks in the same way humans do works for AIs as well. You take a complex task, write a list of things to go through, and execute it.

You break down complex tasks, figure out whether there’s any information you need, maybe write some code, ask some questions, or query some data. Humans don’t usually write code to do their taxes, but they do go through a list of pay stubs. That’s also executing a for loop.

I like data sets that don’t exist. People clicking on things on the web represent a gigantic data set that is currently unowned, and I think that’s pretty exciting. There’s a lot to learn from it.

You’re hiring, and one thing we’ve talked about is that it’s a funny thing to hire for when you’re building products with this new set of technologies. Working in machine learning for the last decade may not help you that much. How do you think about the people you need?

Alex Graveley

It’s been strange. One thing I don’t know if other people have noticed is that it seems very bimodal. Very senior people get it, and very young people get it. I don’t know that the middle has caught up or realizes what’s happening yet.

Are we on the other side, or are we in the middle?

No, I just mean the middle in terms of experience. It’s kind of like an experiential version of the midwit curve.

You tell someone who is very naive and someone who is very experienced, “I can make magic. Here’s how I can do it.” The naive person says, “That’s awesome,” and the experienced person says, “Maybe.” Almost everybody else says, “That’s not possible,” or “I don’t see it.”

A number of companies I know are trying to build that intuition internally. This is the first time in a long time that I’ve seen certain founders start coding again. They’ll have a multi-person company and become so enamored with what’s happening that they dive in.

Some founders have told me that certain team members just don’t have the intuition for what this can do. They don’t know what to build, where to start, or how interesting and important it is.

It’s almost like a founder mindset is needed. This is a new set of capabilities, so how do you learn what they are and then apply them? People lack a natural intuition for what this does right now.

Alex Graveley

It’s definitely not intuitive. There are many kinds of learners and many kinds of programmers. It takes a certain kind of programmer to be comfortable with the idea of, “I don’t know what’s going to work. Let me try some things.”

That’s similar to the attributes you need for the web or for making products: being able to test things, look at the right metrics, and find the right metrics. Those are useful, reusable skills.

The intuition and tenacity it takes to say, “This doesn’t work at all. What do I do?” are still rare in the field because there’s so much uncertainty. It’s easy to try 10 things, see that they didn’t work, and conclude, “This is impossible.”

The natural reaction to something with 10% performance at the beginning is, “Why even bother? We’re never going to get there.”

Alex Graveley

The crazy thing is that these things work at all. Not only that, they scale and improve as they scale. Most things break at scale. Almost everything breaks at scale, and that’s what you hire people to deal with. With every doubling, something usually breaks and you have to fix it.

It’s pretty crazy to think about what kinds of emergent properties might still be out there if we can make these things 10 times bigger. I’m always in favor of people pushing those limits. Even the best people can’t explain what’s going to happen when you go bigger.

We built the Large Hadron Collider for the same reason. We think we might find something. I’m grateful to be alive during this time.