Shopify 的 AI 阶段跃迁：2026 年使用量爆发、无限 Opus-4.6 Token 预算、Tangle、Tangent、SimGym——Shopify CTO Mikhail Parakhin

Shawn Wang

Okay. We're here in the studio—a remote studio—with Mikhail Parakhin, CTO of Shopify. Welcome.

Thank you. Welcome.

Shawn Wang

I don't even know if I should introduce you as CTO of Shopify. I feel like you have many identities. You led the Bing ML team, I guess, or the ads team. I don't know; people variously refer to you as CEO or—I don't know what the previous role at Microsoft was.

My previous role at Microsoft was—I actually was the CEO of one of Microsoft's business units, which included, as we discussed, all the things that people like to laugh about, including Windows, Edge, Bing, ads, and everything.

Shawn Wang

Yeah, yeah. What a wild time. You've obviously done a lot since you landed at Shopify. One of the reasons I reached out was because you started promoting more internal tooling, primarily Tangle, but also a lot of people have seen and adopted Tobi's QMD. Obviously, I think Shopify has always been leading in terms of engineering. I think it's just more recent that you guys have been more vocal about your AI adoption. Is that true?

Well, I think AI tools in general are a fairly recent development. At Shopify, at this stage of its development, we're developing AI in-house, building tools that use AI, and interfacing with the wider AI community. Those are on a runaway trajectory, so it's just a natural byproduct that we talk about it more.

Even yesterday, Andrej Karpathy tweeted about ways to organize your agents to store the data and then look up the data so that you don't have to research or lose context every time.

A little bit tongue in cheek, I tweeted that we've done it much earlier, and we even have different approaches—Tobi and I. Tobi, of course, is a big fan of QMD, and I'm more of a SQL, SQLite fan. But they're very similar things that we've already done here. The point is, we're a very dynamic, explosively growing company, and we have to be at the forefront of AI adoption, obviously.

Shawn Wang

Yeah. Your team kindly prepared some slides that we were going to bring up onto the screen. I think I can screen-share, and then we can go through some of the shocking stats that maybe put some numbers to what exactly is going on. So here we have an internal AI tool adoption chart. What are we looking at here?

1. AI Adoption Hits Phase Transition

Yeah. This is very interesting statistics. This is the number of daily active workers—think of DAU, basically, the active users of AI tools—as a percentage of all the people in the company, right?

And then there are different AI tools. You could see 2 things here. One is that green is total. Green is just total, so you could see that it approaches really 100% by now. It's hard to do your job now without interacting deeply with at least 1 tool.

You could see another interesting thing. Just as many people commented in December, there was a phase transition when suddenly models got good enough that everything took off and started growing. Many people noticed that small improvements accumulated into this big change in December, roughly.

The other thing I would claim you could see is that CLI-based tools and tools that don't require you to look at the code are becoming more popular. You could see various versions of Claude Code, Codex, Pi, and internal development tools taking off.

Exactly. I think blue is our Rover, just an internal agent for coding. Tools that require an IDE, such as GitHub Copilot or Cursor, aren't exactly shrinking, but they're not growing as fast. The red line is the IDE kind of tools, so you could see that they're not experiencing as fast a growth.

Shawn Wang

As I understand it, basically every employee has their choice, right? You choose whatever tool you use, and then you're just doing a daily survey or something.

Exactly. The push is to get your job done. You can use any tool, and we effectively fund unlimited tokens for everybody. We do try to control the models that people use, but from the bottom, not from the top. We basically say, "Please don't use anything less than Opus 4.6."

Shawn Wang

Oh.

Some people end up using GPT-5.4 Extra High. Some people use Opus 4.6. There are pluses and minuses to going for a full 1-million-token context window versus not, but we try to discourage people from using anything less than that.

Shawn Wang

Yeah, yeah. Got it, got it. The next chart really shows the expansion and the December 2025 inflection, right? People are using a lot of tokens. I think it's also really interesting that no one was abusing it in 2025. Compared with this year, there was almost no growth. It still probably gained 50%.

This is just a different scale. It's still exponential growth, at just a different rate of expansion. There was an inflection point, and I would claim the super-interesting part here is that you can see the distribution becoming more and more skewed.

The top percentiles grow faster. That means the people in the top 10th percentile consume tokens faster than people in the 75th percentile, and so forth. The distribution skews more and more toward the highest users, which is—I don't know what it tells me. It feels not ideal, to be honest. Or maybe it's okay. We'll see.

Shawn Wang

What does it feel not ideal? Is it because of quantity over quality, or what's the concern?

Because, take it to the limit. That means if this rate of separation continues, in a year there will be 1 person consuming all the tokens. It's just kind of strange.

Shawn Wang

Yeah. I think internal teaching and all that will help distribute things more widely. But in the early days, the people who are more AI-pilled will obviously find more ways to use it than the people who are less AI-pilled. Maybe let's call it that.

I'll just pause from the rest of the slides. We'll go back to them, but I want to review this. There are a lot of CTOs of large companies like yourself, and they're all considering some kind of token budget. I think it's something Jensen Huang has been talking about, where if your $200K engineer isn't using 100K tokens every year, they're underutilizing coding agents.

Of course, Jensen Huang would say that, but it seems like a very quantity-over-quality approach. Some people are basically saying, "Is this comparable to judging engineer quality by lines of code?" We know that's flawed, but it's better than nothing. I don't know if you have a management take on how to view these kinds of metrics.

2. Token Budgets Need Critique Loops

Well, you're baiting me. This is my favorite topic. If you let me, I'll probably talk for 2 hours on just this. I have a lot of things to say.

I do think Jensen got a lot of bad press from people saying, "Of course, the cake seller says we don't need enough cakes." But I actually think that's undeserved. I think he's actually right.

Shawn Wang

He's directionally correct.

Yeah.

Shawn Wang

Yeah.

He's directionally correct, for sure.

Shawn Wang

Who knows what the right number is? Yeah.

The thing that I do want to say—and this is something that we learned through trial and error, and it's very important—is 2 things. One is that it's not about just consuming tokens. You can consume tokens, and in fact, the anti-pattern is running multiple agents—too many agents—in parallel that don't communicate with each other. That's almost useless compared with just having fewer agents that burn tokens very efficiently.

Setting up the right critique loop, especially with high-quality models, is important. One agent does something, and the other one, ideally with a different model, critiques it and suggests ways to improve it. The agent redoes it with this critique. It takes much longer, so people don't like it because latency goes up. They have to wait until this debate is happening, but the quality of the code is much higher.

Another thing, since you mentioned it: the overall budget is just like lines of code. Lines of code are exploding for everybody right now—partially because AI is really moving the ball, but partially just because AI can write a lot more code. It doesn't get tired. So you have to have a very strong narrow waist during PR review. Otherwise, the number of bugs will go through the roof.

It's this unexpected consequence of volume trumping everything. I would claim that by now, a good model writes code, on average, with fewer bugs than the average human. But since models write so much more of it, more of it will make it into production.

Shawn Wang

You still have more bugs.

Yeah. You have to have very rigorous PR reviews—also automated, of course. For me, the important metric is the ratio of budget spent during code generation versus budget spent on expensive tokens, like GPT-5.4 Pro or Deep Think from Gemini, checking PRs.

Shawn Wang

I noticed in your chart you didn't have any review tools. Do you just use Claude Code to review tools, or do you have another set of review tools like the Greptiles, the CodeRabbits, or Devin Reviews? I don't know if you've had those specialist review tools.

At PR review time, you want to run the largest models. Codex or Claude Code is not going to cut it. You need pro-level models if you really want to stem the tide of bugs from going into production, and you need to spend a lot of time with the models taking turns. You don't want a big swarm of agents.

In fact, you end up in a different, dualistic world where you generate not that many tokens. You generate few tokens, but it takes a long time because these are expensive models taking turns, rather than many agents trying to do many things in parallel. That's why I feel like I haven't found good tools, so we're using our own for PR review for now.

Shawn Wang

Yeah. I think a lot of companies are building their own, especially to fit their needs.

Mm-hmm.

Shawn Wang

You also have a chart here, going back to the slides, on PR merge growth, where we're now at 30% month over month rather than 10%. The estimated complexity is also going up. This is productivity, right? Presumably, there's more stuff going into the codebase and more features getting worked on.

I'm curious about the backlog. I actually don't mind a pro-level model taking an hour or 2 hours to review my PR, because I've dealt with humans who take a week to review my PR. I keep pinging them on Slack: “Hey, review my PR.” I think there's a trade-off here where it still makes sense.

3. Agentic Code Buckles Pipelines

Exactly. That's exactly my point. On one hand, you can tolerate longer latencies for PRs. On the other hand, the real problem right now isn't spending time waiting for a PR. The real problem is that, since there's so much more code, the probability of at least some tests failing goes up. Then you keep failing, and you have to find the offending PR, evict it, and retest without that PR, so the deployment cycle becomes much longer.

In terms of the overall time to deploy, it's a total time savings if you spend more time on a longer model thinking for an hour, because then you don't have to spend all that time during testing and rolling back the deployment.

Shawn Wang

Yeah, totally.

Tobi Lütke

So—

Shawn Wang

That's still worth it. You don't look at the individual; you look at the aggregate and the change in the aggregate system.

Tobi Lütke

Exactly.

Shawn Wang

I'm curious if this PR mentality and the CI/CD paradigm will eventually change. A lot of people obviously want a new GitHub, but I even wonder if Git is the problem. Is that the bottleneck? Is the concept of a PR a bottleneck? Do you guys use stacked diffs? I don't know if that's a merge-queue, stacked-diff type of thing.

We use stacked PRs. We use Graphite; we work with Graphite a lot. The overall CI/CD system in general, and the interaction with the code repository right now, is clearly the main issue and the bottleneck for us. It's the highest thing on our minds.

I would say we probably need a different metaphor, or a completely different design for how to process code in the new agentic world. I haven't seen anything dramatically better yet. Everybody right now is just trying to keep their head above water because there are so many PRs, everybody's CI/CD pipelines are starting to creak, the times are increasing, and the number of bugs slipping by is increasing. You have to clamp down.

We're in a situation where we need to first stabilize that story and then start thinking, “What could be a completely different and new world?” I know some people are working on it, but I haven't seen anything super compelling yet. Clearly, the old things that were designed for humans will need to be morphed into something new.

Shawn Wang

One of the things I think about is that a merge conflict is basically a global mutex on the whole system. In human organizations, we do have something like that: the company stand-up. Other than that, it's actually fitting for us to be somewhat decentralized, somewhat plugged into one stream of information, but somewhat lossy. It's okay that not every delivery is atomically consistent. We're not dealing with a database sometimes.

This is a very good point. Since humans don't write code too fast, that global mutex isn't too bad. Once you start writing code at the speed of machines, it becomes the bottleneck.

What do you do? Maybe—and I can't believe I'm saying this, because I'm a lifelong opponent of microservices, and I always thought they were a really bad idea—now that you're saying it, maybe microservices will make a comeback in the new world. Then you can ship things independently in tiny pieces, and managing all that complexity automatically will be much easier. I don't know. We'll have to see.

Shawn Wang

Yeah. I don't know what the Microsoft or Shopify thing is, but I read this paper from Google where they have a monorepo that deploys into microservices.

The other concept that I think about a lot is the Chaos Monkey concept from Netflix: being able to create a robust system where you have service discovery and independent microservices, and probably a fair amount of duplication. That's how an organic system scales—that you have slack, robustness, and de-duplication. I forget the term. These aren't exactly the terms I'm looking for, but I can't really think of the words.

Okay, I was going to go into Tangent and Tangle. We discussed the overall stats that Shopify has, but some pretty cool stuff that you guys are working on is your ML experimentation and your sort of auto-research training pipeline. Presumably, you're much closer to this one because it's a personal hobby of yours. How would you explain them together? I thought we had a slide with the system diagram.

4. Tangle Makes Experiments Reproducible

Tangle first, and then Tangent as a thing on top of Tangle. Tangle is the third generation, I claim, of systems for running any data processing, with a bias toward ML experiments, but not necessarily. It's for any sort of data-processing task where you need to iterate and share, and you have scale so that you want maximum efficiency.

Normally, imagine you're a data scientist or an ML practitioner. You would get Jupyter notebooks or Python scripts, manage the data, produce TSV files, and put them in some HDFS or something. Then you would notice that they have weird missing values. You go and write another script that replaces them with dashes.

Then you run some operation where you need to filter bots, so you run a LightGBM model that removes the bots. You get everything into shape, then you start experimenting and run multiple experiments. Then you're like, “Oh my God, this experiment is worse.” You undo it, and you can't get to the previous result. You're thinking, “What did I do?”

Finally, you get everything working, and then you start throwing it over the fence to production. You replicate it, those things don't work, and sometimes you don't notice that you forgot some feature naming, so the features don't match.

But then imagine you did everything, and then 6 months later you have to repeat it because now there's more data or you wanted to do another pass, and you're like, “What did I do?” Or this script crashes now, or the path has changed. Then you're spending another month just doing digital archaeology on your own history, right? Now multiply that by many, many teams.

Now imagine you got an intern that you want to ramp up. Now you have to show that intern, “Look, here's the folder, there's the script. Ask your Claude agent to figure it out.” And then the Claude agent does something, and you're like, “Ah, yeah, right, right. It was the wrong folder. I forgot to tell you. I actually have this other thing I forgot myself.”

And that's the daily life we all know if you're a data scientist, machine learning practitioner, or even any data-managing person.

Shawn Wang

Yeah. I used to do this on the quant finance side, in my hedge fund. We did this before Airflow, and then, obviously, Airflow came along, and more recently Dagster. I would say Dagster is, in my mind, what I would use for that shape of problem, where you had to materialize assets and create a pipeline.

Tobi Lütke

And that's a very good segue, because Airflow is great, but Airflow is more about having something that you want to repeatedly run in production on a schedule. It's less about you as a team developing things and being able to share, and grabbing the standard pipeline and saying, “Hey, I want to change this tiny little component in the huge sea of data processing. I want to run 10 experiments on this, and I want to do hyperparameter optimization.” All that is very hard to do with Airflow, and it's very easy to do with Tangle.

Tangle is more about a group of people running experiments—possibly agents nowadays—running experiments cheaply, collaborating, and sharing results. You don't need to understand it fully: you grab and clone somebody else's experiment or pipeline, run it, change a small piece, run it, get it to production state, and then ship it in 1 click. You don't have to port it into any other system to run in production; you can just run the same experiment. It's fully production-ready.

And it has lots of—again, as I said, it's a third-generation system. The original one was, I would claim, Ether, and then, at least in my career, Ether was the first that pioneered this type of approach. Then there was Nirvana at Yandex, which did kind of a second take on this. Now this one aggregates the learnings from all of those and Airflow as well to get to the state where, when you try it, it feels kind of magical.

Everything is now based on content hashes, so even if the version changed but the output didn't change, nothing is rerun. It's very efficient. If multiple people start experiments that need the same sort of data preprocessing, it's not repeated multiple times; it's automatically done only once. If you start 10 experiments that all require some data preparation first, as the first step, you don't have to coordinate for that. You don't have to know that other people are starting it.

You now have very easy composability. Any language you want to use is supported, and it's very visual, so you can see immediately, edit it easily, and assemble small things with just mouse clicks if you want to, and share and clone. Everybody knows it's also fully static in the sense that if we rerun it a second time, it will have exactly the same results. You will never have to do digital archaeology. Full versioning and everything is also there.

Shawn Wang

It's open source. Go to the GitHub repo and check it out, and there's also a really good blog post about it. I think all of this is really appealing. The thing that sells me the most about it is that development-to-production transition, which I think a lot of people haven't really solved, strictly. We develop really, really well in Python notebooks, but that's obviously not a production-ready process. Any way in which that is solved is very appealing.

Then the other thing that you mentioned, which also raised my eyebrows, was content-based caching, which you mentioned is very much an efficiency measure about recalculation only on content addressing. That makes sense. It surprised me that the savings could be this much, but maybe I just haven't worked at your scale, where there's so much duplication that people just rerun because they change a single ID upstream.

Tobi Lütke

It does, yeah. But it's not only that you rerun. The main savings are coming from the fact that you ran it, got your job done, and moved on. Then—

Shawn Wang

Yeah.

Tobi Lütke

—somebody else in some department you don't know existed runs the same task, but on a newer version. Right now, in most organizations, you can't even find out about it, so you can't even measure that you're spending that time twice, right? Here—

Shawn Wang

Yeah.

—if everybody's on Tangle, that's detected automatically, and it's detected that the output is the same. For that person, all it looks like is that the experiment just suddenly moved, jumped forward, right?

Shawn Wang

Yeah. This is one of those things where it's designed to be a platform from the beginning, rather than an individual developer's tool from the beginning, right? Everything's going to stream down from there. That is the Tangle orchestrator, and it manages jobs. We've seen a few versions of this, and this is obviously the unique approach that you guys have figured out. And then there's Tangent.

5. Tangent Automates Research Loops

Yeah. Tangent is basically an automatic autoresearch loop that can help and kind of do your work for you. Effectively, Andrej Karpathy recently popularized it with autoresearch.

Shawn Wang

Yes.

Boris Power

Remember? He said he was speedrunning this. You know the story. Here, we're basically bringing the same capability into Tangle so that Tangent can analyze it. It's just an agent that can run multiple experiments, figure out what can be changed, and keep rerunning it, keep modifying it until it maximizes some goal, some loss function, whatever you need to achieve.

In general, I would say if you're not using an autoresearch-like approach in whatever you do—literally whatever you do—then you're missing out. We saw at Shopify that, taking off like wildfire, anything where you can put measurements can be done dramatically better.

Our speed of HTML templatization, completely new UX templatization, and reducing latency for Liquid themes. With our search, we recently moved from—it’s hard to even quote—from 800 QPS to 4,200 QPS with the same quality, just by pure optimizations in an autoresearch loop that kept running and changing code in our index server on the same number of machines, just increasing the throughput.

We managed to improve the quality of gisting and the machine-learning process. Gisting is the prompt-compression technique that allows for lower latency and actually slightly higher quality.

So literally, whatever the walk of life, and it doesn't have to be AI-related, we had a reduction in storage because the agents would go and find datasets that clearly are derivative, and then you don't need to store things twice. We found, somewhat embarrassingly, that one of the largest tables was hashing random IDs into another random ID, and we literally—

Shawn Wang

Oof.

—did only 1. So it was translating, yeah, 2 random IDs—

Shawn Wang

So—

Boris Power

—hashed into

Shawn Wang

It has access to the code as well, so you can check what the hell it's doing?

Boris Power

So there could be 2 levels. At the superficial level, it could just use existing components and reshuffle them. You can grab XGBoost, a PyTorch module, and other tools and combine them.

At a deeper level, since Tangle is CLI-based underneath, every component is really a wrapped CLI call and a YAML file, and it can analyze code, create new components, and keep iterating as well. So you can both have quick modifications of existing pipelines with components that are already there, prebaked, or you can create new components and keep iterating on those.

Autoresearch is, again, probably the thing I was most excited about in the last 2 months. We see it taking off like wildfire. Every day—well, every day, every minute—I would have somebody Slack-message me saying, “Oh, look how much better I made it,” and it's all through autoresearch.

Shawn Wang

Is this democratized in some way, in the sense that is it your ML engineers and researchers doing this, or do your regular PMs and software engineers also have the ability to use Tangent?

This is an awesome question. Tangle in general and Tangent in particular are extremely democratizing.

Shawn Wang

Yeah.

Boris Power

Exactly. Initially, they were used by ML and AI engineers, but then, literally, as you said, PMs became the highest users. Right now, one of our PMs at work is the highest user. He was number 1 by usage of this because they’re energetic and knowledgeable, and now it unlocks a lot of capability where you don’t have to change code manually.

Shawn Wang

Right, because I don’t need the details.

Boris Power

Exactly. It’s basically Claude Code for your AI development situation, right? Now you don’t have to know exactly how algorithms work. You can just bring your domain knowledge, expertise, and product knowledge and iterate within Tangent until you’ve gotten the results that you need.

Shawn Wang

In my previous roles, every time someone has pitched AutoML, I’ve always been like, “This is not gonna work. It’s always gonna be a flop.” Somehow, it’s working now. Presumably, the answer is that now we have LLMs and it’s good enough, right? It’s an emergent property that we can do autoresearch.

But it doesn’t feel that satisfying. How come we didn’t do this before? We just did parameter search, and I don’t know, maybe that’s it.

Boris Power

Yeah. Bayesian optimization and hyperparameter optimization were used very actively, and they’re also built into Tangle. But I know Patrice Simard very well, and he was such a proponent of AutoML. He literally spent careers trying to democratize it.

Without LLMs, it just turned out to be very hard. You would have flexibility within a certain narrow domain, but it was hard to scale wider. Now, with LLMs, suddenly it’s like a magic wand.

Shawn Wang

Yeah. I think it’s multiple things, right? I’m just going to bring up the chart again. LLMs can do the monitoring very well. That is potentially unbounded and super unstructured. They can do the analysis very well.

Basically, it is much more intelligence poured into every single step. There’s maybe nothing structurally changed about AutoML, but this is just more intelligent and more unstructured.

Boris Power

Exactly.

Shawn Wang

Any flaws that you’ve run into? Everyone is drinking the Kool-Aid: “Oh my God, time savings, performance improvements.” What issues have you come up with?

Boris Power

This is really cool. It’s not a solution to all the world’s problems, for sure. This is where we get into a bit of subjective territory. I can only share what I’ve seen so far, and I’m sure the situation is changing. Maybe after I say it, many people will reach out and say, “Hey, what about this? You don’t know that.” Then we’ll probably be right.

What I’ve seen is that autoresearch is very good at doing obvious things that you don’t have the bandwidth to do, that you didn’t notice, or that you may not be aware of, like some standard practices. It is not good at doing something completely out of distribution, something that you have to think about for multiple days and do something completely novel.

I set up an experiment once on my hobby project, and I let it run for several weeks. It was full-production-scale, so it had slow runs, and it performed over 400 experiments in the end. Only 1 was successful. I was like, “Okay, that’s good,” but—

Shawn Wang

But it saved time.

Boris Power

Yeah, it saved time. That was the thing. If I were doing 400 experiments myself, my batting average, as I said, would have been much higher, I’m sure. But, first of all, it would take me 3 years to do 400 experiments.

I didn’t have to do them. The machines did them; the price of electricity did it. And I got 1 improvement that, honestly, when I was starting that experiment, my thinking was to go and show that, “Hey, Andrej, maybe you just don’t know how to optimize.”

I was super smart because my problem had been optimized for many years, and it was fully improved. I didn’t expect autoresearch to find anything at all, yet it did. So instead of making fun of Andrej, I ended up being a big, big supporter. Yeah, that’s exactly the tweet. Yes.

Shawn Wang

You and Tobi really go back and forth online a lot, which is really funny. Think of it as an eval for the optimalness of the code it’s running on. It almost reminds me of a Kolmogorov complexity thing, but I guess there’s some optimal thing that you’re trying to reduce down to.

You should congratulate yourself that you had 99% optimality.

Boris Power

Exactly. Yeah, I think Andrej really deserves a lot of credit for popularizing this approach. This is incredibly powerful and cool, and even him just mentioning it led to a lot of gains in a lot of places in the industry, so we should be thankful.

Shawn Wang

I don’t know what it is. It’s a simple, self-contained project that people can take and apply to other things, which is one thing, but also just the name. Somehow, no one managed to call their thing autoresearch. Naming things is very important.

I think that is mostly our coverage of Tangle and Tangent. Obviously, there’s a lot of ML infrastructure at Shopify that people can dive into. We’re about to go into SimGym, but before I do that, any other broader comments around this whole effort? Where is it leading to?

Boris Power

As a segue to SimGym, all those things start composing strongly. You can see a huge unlock when you look at each one of the tools and see, “Oh, they’re extremely useful.” Tangle is useful by itself. Autoresearch is useful by itself. SimGym is useful by itself.

If you combine all 3, you create a synergistic effect. I think that’s why we wanted to cover them today, because this is something that, if you go back even 5 years ago, would have been unthinkable. Replicating that would have been either incredibly costly or impossible, right? Probably thousands of people would be required.

Shawn Wang

Well, we have serverless intelligence, right? So yes, you do have thousands of intelligences, not just humans. That’s close enough, right? Even if they’re not AGI, they’re close enough to do the task that you need them to do.

There’s plenty for a lot of routine work and knowledge work. Okay, let’s get into SimGym. This is one of those things I was surprised to see, actually. It’s apparently one of your most popular launches, and I think something that—Sim AI, I think Joon Sung Park, who did the Smallville thing—there’s a very small cottage industry of people trying to do the simulate-customer thing.

I think a lot of people don’t super-trust this yet because they’re like, “Well, obviously they would just do what you prompt them to do, right?” But maybe just tell us about the inspiration or origin story.

6. SimGym Models Customer Journeys

That’s exactly the thing I wanted to cover, because if you don’t have the historical data, all you can do is prompt agents in a vacuum, and they will do exactly what you prompt them to do.

In fact, when I first proposed it—and this is a bit of my brainchild initially, if I can boast—even Tobi said, “But wouldn’t they just repeat what you tell them?” And I’m like, “Yes, except Shopify has decades of history of how people made changes and what it resulted in, in terms of sales.”

So now what we can do is—we have this noisy data. Usually, with websites, things are never in isolation. It’s almost never an A/B experiment. It’s always an A/A experiment, which has 2 meanings, but basically, at different times, you run 2 different things.

If you aggregate everything together and apply a denoising and collaborative-filtering-like approach, you can extract a very clear signal. Then you can optimize your agents. And that’s why it took so long: it took almost a year of that optimization, of just us sitting and fiddling.

We had this internal goal of hitting a 0.7 correlation with add-to-cart events, for example.

If we run a real A/B test experiment, it should replicate the same sort of success that humans had, or lack thereof. It took forever, and I don't think that's easily replicable because who else would have that data? You have to have decades' worth of historical data. Now, the other thing you need is infrastructure and scale, right?

Again, what we found is that to get statistically significant results, you need to run a lot of simulations and a lot of agents, and those are expensive things. You're making actions in the browser because you want real friction. You want to be able to get an image of what humans will see, because you want to detect effects like, "Hey, if I make my images larger, will I have more sales or fewer sales?"

Usually, people's intuition here is that if I increase my images, I will have more sales because they look nicer. Designers all like sparse layouts and big images, right? Usually your sales tank, right? But from HTML, all the characters look the same. Only the size tag looks different, right? So it's very hard. You have to take visual information, run this in a simulated browser environment on a big farm, and, of course, have a very expensive, good multimodal model. All of this is what's taken so long.

To share my personal failure a little bit there, Shawn Wang, we always had this bias toward a large-company bias. Whenever we do something, we're like, "Hey, we'll run an experiment, right? We'll make a change, run an experiment, and see which one's better." If it's worse—and most of them are worse—you discard it and keep iterating, hill-climbing.

You're like, "Small merchants can't get statistically significant results. They can't really run experiments simply because in a week there wouldn't be enough data for them." We thought from this perspective. What we didn't realize is that most people don't have A and B. They just have one thing, and they need suggestions of what A and B should be.

So we first built this: we would run simulations on two separate themes and say, "Hey, which one is better?" We then morphed it into what we very recently released. When you have just your site, your theme, we run it and say, "Hey, here are the predicted values of conversions, and here's how we think you should modify it to increase your conversions."

Circling back to what you started with, the proof is in the pudding. If we're not correlating with reality, people will not be using it. Thankfully, we see literally more users every day than the previous day. So right now—

Shawn Wang

It's working.

Boris Power

Yeah. Right now, my problem is how to pay for it all. Our major thing is how to optimize the LLMs, do distillation, and run the headless browsers cheaper so that we can accommodate the increase in traffic.

Shawn Wang

Yeah. I understand that you published a lot of technical detail at GTC, so I was just going to bring it up a little bit. I think this was in conjunction with some kind of GTC presentation or something like that, right?

Boris Power

Well, we did it in several places, but we had the engineering blog as well.

Shawn Wang

Yeah. So you're running GPT-OSS.

This is an older version. Now we run a multimodal model, but we still run GPT-OSS as well.

Shawn Wang

And then you have the VMs, and you also have browser-based. I really liked this one where you said, "It violates almost every assumption that standard LLM serving is designed for." You had orders-of-magnitude differences between everything.

Exactly. It was a bit of a challenge to implement because it violates all the assumptions. For example, multi-instance GPUs, or MIGs, don't work as well. But we needed to get MIG to work because otherwise it's way too expensive.

So we had to deal with lots of infrastructure and work with Fireworks and CentML to help with optimizations for the browser-based setup, as you mentioned. It takes a village.

Shawn Wang

Okay. So there's been a lot of experimentation in the infrastructure so far, and you've published more or less what you have here. I'm less familiar with CentML. I don't do that much work in this part of the stack. Why was it the preferred inference platform?

There are really probably 3 top companies. There used to be 3 top companies, at least that I was aware of, that did LLM optimization: Together, Fireworks, and CentML, not necessarily in that order. CentML recently got acquired by NVIDIA.

What they did is, if you have a model and you want to optimize it to a specific profile of usage, they would do it. We worked with those companies. This was work particularly with CentML and NVIDIA to get the best possible results out of it.

Sometimes you have to retune depending on what you want. Sometimes you want maximum throughput, sometimes you want minimal latency, and sometimes you want the cheapest option, or some combination. These are people who would come and help you.

Shawn Wang

I see. I'm familiar with these people for the LLM, autoregressive stack. But the other interesting category of these optimizers is also the diffusion people, where it's companies like fal and Pruna. That has come up a lot recently as well, which I think is really underappreciated, at least by me, because I thought all the workload would be LLMs, but actually there's a lot of diffusion as well.

Exactly.

Shawn Wang

There's a lot here, so it's hard to cover. But I do think people underappreciate the importance of customer simulation, basically. This is something that I'm candidly still coming to terms with. Your team also prepared this really nice diagram. I assume this is AI-generated.

Yeah. It looks—

Shawn Wang

Maybe it's not.

Boris Power

Yeah, it looks Gemini-ish. But honestly, I don't know how they generated it. It looks like Google.

But the interesting part, Shawn Wang, that we haven't covered, and that I wanted to mention, is that if your store had previous customers, rather than being a new store or a new merchant just launching things, it helps tremendously with correlation in the forecast.

We take your previous customers' behavior and create agents that replicate those specific distributions of customers that you get. Then we apply those to your changes, and that raises the raw correlation with add-to-cart events, conversion, or whatever it may be, quite dramatically. Replicating humans in general seems like an interesting, cool challenge.

Shawn Wang

As a shareholder, I think if people are Shopify shareholders, they should really deeply understand this, because this is basically the moat. The more you use Shopify, the more the images will automatically improve, right? You're doing the job for them.

Yeah. That's what we started with. Otherwise, if you're just a startup, I wouldn't do it if it were my startup, because without the data, as you said, that's exactly the case: whatever you say in the prompt, that's what the agents will be doing.

Shawn Wang

The statistician in me wants to really satisfy the statistical intuition, I guess. To me, the word that comes to mind is ergodicity. Let's say a customer takes this path, a customer takes this path, and a customer takes this path. In my mind, the way I explain it is: here's the 95th percentile, here's the 5th percentile, and here's the median, right?

But to me, what SimGym is potentially doing is that it can sort of model the in-between journeys as well, which may be dependent on the previous states. This may be a very RL-type conclusion, where basically, if you only did naive A/B testing, you only have statistics at a certain point, and you only judge based on the overall summary statistics. But here you can actually model trajectories. Does that make sense?

That makes total sense, because it makes even more sense than maybe you realize. Internally, we have this system—we talked about it briefly once at NeurIPS. We have a huge HSTU-based system that models all the companies and their possible paths.

Shawn Wang

Yeah.

Boris Power

What you're showing is that at any point in time, you can either model the user's behavior, or you can also think about the whole merchant as a company, as the entity that acts in the world. You can model that as well, and then you can do counterfactuals. In your graph—in your blue graph—if you're…

Imagine in the center there, somewhere in the middle, you would have an intervention. I give that person a coupon or send a personal thank-you card or give a discount somewhere, and then you can do forward rollouts from those counterfactuals: What would have happened with that intervention or without the intervention? You can even change when that intervention can happen in time, right? Somewhere in this journey.

So we do this at Shopify scale for our merchants, and then if we notice something they can fix, like there’s a strong counterfactual, we have Shopify pals. They basically get a notification like, “Hey, we think something is wrong with your Canadian sales. It looks like it’s misconfigured. Here’s what you need to do.” Or, “We think you have to set up this campaign with these parameters.” And we do that at the buyer level to literally offer discounts or cashback or things to buyers.

This is—I’m getting very excited. This is my sort of area of interest and hobby. But being able to model something complex as human beings or companies and model counterfactuals on it, where you can have interventions in the future and optimize when to make an intervention and what kind of intervention to make, is such an unlock that previously was completely impossible. It was always dreamed of, but never—how would you even simulate it without LLMs or HSTUs? Very, very exciting times.

Shawn Wang

I just wanted to maybe illustrate this. I’m not the best illustrator, but I am a conceptual statistics guy, and you cannot just do this. This is a dimensionality that A/B testing doesn’t cover, right? It doesn’t have the change over time, the stochastic nature, and it doesn’t have the contextual information—here’s all the context up to this point.

Okay, cool. SimGym, you’re going to burn a lot of tokens on this thing. But you’re one of the only scaled platforms in the world that can do this across a huge variety of workloads, right? I’m even curious, on a sort of human-research level, whether retail behaves differently from clothing sales. Does that behave differently from electronics sales? The Kardashian shoppers—do they differ from people who buy cars and whatever?

Well, very different, with different sensitivities, different modes of shopping, and different levels of what’s important. Totally, you can do aggregations at a store level. You can do aggregations at different category levels.

I couldn’t believe it, but recently, when we were looking at this, we had to bring back CRPs—the Chinese restaurant process. It’s a way of aggregating and naturally growing clusters. Specifically, to answer the question you were just posing about how buyers behave differently across categories, I was like, “I haven’t seen CRP since 2001.”

Shawn Wang

What?

Farhan Thawar

There was a very popular kind of theory in NeurIPS and ML circles in the early 2000s. It was kind of nice, and now it has practical applications.

Shawn Wang

Yeah.

Farhan Thawar

We were resurrecting it.

Shawn Wang

Yeah, amazing. Anyone who knows what CRPs are and has always wanted to use them at work should definitely join Shopify.

I can see how this is a fun job for you, where you get to apply all these things.

Farhan Thawar

Yeah.

Shawn Wang

Super cool. Okay, we have a lot, and I’m being mindful of the time. I did want to cover some other things. I’ll give you a choice: UCP or Liquid?

Liquid. I think UCP is very important for us, and we have structured discussions about it. You can read about them; we have blog posts, and we have a big release this week, in fact, with our catalog.

Shawn Wang

Oh, okay. I mean, we can discuss the release briefly, because when we release this, it’ll be after it’s already announced, so whatever. There’s a catalog that you guys are doing?

7. Shopify Opens Its Product Catalog

Farhan Thawar

Yeah. We’re bringing in the capabilities of the whole Shopify catalog. Basically, now you can search for products. You can do lookups by specific ID. You can do bulk lookups when you need to bring multiple products. You don’t need to know in advance what you’re trying to show, sell, or check out. Now you can have this decided at runtime.

This is a big area of investment for us, for both non-personalized and personalized searches, trying to provide basically a window into the whole universe of products that are being sold everywhere in the world. Shopify is not exactly, but almost, a superset of anything being sold. Now we’re bringing it into UCP, and identity linking is another big thing for us, so that you can use Google or whatever identity you have there, minimizing friction.

Shawn Wang

Yeah.

Farhan Thawar

So, yeah, big release for us. But Liquid AI, of course, we never talk about, and the problem might be more aligned with what we discussed previously on this chat.

Shawn Wang

Sure. The main thing everyone understands about Liquid is that it’s inspired by worms, and I still don’t know why. I’m curious about your explanation. I think you can make things very approachable, and I’m also curious about the potential in terms of the efficiency you get out of Liquid.

8. Liquid Challenges Transformer Dominance

We’re all familiar with transformer architectures, and for the longest time there was a competing architecture called state-space models, or SSMs. Chris Ré is one of the pioneers, along with lots of startups trying to make those a reality.

They have significant benefits, being much faster and having a lower footprint, and they’re not quadratic in length. They’re linear in your context length. But with state-space models, they never quite made it. They’re used in certain niches where they thrive; their hybrid architectures are useful, but they never quite made it.

Liquid neural networks are—you can think of them as a next step, sort of a state-space model squared. It’s a non-transformer architecture that’s more complicated than state space and really difficult to code, if I’m being honest. But it’s very efficient. It’s subquadratic in the length of your context. It’s a very compact way to represent things, and that’s Liquid AI, the company. Their goal is to productize it.

Very often, you have this need when you need to have long context and a small model, and you want to have low latency. In general, it’s basically on par with transformers, and if you do hybrids with transformers, it’s even better.

That’s why at Shopify, when we tried multiple models and multiple companies—we constantly try multiple models—we found that for small models, particularly for low-latency applications, when you have low latency and/or need longer context lengths, Liquid was the best. We still use the whole zoo and always test and use everything, every open-source model, and sometimes even every private model. But Liquid has been taking quite a bit of internal Shopify share.

The reason I’m excited is that it’s the only non-transformer architecture I’ve found to be genuinely competitive. We use it for search, for long context, distilling, and other things. This is the overview. I don’t know how approachable the chart is. Sorry, maybe it’s still too obtuse.

Shawn Wang

Hmm. I think they haven’t been that open about their implementation details. I would say that if there’s a lot of technical detail published, I haven’t read a formal paper on the implementation details. But I did get the relationship between the SSMs and the others. This is one of the charts showing the relationship between full attention and something more like an RNN, in terms of efficiency. The other chart was this old one comparing it with some of the other models.

It doesn’t exactly have the correct y-axis, but it’s close enough that you can see it’s basically a step-change difference in efficiency. I think the surprise to me was that you guys are actively using it already internally at Shopify, and I’m curious what constraints you’re optimizing for. When you say smaller, is it the 1B size? What kind of latency constraint are you optimizing for? What kind of context-length considerations are you thinking about? For example, in audio-type use cases, the SSMs effectively have unbounded context length because they just have to operate on the sliding window of the most recent stuff.

What do you see as the potential here?

Farhan Thawar

The SSMs are effectively infinite context length because the state embeds all the previous information needed, or that's the assumption. The problem with them is that expressiveness is not there.

Liquids are effectively souped-up SSMs that are much more expressive, but more complicated to code. There is a paper on it—you can see it. A differential equation is rolled out and then computed as a convolution. It's a bit involved.

The thing where we use it is specifically where we need super-low latency. It was a very fun project with CentML and Liquid AI themselves. We run it at 30 milliseconds—a tiny model, around 300 million parameters—but we run it in 30 milliseconds end to end for search. When you type a query, we produce all the possible things you could mean by that query: not only synonyms, but a full query understanding, the whole tree of what you might need, including personalization, because you might have made previous queries. We lower it all down into the search server, so the requirements on latency are obviously very strict.

We are able to run it under 30 milliseconds because Qwen doesn't run on this. Even with Liquid, we had to work a lot with NVIDIA because almost everything in the current stack is not designed in CUDA for low latency. Small things that don't matter with large models start mattering a lot, and we had to optimize it.

There is a different end of the spectrum where this is maximum throughput for things like offline categorization. When a new product appears, we need to analyze it, assign where it is in the taxonomy, extract and normalize attributes, and create clusters—"It's the same thing as that other merchant is selling," for example. That is an almost unbounded amount of energy you need to spend on it because it's a quadratic problem, and we have billions and billions of products.

You don't care as much about latency; it's kind of an overnight batch job. You want maximum throughput. In those cases, you also sometimes need long context, as with Sidekick Pulse. We are talking about models in the 7- or 8-billion-parameter range, where we would take a large model—something huge, the largest we can find—and distill it into Liquid for a specific task, such as catalog formulation or Pulse. Then we run it at a very large scale in batch jobs.

It beats Qwen very often. Kimi is more on the reasoning side, so Qwen would be their major alternative. That's when we use it. It's not a panacea, and I wouldn't say that it's a frontier model in the sense that it's suddenly going to compete with GPT-5.4. But it is a phenomenal target for distillation, which is becoming more and more important right now with the explosion of token usage.

Shawn Wang

Is that a now-only thing, or do you think if you gave Liquid $100 billion, they would scale? Is it just more scale, or what is limiting it? What prevents it from running into the same issues that SSMs had?

Their scale is already much larger than the largest SSM I'm aware of.

So, SSMs were just not expressive enough. Again, I'm sure I'll get a lot of pushback, probably privately, but in my opinion, SSMs are not expressive enough, and Liquid models are. I think especially in their hybrid form, combined with the transformer, like in the Mamba fashion, they are probably the best architecture I'm aware of, period.

Of course, Liquid AI is not at the scale of Anthropic, Google, or OpenAI in terms of compute. If they had a similar level of compute, I think they would be very competitive and maybe even beat the largest models, at least from what I've seen. They don't have that level of investment, but they still have decent investment, and for this scenario of smaller models and distilling into them, they are second to none very often.

We are very omnivorous and operate purely on merit. The moment something else becomes competitive, we will switch to it, and we constantly test. But so far, if you see the progression—if I draw a graph of our workloads on Liquid versus our workloads on Qwen, which is another awesome model and probably another standard within Shopify—I would say Liquid has definitely been taking share.

Shawn Wang

I think that's very promising and probably the best explanation I've heard directly from someone involved in Liquid. I do have Maxime Labonne coming to my conference in London this week, so we'll hear more from him.

There was this Liquid investor day about a year or a year and a half ago, and I think there just wasn't that much technical detail speaking to my crowd of potential customers and users. Maybe we still need to wait for more results to come out before this, but I think it would be news to a lot of people that you guys are already actively using it for high-frequency use cases.

I also wanted to highlight Sidekick Pulse, which we didn't cover and probably don't have time to cover, but it's something that you also launched recently. Basically, it's recommendation systems, but it's also something that I've been covering a lot from the YouTube side. Even xAI's recommendation systems have been LLM-based, right? I think you are also effectively using Liquid models for this, but they are just throwing transformers at the problem.

Maybe this is the sort of hybrid-architecture shift that will happen in order to accommodate the kind of long context and high efficiency that you need. I don't really have a strong opinion there, apart from highlighting to anyone that the LLM-based recommendation systems community is doing very interesting work.

The thing to get you excited is that it's not just LLMs looking at things. It's also an HSTU model doing that counterfactual analysis, where we model the whole enterprise as an entity and its actions, and then see what will happen.

Shawn Wang

Overall, I think this presents an enormous opportunity. There wasn't that deep of an AI story to Shopify when it started. It was just a WordPress plugin, right? But now you are the storefronts and e-commerce guardians for so many people, and you're really applying all the AI methods and state-of-the-art techniques.

I think our conversation today has really opened my eyes to a lot, so thank you for doing this. This is an amazing overview of what you're doing.

Thank you for saying that, Shawn, and thank you for having me. Of course, it's always a pleasure to talk to people who are deeply technical and know what they're talking about.

Shawn Wang

Very few people are as technical as you, but at least I can somewhat vaguely follow along.

There is a hiring call. Are there any particular roles that you're looking for where you're thinking, "If you know how to solve this problem, reach out"?

The things I would definitely call out are that, if you're an ML person or a data science person, we have a huge need for more people munging data, so to speak.

Or, surprisingly, if you're a distributed database person, we think there is a way to use LLMs to reimagine how we do distributed databases, and we're working a lot with Yugabyte there. If you're interested in those areas, Shopify might be the best place in the world for you. But it's a pretty good place for other disciplines as well.

Shawn Wang

I think that was all the questions I had. I have one bonus thing, if you want to indulge in some Bing history. What are your takeaways, or do you have any fun anecdotes about Sydney?

Any fun anecdotes about Sydney? Well—

Shawn Wang

It was very interesting. I think it woke people up to this personality that emerged.

The most interesting anecdote is that Sydney was first shipped in India for a while, and it was not noticed for a long time.

And the first implementation of Sydney didn't even have an OpenAI model under it. It was tuning Megatron-Turing NLG, the Microsoft and NVIDIA collaboration model. Yeah, exactly. That's the one.

People thought it was a prank because not many people were familiar with LLMs at that point and thought, “That cannot be automatic. You must have people thinking.” Then they were complaining, “Oh, my—this chatbot is gaslighting me.”

What almost everybody doesn't fully realize is that it wasn't by accident that Sydney was Sydney. We spent a lot of effort on personality shaping. It was a bit of my Yandex legacy. Previously, we did Yandex's Alice, a digital assistant.

Shawn Wang

Chatbot, yeah.

Yeah. We learned the importance of personality shaping, and so here we did a lot of personality shaping. It was not fully an emergent scenario. It was also a little bit edgy.

What we learned in those experiments is that you want to be polite, but you want to be a little bit on edge, and that draws people in. Ever since those days, I haven't seen anybody trying exactly that mode. I think we will see more of this at some point.

Yeah, lots of good memories. And by the way, the very first Sydney dev lead, Andrew McNamara, is working at Shopify and is the head of Sidekick. Lots of these are actually in his purview.

Shawn Wang

Oh, okay. That's another fun fact. You're assembling the team again. Yeah, it's cool. I think a lot of people woke up to the idea of AI personality for the first time there.

Now, with maybe OpenClaw explicitly prompting a fun personality, I think that is a real selling point for people, right? And then I guess maybe the only other time that it's really emerged into public consciousness is with Claude. But hopefully someday we'll get Shopify Sydney.

Well, we have Sidekick. It's a slightly different thing.

Shawn Wang

Yeah. Sidekick was your original big launch for AI stuff. Amazing. Thank you so much. You guys do amazing work.

Honestly, if I was a Shopify customer or a Shopify investor, hearing all the work that you guys are doing on the technical side makes me feel more confident in, “Okay, just choose Shopify,” right? You're never going to do this in-house, which is obviously what you want.

That's what an ideal platform is: you're doing all the things that no individual could do at their scale, but you can at your scale. Very exciting problems.

Exactly. And creating network effects, it's hard to disagree. If you're not using Shopify, you should.

Shawn Wang

Yeah, amazing. Okay, well, that's it. Thank you so much.