家用人形机器人真的要来了？| Nikolaus, Rerun

We had this idea that visualization would be hard to monetize, particularly for physical AI applications. You need to visualize every single step everywhere you might interact with data. So we've actually redesigned the data model probably 4 times at this point.

1. Exploring Rerun.ai and Its Innovations

You're listening to Gradient Descent, a show about making machine learning work in the real world. And I'm your host, Lucas Bewald. All right, this is a conversation with Nico West, who is the founder of Rerun.AI, which is a company that I've admired for a while. They're in a similar domain to weights and biases. Um, robotics and embodied AI and augmented reality and they have a a very cool logging product that operates at super high scale and then a database or kind of system of record for bigger customers to track everything they do. They've made a lot of interesting decisions, some same, some different than than Weights and Biases. So, it's fun to get into the weeds about um how their product works and how they think about it. It's also just interesting to talk about robotics and the state-of-the-art because it might be the coolest application of AI right now. Hope you enjoy it.

Oh, wow. You know, AI and robotics seem like such hot topics. What are you seeing that people might not realize is working now that didn't work a year ago?

We're just seeing incredible progress in the ability to do quite advanced manipulation. This has been a really difficult problem in robotics for a long time. Another one is reinforcement learning. Classically, that's been for walking and motion, and what's starting to work a lot now is combining some of that reinforcement learning with imitation learning.

Are there benchmarks that should be created?

There aren't any great benchmarks. Often, the way to solve real problems is to co-train and co-design for your hardware. It's like you build and design these things together, and then it becomes quite hard to benchmark in a good way.

Maybe I'll start with a little bit of why I wanted to talk to you on this podcast. You're the founder of Rerun, which is a company that does logging for robotics and also seems to function as a system of record for robotics companies. It's very similar in some ways to what we do at Weights & Biases for AI teams.

Looking at your website, it seemed like you emphasized a lot of the same things as us, like a simple API and high-speed logging. It also seems like you have similar customer love to Weights & Biases, which I really respect and admire. Maybe not a lot of people know this, but the original use cases of Weights & Biases were pretty heavily in robotics. When we first started working with OpenAI, their main focus was robotics.

This was before GPT, and you can actually find a very early interview with Wojciech, one of the founders of OpenAI, talking about robotics projects there. It must have been like 5 years ago. Anyway, you started your company a few years ahead of us in a different domain and made a lot of different decisions. I was really excited to talk to you about your creative and product process. I also think everybody loves robotics and its applications, and maybe now we call it embodied AI, so we can get into that too.

We get compared to Weights & Biases every now and then, and that was one of the early signs that we were really happy about our API, when people were saying, “Yeah, it's like Weights & Biases for robotics or spatial computing and that kind of field.” So, yeah, where do we want to dig in first, basically?

Maybe we should start with the range of use cases that you're typically used for and how Rerun works there.

Got it. So maybe just to set the scene: Rerun has an open-source project that is very popular, and that project is an SDK for logging, modeling, querying, and visualizing really multimodal data, particularly multimodal data that changes over time.

I'm picturing sensors from a LiDAR or—

Yeah, like 3D sensors set up by multiple cameras moving around. Maybe you have regular RGB, motion sensors, and whatever other normal signals and tensors you can think of. Our first early beachhead was in spatial computing or augmented reality, so companies building headsets and that kind of stuff.

And so what are they logging there?

They're logging per-sensor images. You're logging camera calibrations, where a camera is in space, as well as normal text logs and time series. That could mean whatever—CPU time, some confidence of a neural network, or anything like that. So, just both the input sensors that you have and any sort of internal computation that you might do. You have the output of a neural net, so a little bounding box that you detect, or anything like that.

And where has it gone from there? It started out really focused on making it 100 times easier to debug these sorts of multimodal systems, like computer vision or robotic systems that do things over time. That was the focus quite heavily in the beginning, and it got a lot of its first usage in spatial computing.

Then, quite broadly, it's used in hedge funds and weird different things, but now it's very heavily used in robotics, particularly learning-first robotics. On top of that commercial, sort of open-source project, we're building a data platform. It's more of a cloud database or data lakehouse, if you want to get into that.

We're really focused on making it a lot easier to get data from a robot into the right shape, and the right subset of that into training, basically. That way, you can run the record, curate, and train loop as fast as possible, but we're not taking part in the training, which is what Weights & Biases does—the data pipelines leading up to it.

Interesting. I can see how you start in VR and AR and then move into robotics, but what is a hedge fund logging on your platform?

I think we have all this breadth of random use cases as well. I guess it ended up that we put a lot of effort into performance and ease of use.

Mhm.

It actually turns out that it's pretty difficult to get really fast, simple-to-install streaming visualization of even simple things like time series and so on. We put a huge amount of effort into building an in-memory database from scratch, basically, that runs inside this visualization app. That can speed things up, and it's highly performant and easy to use.

It's not perfectly meant for that, so I don't imagine it will take off in a big way, but we find a lot of random use cases like that as well.

Interesting. I noticed that, just like Weights & Biases, your core API call seems to be a log.

Yeah.

I think one of the things we often thought about was how opinionated to be about people's data types and how opinionated to be about displaying them. How did you think through that trade-off?

That came from—first, it's a really important trade-off, and I think we got—I don't want to say we got it right, but we got pretty good. I'm quite proud of it. It did go well. Of course, there are always things that you want to improve, but I've built similar systems many times.

The 3 of us co-founders worked at a 3D computer vision company a long time ago, and we built something similar there. We'd built several versions of this at different companies and came in with the idea that it was really important to have incredibly low friction when that's what you wanted.

So if you're debugging one of these systems, there's so much state in your head, right? It's very similar to debugging machine learning training.

Mhm.

It just needs to be dump-and-forget, with no thinking, at a high level—very easy. But with experience building these things in perhaps more rigid ways, you can always get stuck somehow.

So it's very important for us to have it be flexible, so that you can say, “Okay, I added in this output from a neural net or a 3D point cloud, or something that my neural net is estimating, but I just want to understand what the confidence level was for this one point, and I want to just hover it and look at it.”

Then you need to make that work. I can't know upfront what all the users' data models are going to be, so we need a very flexible data model. We designed a new data model from scratch, inspired by entity-component systems. This is a way of modeling data in games. A couple of the early employees had a gaming background, and we basically designed an entity-component system.

And what does that mean? How does that work?

It's basically—let's see if I can explain this very simply. Instead of objects, if you're in a normal object-oriented program, you might have an object and say, “Okay, you have an image; it has its size and the memory buffer with the things in it.” You just have a memory buffer as one thing, image size as another, and you can compose all these components. Then there are systems that know how to interpret them.

So there's some visualization system that says, “If I see an image or image buffer and a size, along with all the other things that I need, then I will just draw it.” It's a bit more flexible than saying, “We just have this fixed object that we know how to deal with.” You can easily compose the things that you might want to visualize, log, or understand more freely, basically.

What are the most common things that people are logging here? If we look at where Rerun is particularly popular right now—in heavily learning-based, AI-first robotics—what are people logging?

People are logging motion sensors, images, and video. It's really important to have flexibility and high performance in how you represent video, along with the other debugging-system things that you might have in a robot. Those would be the most important things there, and then point clouds and LiDAR-type sensors.

2. The State of Robotics and AI

Maybe we could talk about the state of the art of AI and robotics. That seems like such a hot topic. What are you seeing that people might not realize is working now that didn't work a year ago? I think even these last couple of weeks have been incredibly exciting. I don't know if you've been following all the releases coming.

Maybe not. What are they?

We're really seeing incredible progress in the ability to do quite advanced manipulation. This has been a really difficult problem in robotics for a long time. Classic robotics is built around being incredibly precise: You preprogram every movement up front, and everything has to be precise down to the millimeter. Flexible, messy manipulation—folding laundry is a classic task—has been really elusive for robotics for a long time.

Folding laundry has gone from being impossible to being boring over the last year, basically. It's really come through these end-to-end learning methods, first on manipulation tasks through imitation learning. It's a kind of robotics version of supervised learning.

Can you describe how imitation learning works for folks who might know?

Yeah. So the simplest case is you teleoperate your robot. That means just a person is remotely controlling the robot and you record everything that happens. Say you want to fold clothes: for one episode, you record data as an episode. You start by taking a piece of clothing from the hamper or whatever, put it on the table, fold it, and put it in a pile or something like that. That could be one task. You collect data like that—demonstration data—and collect all the state of the robot at that timestamp. What are all the joint angles and some perception or prop perception input that you can use? In simple terms, you train a neural net to repeat that task. But this is a real messy world, so it's never going to be an exact replica of the task. That's the kind of robustness that you try to train in. I think it was about two or three years ago that this started to really work in more serious ways, and that really started the heavy excitement in AI-first robotics that's happening right now. That's sort of one of the learning paradigms. Another one is reinforcement learning.

Mhm.

Classically, that's been used for walking, motion, and things like that. What's starting to work a lot now is combining some of that reinforcement learning with imitation learning. We're seeing some really awesome results and very robust robotic manipulation and handling coming out over the last couple of months.

Yeah. Do you think it's been algorithmic insights that have made these strategies work, or is it a bigger focus on training data, better hardware platforms that actually work better, more compute, or something else? What's driving this?

I think it's all of the above. My view tends to be that, generally, the things that matter more may be more this ecosystem or economic engine or something like that. What I see as the big arc is that the LLM hype—the ChatGPT moment—really showed the world the power of scalable machine learning.

Mhm.

That really primed everybody to look out for signs of whether scalable learning could work for robotics, because previously it had been this thing where you add more training data and it gets worse. The models just don't generalize. A couple of years ago, there was a bit of a breakthrough, and you could start seeing signs of scalable methods working for robotics.

That includes using transformers, but there were a bunch of innovations that had to be made in terms of how you model a robotics problem, which is different from a text problem. Then that gets going, more people get money, and more people get excited, so there's more hardware being built, which gets cheaper and cheaper. Everybody knows it's scalable, so that means you need a lot more training data. There's enough excitement to invest in collecting data.

My impression of robotics right now, in late 2025, is that I'm super excited about it, obviously, and I see these amazing demos all the time. But when I visit our robotics customers, the demos don't seem quite as impressive as the YouTube videos that I watch. I'm not talking about the Boston Dynamics ones, where it's obviously more traditional robotics, but even the ones where you see this really fluid motion and clearly trained behavior.

I would love to buy a robot that folds laundry, but I don't have one. What's the gap right now? How far is the gap, and how would you characterize the state of the art of robotics?

I think there are different kinds of robotics companies. A lot of the companies that are swinging for the fences directly produce these incredible demos, but they're actually quite far from a product. I also see another batch of robotics companies that have a very different approach. They're much more practical: They build scrappy robots early, ship them to customers, have a working product, and aggressively deploy it.

Those don't get as much hype because it's not as exciting to look at a kind of aesthetically unappealing warehouse robot.

3. Practical Applications and Future of Robotics

Talk about the working robots that you see in warehouses. What's happening now?

They're really using a lot of the same models, but perhaps with less focus on research. In parallel to all of this, there's also been a lot more open-source models coming out. In the same way as in LLM land, there are some great open-source models, heavily driven by Hugging Face.

I see them being scrappy and using VLA models—vision-language-action models—but perhaps being more practical about building other systems around them and using teleoperation where needed. It's a more product-oriented and scrappy approach to making things work.

Being practical, what are you actually seeing? I've talked to guests on the podcast from every part of this spectrum. We've talked to folks who have carts in factories where you can put something on them, and the cart will go to the next location. Clearly, factories are doing that a lot.

A lot of people talk about picking stuff up in factories and moving it, although that seems like something people have been dreaming about for years and years. Is picking up objects actually happening in real-world factories, and does it matter? Or is the state of the art past that in terms of what's actually deployed in production?

I would say there are pockets where it's working, but it hasn't been scaled up.

What are those pockets? That's, I guess, what I want to know. In other words, where is it actually possible to do this now?

We see companies that have deployed anywhere from tens to 100 robots in manufacturing. They use this kind of flexible, learning-based manipulation for moving things in manufacturing, for instance. It can be simple stuff like pick-and-place.

Mhm.

They have that working, but it's not at massive scale yet, basically. There are smaller companies that are growing their number of robots and making it actually work in production right now.

These are vendors—robot vendors that factories buy from?

Yeah. Exactly. I haven’t heard of the factories themselves doing that yet. It may happen, but I haven’t heard of it.

I see.

Yeah.

And then what about on the consumer side? I’ve had a couple of rich friends who have bought humanoid robots, and they seem to collect dust in the corner right now. I’ve been to a robot fighting league as well. We were there too, which we’re not sponsoring, but I’d say the fighting seemed a little unimpressive compared to the hype around the fighting.

What’s the state of the art there? If a robot can actually fold laundry, and the state of the art is past that, why isn’t there a consumer product that folds laundry?

So, I think you can think of what it would take to fold laundry in your own home.

Mhm.

There are more things to it than setting it up. If you had a very expensive robot but it required you to set everything up perfectly, maybe some people would buy that, but most people wouldn’t because of the total setup and management of it. The physical world has such a huge long tail or fat tail, or whatever you want to call it, of weird small variations that make things a lot harder. At the end of the day, it’s not just a robot that can do a task. You need to build a product, right?

I think that distance and this whole systems aspect of deploying robots is very difficult. You need to service robots. You need onboarding tutorials and all these kinds of things. I think we’re still not at the point where most advanced frontier robotics—intelligent robots—are at the full-product level yet. But we’re seeing some people starting to ship robots for the home soon, I guess. It’s exciting, but I don’t think we’ll see full autonomy in the home for a bit.

What do you think will be the first applications that people use in their house?

Probably picking things up. I mean, the first one—the one that’s used right now—is vacuum cleaning, right?

Well, I feel like vacuum cleaning has been around for a long time.

Yeah, there are some newer vendors that make that significantly better, right?

Interesting. What are you thinking of?

Well, I like Matic a lot.

I have a Matic in my house, actually.

So that’s a big step up from a Roomba, for instance, right?

Actually, it’s funny. My daughter was asking me what’s better about a Matic than a Roomba. I think we like the mapping that it does, but honestly, that’s not really an autonomy feature. What is better about a Matic than a Roomba?

The autonomy comes from the mapping, right? That allows you to handle more robust situations, like when something is in the way. It doesn’t get stuck, and it can avoid driving over your cat.

Right.

The collection of these things makes it more robust and more useful. You get into this realm in the physical world where there are just so many small things that can go wrong, and they add up. If the probability of one of them happening is high enough, you end up not using the product, basically.

Sure. Sure.

It seems kind of funny. I remember when Roomba launched; it must have been 2004 or something like that. Here we are 20 years later, and it seems like robot vacuums are still by far the main consumer use case.

Yeah, and I think that is the simplest, right? So it makes sense for that. I would guess robot vacuums that also pick up toys are probably the next step. I know some folks are working on that.

Uh-huh. In terms of really practical things that can work autonomously, I guess it’s just small steps like that that will happen first. What about fun things? I feel like suddenly we see robot dogs everywhere.

Yeah.

What else are you seeing in that domain?

I haven’t interacted that closely with the companies that are building more toys, but I’m super excited about that. I think there are so many fun things like that that can exist. There are AI-powered robot toys and so on, but that’s not a space I know really well.

I guess what else are you seeing in terms of applications?

I think very broadly, we’re seeing things in agricultural picking. Something I didn’t really realize was the breadth of how many places have this last-mile logistical pick-and-place problem. So there really is a lot in different parts of the logistics chain, as well as many kinds of small autonomy—last-mile delivery and these kinds of things—that put them in the same realm as self-driving, but sort of the same realm as robotics as well. Drones have a lot of different applications there.

Yeah. Am I maybe overly focusing on robotics? Do you see other embodied-AI or other applications outside of robotics that are meaningful to you? How does your customer base break down?

I think our customer and user base breaks down according to whether you’re trying to do something intelligent for the physical world, mostly. That splits into the big categories: AR and spatial-computing kinds of things, autonomy more broadly, and robotics.

Mhm.

But then there’s a long tail of smaller things. They can be really small things, like a med-tech application or the little AI thing in film production that makes focusing easier. There are a huge number of these small problems. There’s quite a lot in security and surveillance, like analytics for construction and that kind of stuff. Basically, for any physical-world thing that’s happening out in the world, someone’s probably building a product to either observe and analyze it or actually improve it.

Are you seeing a lot of satellite-data applications?

We don’t see a lot of that internally because they’re just product features. We haven’t built a lot of specialized satellite-data handling. We have basic mapping, but that’s a whole very deep field, so we don’t see them necessarily as huge Rerun users yet.

Mhm.

But aside from that, I see some cool stuff happening.

Cool. All right. I kind of want to go a little bit deeper into your product, if you don’t mind. I think one of the things that we really designed our product for is making sure that it doesn’t break the user’s process. Was that a big thing for you? I’m curious how you like to actually stream the information back to a central server.

4. API Design Tradeoffs and Data Model Redesigns

Yeah, that was important for us too. You should never crash the user’s process, of course. We originally designed it heavily for cases where we focused a lot on high-performance streaming. There’s the whole robustness aspect. I don’t know that I have anything particularly interesting to say other than that we spent a lot of time making sure it could never crash.

But in robotics, we didn’t design it for the case of, “Hey, there’s always a server listening.” Quite often in robotics, you can’t guarantee that, and the normal thing to do is write to disk first. So the 2 core original use cases were either writing to disk or streaming to a remote viewer or something like that—a remote server. We designed for both of those, and our on-disk format is basically the same as the streaming format.

And so, to make these things fast, we built this sort of system where you log data. Depending on the size, you can control these things, but we basically run a microbatcher.

Mhm.

You buffer up a little bit of signals in memory, and if you’re having really, really fast time series, maybe you buffer up more and then send it over the wire in chunks like that instead. Balancing that and doing it efficiently is one of the important parts of getting good performance out of these things.

I mean, one of the decisions we made—that I’m never sure was the right decision or not—was to have the logging be open source because it’s running in the customer’s or user’s environment. But then we made our visualization, the centralized visualization web server, closed source. I think you made that whole thing end-to-end open source. Is that right?

Yeah, exactly.

And how did you think about that? Is that why you had to make a different product to monetize, or did you have a more enterprise version of the visualization server? How did you think about that?

So we had this idea that visualization would be hard to monetize, actually, because of how important it is. We had the idea that, particularly for these physical-AI applications, you need to visualize every single step of everywhere you might interact with data.

You need to visualize when you're prototyping in a notebook. You need to visualize when you have maybe 1 robot or 1 device next to your desk and you're kicking it to figure out what's happening live. You want to visualize your data pipelines as you're preparing for training—basically everywhere. You want to use it for your operation. It's just too widely deployed.

And there's no good business model that lets you have visualization at all those different points that actually works. Mm-hmm.

You can make the business model around a per-seat license or something like that. That works for your expensive engineer who's looking at the data all the time, but you also want to have the same visualization for your support person who looks at it every now and then. That balance becomes very difficult.

And having it closed source, if it's so central, actually becomes very difficult to adopt for a lot of companies. So we basically decided to have the base visualization be open source, and it kind of always has to be, just so it's easy to embed. You need to be able to build your own tools around the visualization that are custom to your company. So you should be able to take the

visualizer and embed it in your application and customize it.

Mm-hmm.

5. Exploring Logging SDK and Visualization

So the open-source visualizer is sort of client-side. That's kind of how we made that split: there's a logging SDK that runs in your process and directly communicates, maybe via the file system, with a visualizer app, and there's no real serious server in the middle.

Mm-hmm.

That's kind of how we started out, and I wouldn't say the commercial product is a different product, but it's a cloud backend for it instead that supports very large amounts of data.

I see. So we might have actually ended up in the same place, although you might have more emphasis on the client-side visualization, which we've actually just implemented recently.

Yeah, it's kind of similar in that manner, but we just took a different route, perhaps.

Interesting.

Yeah.

I guess, have there been interesting API design trade-offs that you made? Are there any API decisions that you sort of regret? I remember we made a ton of decisions in the early days very quickly, with no information. It's amazing how some of the weirder ones that we made have stuck and haunted us for years.

I think there are more detailed things that—I don't know if they're worth getting into—specifically how we designed the tensor data model, things like that. But I would say we'd rather design things wrong and then redesign them. We've actually redesigned the data model probably 4 times at this point.

Oh, wow. That has been very effortful, right? It's been hard. We had to keep some backward compatibility and do all these things. It's been quite difficult, but—

But, yeah, we really cared about getting those core pieces right and have been ready to redesign them.

I mean, this may be too in the weeds, and people might just skip this episode, but what are decisions that you made in the data model that you then changed in the next version of the data model?

Well, the first one was this: we started out with a less flexible API. In the beginning, it was like, “Okay, this is getting very detailed,” right? It was like rr.log_image, and you'd give it all the fields. There was still a flexible way you could give

Right.

your own custom data into that. That was the first huge rewrite, where we formalized the system to have higher-level archetypes, which are easier-to-use wrapper objects that you can log instead. Now it's like rr.log("image"), say.

A lot of the underlying data model things we've been changing are perhaps less clear on the surface, but they tend to have to do with how you represent columns of data. It's more about being able to log more data for multiple time steps at the same time. For instance, here's a whole column of data, or here are 5 columns of data, and exposing those APIs. That's been one set.

The most difficult and impactful things have perhaps been how it's represented in memory and how the data model works in the query engine of the database. Those rewrites have been the most impactful for performance, storage, and things like that, and pretty painful to redo. I think we've done that 3 times or something.

Yeah, totally. That's funny. There are a lot of similarities between our companies. What database are you using now to store all the data?

On the open-source side, it's custom—basically everything. The whole thing that's open source is built in Rust from the ground up, and we use Apache Arrow.

Yeah, yeah, we do too. Yeah.

So that's basically built around Arrow, and everything is custom. It comes down to the special properties of what we call physical data.

Mm-hmm.

Physical data is multimodal, multirate, and often episodic.

Mm-hmm.

That changes how you need to store that data. You can't fit that kind of data in a table. Because it's multirate, you may have motion happening very fast, and you have images coming very slowly. Data on a robot is happening at different rates, so it doesn't work—you can't fit it into a normal tabular model.

You have to store data differently. It's a new file format for doing that efficiently. You could think of it as a sparse version of Parquet or something like that that we use, and an indexing system. We built all those things from scratch.

6. Customer Insights and Simulation vs. Real Data

I was curious about your customers. Across all of your customers, are there consistent blind spots or things that they consistently discover when they start to use Rerun to visualize all of their logs?

You can split them into customers and users to start with, and then look at the users of the open source. I think the really big qualitative thing there is that what the open source is mostly giving them is visualization, but it's radically simpler for them to do. So they do a lot more of it, with higher performance and so on.

Everybody is already doing some kind of visualization. It tends to be about volume: you look at a lot more things, and you're debugging the intermediate steps of your data pipeline and so on. It's a higher volume and therefore a higher hit rate. We'll have people tell us that they're working on some big self-driving car project and found bugs in their training data pipeline that had been around for 3 years. Those are the kinds of things on the open-source side.

On the commercial side, the biggest aha moments tend to be that they can build significantly simpler data pipelines because we have a different storage layer than what's normally used for this kind of data. The fact that the data pipeline structure can get so simple tends to be one of the really big unlocks—that they can do it very differently.

One place that guests on the show have really differed in perspective in physical applications is how they feel about simulations. It almost felt like a religious feeling in some of the interviews I've done. I'm curious where you land on training in simulation, and what fraction of your customers are training in simulation?

I'm more on the agnostic side personally. Coming from the fact that I'm not personally building these things, I try to keep an open mind.

We see both, I would say. The users of our commercial platform do not only rely on simulation. If you're really relying on simulation, then you have a little bit lower need to care as much about how you're saving your real data. If you don't care about saving your data at all, then you don't need a database to store it.

Because of that, we get a bias toward people who care about real data. That's also what I see mostly in the companies that are building products that interact with the real world. I don't think anyone knows yet that it's possible to do that well with just simulation. You most likely need a lot of real-world data to do that.

Totally. So you're a practical guy like me, building for practical people—a practical product.

But I was curious: have you ever built a feature in your product because it gave you a spark of joy, even if there was no obvious use case for it?

Like, it would give me personal joy? You mean that it's there?

I don't know that there's no case for it. We probably have, but I've forgotten about it. I think there are some aspects where we've really prioritized small, micro-interaction performance.

We've really cared about how fast it is to scroll back and forth in time. The Rerun Viewer feels a little bit like a video editor or something that you can scroll back and forth in.

And just having incredibly low latency on that.

That probably doesn't matter as much as we care about it, but—

Maybe it does.

It just feels really nice. It has this feeling of, “Oh, it feels amazing.” I think people react to that. We get a lot of people who just say, “I don't know, that gives you something visceral,” but you can't put that on a spreadsheet.

For sure. Yeah, totally. I love that answer. I have a couple of rapid-fire questions here, just to see where they go. Do you think there's an underrated robotics startup that you'd want to call out?

Yeah, I don't want to pick favorites here.

This is looking positive, though.

Okay. Yeah. Underrated. Let's see. I'll say one on the modeling side that I think is talked about too little: Generalist AI. I'm very impressed by what they've done on the shipping side.

And why? Tell me why you think they're underrated.

I think they had a lot of foresight in how they were going to collect data and really went for that very seriously.

And what was the method? Can you say?

Yeah, this is public. It's called UMI. Basically, I don't know if you've seen these videos where people hold a little robot gripper as if it were a person's hand, with a camera on it and so on. They're essentially puppeteering a fake robot. They bet on that, really scaled it up, and ended up showing really impressive dexterous manipulation. I was impressed by the strong bet on that direction and by really making it work. It's been really cool.

And on the kind of the two companies, the other examples I think are great are more on ship and deploy. I think Ultra is a really great company in the more pick-and-place for logistics.

And why do you think of them?

They're just very good at shipping and learning from that and building real systems while still being AI-first, and just seeing them doing that very well. There's a European company called Syriak that's really good at that same thing. They're in a different space, more manufacturing, but I've started seeing them do very practical, shipping-oriented work, and I think that's really cool to see.

7. Future of Robotics and Open Source Contributions

Cool. Do you think there are new data formats that you're going to need? Say, if we roll forward 3 years, what do you think are some new sensors or new types of objects that you're going to need to visualize in your product?

I think force and tactile sensing are clearly becoming a lot more popular. It's natural, right? If you imagine losing your sense of touch in your hands, picking things up is a lot more difficult, and so, unsurprisingly, that helps robots, too. Audio, too—more things involving audio. Those kinds of things.

In the LLM space, which I'm much more familiar with—the language space, especially where I started and where we see more of our customers these days—there are tons of benchmarks, and there's always fighting about whether a benchmark is good or bad. People are too oriented around benchmarks, but I feel like in robotics you don't really see benchmarks, as far as I can tell. Are there benchmarks that you think are meaningful? Are there benchmarks that should be created?

This is a hot topic in the field. I don't have a super-strong opinion that's better than anyone else's.

Yeah, give us your opinion.

I'll say it's a huge problem for the field that there aren't any great benchmarks.

Maybe who can fold laundry the fastest.

Yeah, but you end up with this difficult situation where the way to solve real problems is often to co-train and co-design for your hardware. You build and design these things together, and then it becomes quite hard to benchmark in a good way. How are you going to do that logistically? For something to be practical, it almost has to be simulation-based, and then you have all the problems with simulation not being particularly good for manipulation and interaction with the physical world. A perfect simulator doesn't exist.

I don't actually know what the solution is, but it's super, super important. I imagine we'll see much more of companies driving that for themselves—their own internal benchmarks. Companies will have these farms of robots that they evaluate: “Hey, we have a new model; deploy it,” and they'll run tests live on real things as a way of testing. I imagine that will continue to be that way for a bit.

Okay. A few years ago, when I was messing around with robots in my garage, it seemed like everyone was using this thing called ROS. It seemed like everyone kind of hated ROS but still used it. I was surprised to see on your website that ROS has incredible staying power—you still integrate with it. What's your take on it, whether it's good or bad? Why hasn't it been replaced if so many people are griping?

I think it's one of these things where it's almost a part of every roboticist's journey to have a stint of, “I'm going to rebuild ROS, but this time...” Very many roboticists have done that at some point and then given up. I think the core thing is that ROS is proof that network effects work. There's an ecosystem around it and an evolving standard, and it's very hard to come out of that.

The other part is that it's solving a problem that's difficult to make general. At the heart of ROS, it's kind of a message-passing system. There's a common set of message definitions, and a message could be how to represent an image that you can pass around, or a 3D pose—basically, where something is—or just any data that you might have. It turns out to be very useful to have a standard set of those. Most companies will have their own versions, but if you have a set of standards, then you can build pluggable systems. Someone builds a pluggable, open-source navigation module, and those things all add up. That's really, really hard to get out of.

Mhm.

We actually see a bunch of more serious efforts now to replace ROS. Let's see how it goes. I guess it's definitely a difficult problem, but I think in particular it's hard to fund.

Right?

ROS would be better now if it had been better funded, but it's just hard to find a good funding model.

Well, it's astonishing because so many well-funded companies are using it.

Yeah. Yeah.

You'd think there would be more of an open-source collaboration, like you see with databases or something else.

Yeah, I don't know that we see that with databases at super scale, either. Maybe there are a couple of databases like that, but usually there's a team first that really takes it very, very far.

Mhm.

And then that kind of continues, but I don't know why that hasn't happened with ROS. It certainly hasn't.

Well, good—it's a segue into one of my final questions. When you look at some of our joint customers that are doing these general-purpose robotics applications, raising at astonishing valuations pre-revenue—we've never seen these kinds of valuations pre-revenue before—does it feel like a bubble to you or not?

Yeah, it's hard to know. There's some of that, but I think they're going after astonishingly large markets. I think there's something particular about general robotics, which is that one of the hardest parts of robotics is making a decent price point for the hardware.

You kind of need a lot of scale for that. The best way to generate scale is to be able to serve a lot of use cases with the same hardware. That also requires a lot of scale, because then you need a very large amount of data and compute to make that happen. So I kind of see why a lot of capital gets amassed to go after it.

Are you a fan of the humanoid form factor or not?

I mean, I'm a fan because I'm just sort of an optimistic person, I think. I would love to have a humanoid that was doing my dishes.

Do you think it will be a humanoid?

I think, in the long run, for home tasks, there are a set of tasks that are likely to be humanoid or humanoid-like. In the shorter term, in many cases, I think a semihumanoid is a much more practical form factor: a wheeled, fixed-base system with humanoid-like arms.

It's like The Jetsons.

Yeah, a little bit that kind of thing. I think the thing that humanish form factors have going for them is data collection.

Mm-hm.

So, you could imagine that maybe it's more efficient to have 4 arms, but it's really hard for a human to control 4 arms at the same time. So as long as real-world data collection is important, I think that's a really important factor that will be hard to get away from.

Mm-hm. Is there a tool out there in your space that you wish existed but doesn't?

That's a good question. I don't know. I don't know that I've—

You'd probably make it if there was one.

Yeah, I'd probably make it in that case.

All right. And last question, I guess: What do you think is the biggest breakthrough we're likely to see in the next 1 or 2 years in your space?

Honestly, I don't know. I think significantly improved robustness, and that really is one of the main things that matters. So I think over the next year, seeing more and more larger, longer, and more complicated tasks being performed in a robust way—self-correcting and so on—and a little bit more learning on the fly, I think that's a likely trajectory if things go great this year.

My guess is that it's going to take a little longer to combine that with really great high-level reasoning as well. So maybe that's the year after. But no, I'm not just guessing.

I guess, in your experience, how is training embodied AI or robotic systems different from training LLMs?

Yeah. Maybe I can start with some things that are common. I think the absolute best AI teams—the best LLM teams and physical AI teams—kind of have the researcher or modeler work on both the data and the modeling together. The same person is making complicated decisions about exactly how to preprocess data, what data to include, and the modeling.

That's super core. It just turns out that it's a lot easier to do for LLM teams.

Why is that?

There are maybe 3 core things that you need to be doing. You need to be looking at your data all the time. For text, you can just read the text; it comes in. If you have other signals, they're very easy to visualize. But doing that for physical data is very difficult. That's a little bit closer to being solved with tools like Rerun.

The other part is that you want to have a very flexible system that makes it possible for the researcher to edit the data pipelines and so on. You also need to have high performance when you do that. So you want to have both of those things: high performance and flexibility.

For LLM data, that's text and other normal things that you can use with robust, well-proven storage formats like Parquet and Iceberg, and high-scale data-processing engines like Spark and Databricks. Those tools have been around and matured quite a lot. They give you easy, declarative, database-style APIs to edit how the data comes in.

But for robotics, none of that exists, basically. All the data formats are built for a world before machine learning.

Mm-hm.

So they're not flexible. They're really optimized for fast logging rather than flexible querying.

Mm-hm. And when you say flexible querying, can you give me an example? I'm imagining I want all the places where it was recorded at night or something like that.

Exactly, because I want all the times that it was at night, the left gripper didn't open, and there was a failure of whatever type—Type B. That might require you to query the metadata and the signals inside a specific recording.

In fact, are you doing some kind of semantic embedding to make that possible?

Some users do that, but you could just be doing normal querying: this signal value is above 5, or this other thing is this, and the text is whatever—the text description.

The normal robotics stack doesn't have a query layer or a format where you can represent this sort of multimodal physical data, with 3D or whatever—all these different things that you can store in a file format—and have some kind of database query engine operate on top of it. So you end up putting it in file formats that are optimized for saving logs.

Mm-hm.

And then there is no query format. The way you answer all these questions is that you write a huge parallel job that reads the file into memory.

Mm-hm.

It reads everything field by field, and you're writing imperative custom code for every such query, running it as a huge parallel job, and aggregating it manually. So it's a huge amount of work where it should just be a SQL query or something like that.

I feel like I'm seeing a kind of convergence, in some cases, of LLMs and embodied AI models. For example, one of our customers, Wayve, actually uses an LLM to describe what the car is doing as it's driving around, which is really interesting. Do you see other things like that happening?

Yeah, particularly for data annotation, that's a really big one. I see a lot of physical AI teams training one model or using an open model to annotate their robotics recordings—to say, here's where it started or stopped, this is the task that it's doing right now, and things like that. Obviously, using embeddings for images and motion and stuff like that for search and curation is also super common.

One thing that I think about for Weights & Biases is that we see more and more agents interacting with our product, and we've been pretty passionate about doing really good visualizations.

Yeah.

But I kind of wonder—I think a lot of people now just pull the data out of Weights & Biases and then use Streamlit or Marimo or something to—

Yeah.

—conjure their own custom visualizations. I've been thinking: Do visualizations even really matter in 2027? Do you have a perspective on that?

I think they'll continue to matter, but potentially not in the same way. People will do more and more custom visualizations based on specific needs. We actually see that happening quite a bit: people use Codex or Claude or whatever to say, “Oh, I need this special-case little visualization,” and then kick off an agent that builds their specialized tool.

So if SaaS-style applications involve building a very dedicated tool like that, I would imagine that gets harder and harder. The underlying systems, assuming they need to be very high-scale and robust, will probably take a little more time.

Okay. There's another thing—I'm feeling like I'm just fighting my own demons here—but are there any trade-offs between flexibility for people in the first case, where they're just setting things up and messing around, versus real scalability at production volumes?

I don't know that there's an exact trade-off between scalability for production volumes and flexibility, to be honest. I think we kind of solved that. That took about 2 years of iterating on this data model, and I think we got a really good design that can fit both.

Of course, we don't get all the way there. For instance, the way our system is designed, we'll never get to the point where our logging is useful for really low-latency teleoperation. It's just not built for that.

Yeah.

So we won't get all the way there. But maybe more importantly, on usability, we want to have something that's flexible enough for researchers to use. One of the things that goes into that is that we don't want to require you to compile a schema before being able to log something, right?

Mm-hm.

In production use cases, having systems that are built more around a precompiled, known schema can fit a bit more naturally into those systems and give you a little performance, although you can work around most of those performance problems.

That low-friction usability for a researcher versus, “Hey, I'm plugging this into my already existing schemas and production system”—that's taken us a lot more work to support as well. We still have a little left to go on that journey to really cleanly integrate with those production systems, those robot production systems. That's been a big trade-off for us, for sure.

What about open-source contributions? Do you get a lot of them?

We do, but I would call it a medium amount.

Do you think it's worth the overhead of managing them?

I think it is for us. If it was just for the sake of the contributions, it would not be worth it.

Mm-hm.

There's a lot of work: We need to review pull requests and so on. So for that purpose, no, not really. For us, open source gives a lot of trust.

Mhm.

Right. So it’s very easy to adopt, and I think that just makes it very easy. Rerun gets built into a lot of other open-source tools. One of the most impactful open-source projects in robotics learning has been Hugging Face’s LeRobot project, and they built Rerun into that for the visualization, and that wouldn’t have happened. We built it into NVIDIA’s new simulator engine, and it gets built into that, and so—

Those kinds of things matter a lot in terms of contributions.

I think it’s actually more in terms of some person—someone who really loves Rerun—having some little thing that they’re really annoyed about and having a way for them to do that.

And maybe it would have practically been easier if they just asked for it and we did it, but—

That kind of participation builds a kind of relationship with people that I think adds up more than the particular—

Mhm.

thing. Yeah.

Awesome. All right. Well, thanks again.

Yeah. Thanks. Yeah.