Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson

Today my guest is Joseph Nelson, CEO of Roboflow, a computer vision platform that supports more than 1 million engineers and more than half of the Fortune 100 as they seek to turn proprietary image and video data into a competitive advantage. We begin with an overview of computer vision capabilities today. Joseph notes that while language is fundamentally a human construct and inherently optimized to be understood, the real world contains a fat tail of chaotic scenes that are not at all optimized for understanding.

And thus, just as the Vision Transformer came about 3 years after the original Transformer, computer vision today is roughly where language capabilities were 3 years ago, with the introduction of ChatGPT and GPT-4. Which is to say that while frontier models can do amazing things, and most problems can be solved if you're willing to put in the work to fine-tune and pay any inference cost, we have a long way to go before foundation models will really be able to do it all.

To make this concrete, Roboflow maintains a site called VisionCheckup.com, which highlights the spatial reasoning, precision measurement, and grounding failures that still plague even the best multimodal models today. And importantly, even when frontier models can solve a particular task, you can't wait 40 seconds for a reply when you're powering instant replay at Wimbledon or monitoring for defects on a high-throughput manufacturing line. And so, there's often still a lot of work left to do to get vision models running efficiently enough to meet production latency and edge-deployment requirements.

This is where Roboflow comes in, and I was very interested to hear Joseph describe what it looks like to go from an open-source vision model to deploying your own task-specific model today. He emphasizes first the importance of establishing clear requirements up front, because the performance thresholds that different customers need to hit for their respective use cases can vary quite dramatically.

From there, the process often involves distilling frontier-model capabilities into much smaller models, like Roboflow's own RF-DETR model, which they derived from Meta's DINOv2 backbone using a really interesting training technique called neural architecture search. Neural architecture search, in turn, uses a weight-sharing technique to train thousands of network configurations at once, all within a single training run.

This process ultimately produces a set of models of varying sizes that collectively map out a performance Pareto frontier. And today, Roboflow has productized this approach so that anyone can now run it on their own dataset and come out the other end with an N=1 model optimized specifically for their problem.

From there, we cover a number of additional topics as well. Joseph explains that Chinese companies have consistently led in computer vision, how much the American open-source ecosystem currently depends on Meta, and why he's optimistic that NVIDIA will fill the gap if Meta's new AI leadership changes priorities.

He also describes how coding agents are expanding the market for Roboflow's tools, how skills are emerging as a new go-to-market vector, and how Roboflow plans to use a first-party agent to guide users through the process of building computer vision pipelines. We also discuss the state of AI's aesthetic taste and why the inherent subjectivity of aesthetic preferences makes this such a hard problem.

We hear about the emerging S-curves that Joseph is watching, including world models, vision-language-action models being developed for robotics, inference-time scaling for vision, and wearables, which are now selling millions of units per year. We get his vision for how computer vision contributes to a good life as AI matures, which includes everything from precision agriculture and food safety to self-driving commutes and real-time sports analytics.

And finally, he explains why he worries that overly opinionated regulation could accidentally stifle all sorts of surprising but valuable use cases, and why he recommends that policymakers focus on outcomes instead of trying to regulate the tools that people are using. When it comes to computer vision, Joseph has quite literally seen it all. So whether you're looking to catch up on the field, as I was, or looking for a practical framework with which to approach a specific challenge, I think you will find a lot of value in this conversation.

Joseph Nelson, CEO at Roboflow, welcome to The Cognitive Revolution. I'm excited for this. Regular listeners will know I really got into AI in a full-time, obsessive way in my role as founder of Waymark. And it was such an exciting time 4 years ago, when things were just starting to work.

I ended up going really deep on what was available in computer vision at the time, with CLIP, BLIP, BLIP-2, and CLIP embeddings, trying to figure out how to solve the problem we had at the time. We had all these small-business users. We had developed a pretty good technique for scraping their websites and their online presence and creating an image library for them.

But then what do you do with that image library? It was initially just a total jumble of photos. We couldn't make any sense of it. We made very blind guesses as to what we would actually put into content for them, and that obviously had a long way to go before it really started to work.

I had a ton of fun in the 2022 into 2023 timeframe getting deep into the weeds on that stuff. Obviously, a lot has happened since, so I'm really excited to catch up on a few years of computer vision progress in 90 minutes or so.

Maybe let's start by setting the stage. Where are we today in terms of computer vision? You can come at that from a lot of different angles. Maybe start with use cases: What are the use cases out there that are really well established, that are driving the most volume, and that are driving the most value? Give us a survey, a lay of the land.

Since you brought up CLIP, maybe we can start in terms of some of the research that's powering what's now possible, and then let's also do the use cases that flow from there. With vision, it's funny because we can think about AI and the trends in machine learning. Originally, a lot of it was vision. You had ImageNet, you had MNIST, and deep learning gave rise to the question of whether a photo was a cat or a dog on the internet.

And then you had language, I would say, almost jump out and take the lead in terms of wow factor and understanding, with the introduction of the Transformer in the “Attention Is All You Need” paper in 2017. Then you almost have 5 years of language cooking with scaling laws, Chinchilla, and GPT-2. And then you start to get language products in GPT-3 and GPT-4. I would say GPT-3 and GPT-4 were really where things started to break out, along with ChatGPT in 2022.

That 5-year time delay from the introduction of the Transformer to products that become used by nearly 1 billion users, I think, every single week now, is now happening in vision because you had the Vision Transformer introduced in 2020. And so that ends up being another stepwise change in what's possible and what capabilities are easy, or easier, out of the box.

But to your point, what's interesting is that historically there's been this divergence of, “Is this a language problem, or is this a vision problem?” Modalities are crashing together because, just like our brains, you get more context if you can use language and vision together.

However, there are some pretty meaningful differences in visual understanding, both in the way that visual models work and in the use cases where visual understanding is most impactful. One of the biggest ones is that I think about the way our brains work as a useful analogy for how our systems for visual reasoning will work.

We have this big LLM reasoning engine in our heads that is our brain, but we also have rods and cones in the visual cortex that operate and make decisions in what we jokingly call your lizard brain—a fast-reaction way of understanding the world. Biology has evolved specialized systems for visual understanding, distinct from broad-scale reasoning, and the number of neurons dedicated to that is greater than for any other sense. I think the same will be true, based on biological inspiration, for the systems used in machine learning.

So what does that mean in practice, in that abstract idea? It means a lot of stuff runs at the edge. A lot of stuff runs at low latency. A lot of stuff runs out in the real world.

For example, in a lot of language, multimodal, or multi-agent reasoning problems, you can have the benefit of assuming you have near-infinite compute because you can run a long-running job in a data center. A lot of visual tasks where vision is most useful tend to be where you don't already have a human or eyes on the problem.

You're understanding an environment, maybe in a remote location. Maybe it's a manufacturing line. Maybe you're shipping a product. Maybe you've got cells underneath a microscope. Maybe you're looking through a telescope and discovering new galaxies. Maybe you're building robots.

And for a lot of those use cases—not all, but many—you need fast reaction times in addition to large-scale reasoning. And so you see this increasing divergence and specialization of where vision is especially helpful: for low-latency tasks and for things that are, perhaps intuitively, out in the real world. These are systems that we want to observe.

LLMs and language are inherently human constructs. But the visual world isn't inherently a human construct, right? Language only exists where people do and in systems that humans have crafted. The world's much bigger than just language. Anecdotally, the number of distinct scenes in a day is more diverse than the number of unique words you probably read in a day.

And so, that heterogeneity and richness make visual reasoning harder, I think. It means that the long tails are fatter, and it means that the use cases tend to be out in the world, for lack of a better way of describing it. The use cases that we see become a natural sample of where visual AI and computer vision are being used in the real world. About 1 million developers download open source every 30 days, and about half of the Fortune 100 are building on the platform. So, we have this insight into what's actually making its way to production and where people are tinkering.

It tends to be these operationally complex problems, maybe in the enterprise sets of use cases. From the platform, you get broad amounts of inspiration, which could be a hobbyist who wants to understand—I like to play board games. The dice that you just threw—I swear, every time I play Catan, my numbers get drawn the most. So, I want a camera to prove to my friends that I'm the most resource-efficient compared to the resources that I drew, based on the way the dice came out.

That's a serious and a joking example. There's this YouTuber who maintains a channel called Dave's Armory out of Canada, and he's built a flamethrower, weed-killing robot with Roboflow. He built his son a self-driving couch that follows him around the yard. These are silly things. Then you have more serious use cases, like powering instant replay at sports broadcasts at Wimbledon, or doing quality assurance on products being produced at Rivian, or any sort of physical-world thing.

But one thing that I deeply believe, and I think the rest of the world is coming around to this, is that visual AI, visual understanding, and at least that part of multimodal reasoning are going to be bigger and more important than just language. The way for AI to reach its full potential is for it to be out in the world. It needs to understand, see, and reason. The world is a pretty big place, and the universe is even bigger. The systems that we want to use and rely on need that type of capability.

So, linking that back to the research that's progressed, there's a lot of work to be done, but there's been a lot of progress from CLIP to the present that we can talk about in more detail. To set the stage, I would basically be really optimistic that we're approaching the ChatGPT moment for vision. The infrastructure to power all of that is coming online, which means you're about to see a Cambrian explosion in all the places where it can be used, and consumer expectations are just going to be disappointed absent the ability for folks to have visual understanding in the products and services we use day to day. That's how I'd think about what's going on and what that means. These use cases underpin the ways the research is making its way into production so far.

When you said that you think we're coming up on the ChatGPT moment, I thought that was quite interesting. Sometimes I give talks to an audience that I'm trying to catch up on what's going on with AI, and I often give them the MNIST example: look how simple this is for us, but we still can't write explicit code to identify these handwritten digits. As simple as that problem is, there's no explicit algorithm for it even today, right? That gives people a sense of why it is that we need this sort of fuzzier kind of intelligence.

Then I zoom forward to the GPT-4 system card, and I show the image that I'm sure you're familiar with of the guy hanging off the taxi in New York, ironing on the back of the taxi. I show them how we went from an ImageNet breakthrough in 2012 to that capability 10 years later, where the model says, "It's unusual to see this guy hanging off the back of the taxi doing ironing."

I was going to ask—and it sounds like your answer is going to be no, but I want to get a lot deeper under the hood on that—to what degree could we consider vision almost a solved problem? Are there things—and I know that not everything works immediately out of the box, and not everything is going to work at the cost profile or latency requirement that you'd want—but if we started off with just this: is there any vision problem that we couldn't solve if we really put our minds to it today? That's the working definition I had for a solved problem. Do you think we're not there? And if we're not there, why aren't we there? What can't we do yet?

The way I think about something being a solved problem is that I just ask the model, and it almost impresses me. It delights me that it already understands and can do the thing I asked it to. I think that's why ChatGPT was such an aha moment for folks, because you no longer had to train a model to understand sentiment or describe text or whatever. I just talk with it, it talks back, and it feels like talking to a first-grader, perhaps.

In vision, solved problems are a subset of places where that's true. But it's not nearly as solved as language is. I think the reason for this, in my mental model for what it takes to get us there, is what we were mentioning earlier: the world's very heterogeneous compared to language. If you think about this in a very first-principles way, consider the amount of data that it takes to encode text with Unicode. I can represent all text using Unicode in memory much more efficiently than even representing a single image, because I have 3 color channels, 0 to 255, RGB, pixel by pixel. That data disparity—how much more information it takes to even encode a visual scene—is an anecdotal example of why there's more heterogeneity in understanding visual scenes.

In a more concrete way, again, the number of scenes in a day is different from the number of words that you read in a day. My mental model for this is to think about the world as a standard bell-curve distribution. In the fat center of that curve, what we're measuring is the frequency with which a thing exists out in the world. If you went out, let's say, and took a walk, or maybe just went throughout your full day, and wrote down every object that you saw, then looked back at your notes and how many objects you saw and maybe how long you looked at them, you would have something of a bell curve of the things that you saw that repeatedly showed up: person, car, food. There would be some things that are longer-tail, if you will. Maybe one day you were changing your oil, so you were under the hood of your car. Even that's something that you're not going to do every day.

So, for vision, having a model that can reach into those long tails is taking just a bit longer, because it's heterogeneous and because I think those tails are fatter. It takes longer to have the data represented and to have models that can reason about all the various different scenes and videos that exist out there. What does that mean? Some things are, quote-unquote, solved problems, like counting people in an image. Increasingly, OCR feels like a solved problem.

There's a model, GLM, that we're really excited about. It can run in real time, and you can query it with, "Hey, how much was my salsa from this receipt?" Or, from these Google Street View images, "What's the house address on the left?" It's able to visually reason, extract, and pull the correct answer almost always. Something like that feels closer to a solved problem. But the nature of how diverse some scenes are means it's going to take representation and probably some reasoning models to be able to reach into those long tails.

What's happening in vision—I used to say in slow motion, but it feels faster—is that you're getting models, multimodal models. This is the big LLMs like Gemini just as much as open models like Molmo, just as much as models like the DETR family of transformers, increasingly pushing outward on this visual bell curve. More and more of the things that you see on a given day are understood zero- or few-shot. So, it becomes maybe a semantic question of what you consider a solved problem.

If you're in the middle of that bell curve, yes, it's a solved problem. But if it's something where it's so impressively surprising and delightful, it becomes a question of how long until someone starts to query and ask for things that wouldn't have been represented and trained on. I think we're riding that curve. The expectation is getting faster.

One other complexity in vision is what we talked about earlier, which is that a lot of vision is in edge-constrained environments. You want answers now. You're running a webcam, it's on your phone, or it's in the palm of your hand. That means you also don't have the benefit of waiting 40 seconds for a reply from a model about the thing that you were interested in querying. It doesn't mean that the problems are intractable, but heuristically, I see maybe an 18-month delay between a SOTA capability from a multimodal, cloud-available model and something that you can get to run on an edge device.

Here, we could define an edge device as maybe a Jetson Orin-level computer, or maybe even an iPhone, where it's opaque what exact GPU comparison you would make. Those things make vision feel unsolved. But I still think what's going to continue to happen is the expanding nature of that bell curve. If you think about that mental model for where we're going with visual capabilities, then I think that's a good way to think about where the field is headed.

So, I want to work through that kind of Pareto frontier of performance, cost, latency, and where it can run—the trade-offs. But let's do one more double-click on the most expensive end of that curve, which is the cloud-available frontier models, I think you described them as.

Obviously, these things are famously spiky. I would say it's been a while since I've had an image use case where I was like, “Oh, this thing can't do it or can't see it.” I do remember some of those not too long ago, for example, with the ARC-AGI puzzles. I remember trying frontier models on some of those puzzles, getting strange results, and then working my way back to, “Can you just describe what the starting state is?”

And then I was like, “Oh, well, no wonder it can't do the problems. It can't see the starting state accurately. It can't accurately just define which boxes are colored what colors.” But that's been a while. I guess I don't know—you might, I'm sure you would know—if we just take ARC-AGI puzzles and put them in today, are they accurately perceived?

Are there other things that would be good representative examples of spikiness, where people might be surprised that a frontier LLM wouldn't be able to see this the right way? Also, do they work with few-shot prompting? Obviously, few-shot has been a huge unlock in general, but does it work for vision? I really don't know that. So, I guess to sum that up, can we go one level deeper in terms of the capability profile of the frontier LLMs?

Totally. Yeah, we spent a bunch of time thinking about this and trying to help people navigate what their expectations should be for the problem that they're solving, where they may be able to have a zero-shot or few-shot problem, or where they might be in a world where they need more representation of that problem before they can count on their model.

What are the common types of failure, and where are you most likely to be disappointed? One of them is grounding, in particular. Grounding refers to segmentation and detection—traditional tasks. But if you want to say, in your example, finding the starting position in ARC-AGI, or sometimes I'll try to do crosswords, and I'll be surprisingly disappointed by the model's ability to know where the word goes in the crossword.

However, if I just treat it as a text problem and say, “Here's the clue and here are some of the letters that I know are in the word,” the model almost always does better if it just thinks about it like text, if it doesn't also have to think about where that issue is in the puzzle. Measurement is another example. You can think about this as a subset of problems that are inherently very precise, where there's lots of precision involved.

Some of this, I think, is the result of the post-training that's applied to these problems. In a lot of the labs—this is a little bit hearsay, but seems to be increasingly common knowledge—they're not as interested in just solving the segmentation problem. They're interested in solving what the user's intent was, and whether segmentation is a tool call as part of that intent.

But even still, the segmentation portion of that chain of thought is pretty unsolved because there are so many different things that you would want to measure and see, and have a pixel-perfect representation of. You will see that when you take more time to reason—aka, you do more tool calling and find more specialized expert models for the scene you're looking at—you'll get better results. But in general, I would frame grounding as still a pretty difficult issue when there's a precise, pixel-level need.

The second place where I think there's disappointment on the Pareto frontier you described of accuracy and speed is actually speed itself. I was using Gemini 3 the other day to try to automatically label a bunch of data for me, prompting it, and it would do it, but it would take 40 seconds each time.

Interestingly, the nondeterministic nature of generative AI also led to some pretty difficult downstream results because, for that example, I wanted really precise, consistent—not necessarily what you think could be correct, or your best guess—results. So, that's another challenge: you and I could go try the same problem and get different results from the same model at the same time of day.

We maintain this other property called Playground at playground.roboflow.com, where you can do SAM 3 versus Gemini versus Claude Opus. What's really funny to me is that I'll find these failure cases, report them to our team, and then they don't reproduce. It's actually not because of our use of the models; the model itself doesn't reproduce the same way.

That continues to be a little bit of a challenge. After speed, I would say the next issue is the reproducibility challenge. Then, I mean, it's a little bit redundant, but we talked about the representation. You get into those long tails, and this becomes a function of the type of question you're asking, but there is still a lot of the world that's just not understood by models, as well as the challenge of being able to articulate not just the pixel-wise segmentation, but where one thing is with respect to another thing.

Those are the common sorts of failure patterns we see.

Now, you had a good follow-up, which was: How does few-shot help address these things versus zero-shot? The answer is pretty good, but still not infallible. It helps. How much does it help, obviously, is an interesting question.

One way that we think about these problems is that we introduced a benchmark at NeurIPS last year called RF100-VL, or Roboflow 100 Vision Language. Basically, folks that use Roboflow for research will share their work in an effort to build upon others' work and bring the whole computer vision community forward.

There's a large set of open-source datasets that folks can learn from and try, to accelerate the problem they're solving. We went and worked with users, researchers, and the hundreds of thousands of folks who are sharing open-source projects to move the whole community forward. We created a basket of 100 of them—problems that seem to be represented in visual AI.

The domains broke down into problems like industrial, health care, flora and fauna, and documents. There's a miscellaneous category because, of course, it's tough to put everything into a single bucket.

We evaluated Gemini, SAM 3, and OpenAI, of course, along with a number of multimodal LLMs. We also evaluated models like OWL-ViT, which is an open-source model that supports few-shot prompting. OWLv2 is the most current version. We also evaluated a model called Grounding DINO, which is the new version of DINO; it’s behind an API but still more open in general.

Basically, the question was: “Can you do successful segmentation the same way? If I pass these same instructions to a human annotator, how does the model do? How would the person do at finding all things in an image based on those domains?” The best model at the time we published the work was Gemini 2, but that was only 12.5% across all domains.

That shows the gap in how far these models have to go. This data wasn’t arbitrary. What’s really interesting is that this sample wasn’t a perfectly curated research dataset like COCO or Objects365, or one of those works that are very helpful contributions to the field. These were the places that folks are actually using models.

The second thing we did was run a competition at CVPR on a 20-dataset subset, just because of compute constraints. We thought we could get the point across with RF20 instead of RF100. We said, “If you had few-shot—that is, 1, 2, 3, 4, and 5 image examples—how much do you see the models improve and progress by comparison to one another?”

The lift there, I think, maximally was around 10% for a single model. I’d have to check the average across all domains, which is meaningful, especially when you’re starting at 12%, but it’s not a panacea. It’s okay, great—I’m helping ground the model with the domain that I’m looking at—but it doesn’t solve all the problems.

I will say that’s a place where I’m bullish. Specifically, I’m bullish about few-shot for visual problems and providing prompts, perhaps as image-text pairs or even just as images with the task you’re interested in, whether that’s grounding, description, measurement, or what have you.

The story is clear: We need continuously better representations of the real-world problems people are trying to solve, and we still have a bit of a way to go as a community before it’s totally solved. But progress is happening pretty fast. That’s the progression of where things are.

Okay, well, that’s a good start at the top of the curve. Let’s maybe work our way down the curve. There are obviously multiple reasons that one wants to go down the curve. You may even add to my list, but faster response time is obviously a huge one. Lower cost is another great one. The ability to run on the edge is another great one, and the ability not to have to send your data over the wire is another great one.

There may be even more beyond that that you would highlight, but take me from, “I can naively send images into one of a few frontier APIs, maybe with few-shot,” to something else. I’m not sure which way makes more sense to organize it. You could go from large models to the smallest, most able-to-run-on-the-edge models, or maybe a different way.

Maybe they line up, but maybe there’s a difference in how you would think about actually coaching people, starting with, “Well, why don’t you just—you know, at least what I normally do when I have a new challenge is, well, let’s just see what a frontier model can do out of the box.” Once I’ve calibrated myself there, I can approach optimization and fine-tuning in any number of ways.

I’d also be interested in whether that’s the same as the curve of trading off convenience for all these other goods of latency and cost. I’d be interested to hear how you recommend people navigate the path from that kind of first naive baseline performance to where they should go next. What model should they try? How much data are they going to need? What technique should they use? If that doesn’t work, what do they do until they’re finally in some happy place where they’ve got everything that they wanted?

I appreciate the way you broke that down. There’s the speed-accuracy curve, which folks know straight off: A bigger model takes more compute and is slower. Then there are these other dimensions that don’t fit neatly on a graph but might be really important to somebody. You may want to own your AI. You may want to be building your own IP as a business, or it may be important to you as an individual.

Maybe there’s a constraint in the business case itself, in the problem that you’re solving, where low latency is super critical. That could also be a privacy consideration, where, as you said, you want to keep the data local to your thing. That could also be security, as a close cousin to that. There are a number of things that frame where someone is going to fall along those axes.

Now, how to navigate it—and what I think often matters—is that it stems from the job to be done, if you will. Making that less generic and more real, and something that people can actually think about, is important. What I think about is that many problems, especially in vision, require an instantaneous response.

You have something going down a line. You’re watching a live sports broadcast. You need a decision right there, right then. Of course, that puts someone into a real-time category. Usually, there’s a need to colocate compute or run something on the edge, and you already know that you’re in a class of models where, inherently, if you’re going to run on the edge, you need to own it.

You need to own the model. You need to have the weights. You need to put it into your environment. That’s where open source comes in. We invest a ton in open source for this reason—in terms of publishing our own models and supporting open-source repositories.

I’m very optimistic about the future of open-source AI, both because I think it’s an important way for everyone to realize the benefits from it and because I think it helps discover, bottoms-up, all the ways this technology is going to be impactful.

A pattern that we see in navigating these two is similar to what you described: Is this problem tractable? Is this doable at all with the types of models, and then maybe the reasoning and nudges I would use for pre- and post-processing of the model? Then, fitting it to where I want to run it.

We see a rising trend of distillation. For example, SAM 3 is a promptable model where I can say, “Find all the sheep. Find all the people. Find all the hockey players in this image,” and it’ll do a pretty good job. It’s not infallible. There might be perspectives you didn’t include, but it’s state-of-the-art for open-vocabulary segmentation, at a minimum.

Let’s pretend that you’re one of our customers who does instant replay clipping at Wimbledon and the US Open. They actually bring compute to the courts because you have a live broadcast and sub-10 nanoseconds to put something over the wire. There is compute that could run SAM 3, but in this case, it just wasn’t economical to get the compute load of that size.

Basically, you’re in the situation where you want a model that you can own and run at the edge, live, over the wire to, in their case, frame the instant replay that you want to put on broadcast networks.

Maybe you see it on ESPN+ or CBS or something. The things they want to know aren't an open-vocabulary list, right? You don't need a model that can see everything. I could prompt SAM 3 for poker chips; I could prompt it for deer. The odds that those things show up at Wimbledon are pretty limited, hopefully, and the odds that I need them for my replay model are even more limited.

So, I could use prior video from a prior year, prompt SAM 3, and say, “Give me the tennis ball, give me the court, give me the net.” I could then have that go and auto-label a dataset. Then I have a really high-quality, curated, specific dataset, and I can train my own smaller model that I can run on the edge—a model like RF-DETR, which is the current state of the art for doing real-time segmentation and object detection.

Not only does it run on the edge, but it's so efficient that I can run multiple streams on a single A100 in this case. So, I get the cost advantage, and then, of course, if something's cheaper, you open up more possibilities. A common pattern we'll see is that I use a model to failure. If it does work, great—then can I make it mine, or can I fit it into a use case where I know I'm going to need to run it on the edge?

The other thing is that even as models eat more of the overall task, it's still okay, of course, to put a model in a harness or do your pre- and post-processing of the model to nudge it in the direction of what you would expect. There's no shame in still using traditional techniques for post-processing.

For example, there are models where I can just ask Gemini to count the tennis players on the court, and it would give me just the count. But I couldn't just ask a Detection Transformer to count. I could say, “Give me the persons,” and then it responds that there are 2 people found. Then I would add a tiny bit of logic—super-fast code—that just counts the class outputs, right?

There's no shame in continuing to stitch together post-processing logic for the purposes of optimizing speed or wherever something's going to be possible. It reminds me of the database wars. The most recent one was vector databases, but even before that, when you had a lot of SQL databases and NoSQL databases, where is it most useful to have document stores, where you have unstructured data that references one another, and where is it most useful to have structured data or structured tables?

At some point in time, of course, you're going to have to deal with sharding your databases if you have everything in those records, versus if you maybe had a NoSQL database that's going to scale for you automatically. There are trade-offs in both those worlds. It reminds me of that, where it's not a question of pure capability; it's a question of the constraints of the job to be done at hand.

Again, running things on the edge in real time, or having a model you own—there's also cost. Streaming video to the cloud nonstop can be expensive if you have quite a few streams, versus maybe using owned compute. These are all things that drive the reasons why you can use maximum-ceiling intelligence and then apply it to a system that becomes one that you own and use.

You've seen this trend in language and coding models, too, right? There's specialization, small models, and expert models. In some ways, that's a place where I think language has drawn inspiration from vision. There used to be a consensus, it felt like, that it's one model to rule them all. Now, it's increasingly flipped back to, actually, those domain-specific models and optimizations to be made.

I think vision is increasingly in the camp of: you do want a domain-expertise model because you might be compute-constrained in where you're going to run your system. I don't know if that's a color you're thinking about, like navigating the Pareto considerations, but those are the things that we at least commonly see when we see folks approach problems like this.

Yeah, that's great. Could you maybe give a little sketch of the scaling laws, so to speak? By that I mean, this is a little dated at this point, but I used to have a talk where I coached people through how to automate tasks, usually language tasks. One of the big things was that you have to think in orders of magnitude of data, right?

If it's not working out of the box, then get me 10 good labeled examples. If that's not enough, you could probably put that into context, and then it might work. If that doesn't work, then you might need to think about 100 examples, and you might have a small fine-tune on your hands. If that doesn't work, maybe 1,000 will.

I wonder what you see in terms of, first of all, what people need in terms of reliability. What do they get out of the box, and how many steps do they typically have to take up through those orders of magnitude to actually get there? Maybe also, how does that relate to model size?

Obviously, bigger models in general can do more, but especially if you're doing very narrow stuff, it seems like you probably can get everything squished into pretty small models. What are the model sizes, and how much data do you need to step up to where people are hitting the thresholds they need to actually deploy?

The data question is one that, as you alluded to, gets informed by the business problem of how many nines you need before you're able to use a thing in production. For example, if you're building a system where your alternative is that you have no eyes on the thing, then you're probably more accepting of a less accurate model.

Maybe you're trying to get a sense of attendance, or you're staffing your retail location. Absent vision, you have no idea how many folks are coming in day in and day out. Maybe you can go check your point-of-sale system; that's one source. But let's just admit you have to try to get to some source of truth where some people are in the store at the same time, or not everyone checks out, or there's a different way we want this information.

The point is, in this case, we might not have any eyes on the problem. We might not know how many folks are going to be present in our store at a given point in time. Maybe a museum is an even better example because people don't explicitly check out. If you add a model that is counting and it's 80% accurate at counting, you might be like, “Great, ship it, put it in production.”

If we have a sense of whether we have a dozen people or 100 people at a given point in time, then I'm comfortable with that. Whereas that's totally bad for some other problems. We have healthcare manufacturers that make critical, life-saving products. You can think of IV bags in hospitals.

For them, an escaped defect—meaning a piece of particulate matter making its way into a product like that—is life-threatening, let alone the detrimental impacts it would have on the company's reputation and so forth. With lives on the line, you need to have really high recall to find if there's any particulate matter.

You're probably comfortable with adding vision to augment whatever system you currently have, whether it's people, lab inspections, or a sampling methodology, and then weaning into where you can get more reliable vision systems. Ultimately, like a lot of things, it's a business question: What level of accuracy tolerance do you accept?

In terms of the number of images or videos that it takes to get there, this becomes a function of how varied the scene of interest is. On one end of the spectrum, you have self-driving cars that are in the big, wide-open, crazy world.

I remember Andrej Karpathy's talk at CVPR about 5 years ago, where he was like, “Find a stop sign. How hard is it going to be? It's the same red octagon everywhere, right?” And he's like, “Wrong. In photo after photo, here's a stop sign that's blocked by a bush in a parking lot. Here's a stop sign that says, ‘Only stop if you're going right at the intersection.’ Here's a stop sign that's on a school bus. That's a stop sign that appears for a temporary amount of time. Here's a stop sign that's on a gate where, when the gate is up, you can see the stop sign, but you don't need to stop.”

You're like, “Man, something as simple and straightforward as a stop sign has tons and tons of edge cases to understand.” That's navigating the world fully autonomously. Yet, of course, look at how long it's taken us to claim victory laps on self-driving cars. Now that they're here, it's almost as though folks are surprisingly unexcited about it.

You contrast that with something like a manufacturing line, where you know the thing you want to make, and there might be a finite number of ways that thing is made wrong. Maybe you produce batteries for an electric vehicle company. The defects aren't always the same, so it's a tricky problem.

Traditional, rules-based methods—look at this image, do some OpenCV, and if there's a deviation, then flag it—aren't always enough. Machine learning is going to be helpful because the defect could be a different length, or it could present itself differently. But the amount of variation that you're going to see in the cross-sectional scan of a battery is much more finite compared to driving on the open road.

The orders of magnitude of data that you need are different. Instead of talking about petabytes of video files—petabytes isn't sufficient for a car—you can probably get away with hundreds, frankly, hundreds of images in the case of a controlled environment to be able to produce something of utility.

The last part of what you mentioned is model size. So, yeah, the intuition here holds: the smaller the model, probably the faster it is, and also perhaps the less recall or precision it’s going to get. Maybe the way to think about this is the RF-DETR family of models, which is the current SOTA for doing real-time detection and segmentation. They come in nano, small, medium, large, XL, and 2XL. At the 2XL size, if you do a fine-tune, it is more accurate than if you fine-tune SAM 3, and 40× faster.

Of course, if you’re doing a fine-tune, you’re inherently saying, “I want this fixed class list.” So it’s a different prompt task type, right? It’s not open-vocabulary; it’s, “I know the things I want to see, and I want to know if those things are present or absent, or how many of them there are.” On the smaller side of the spectrum, you can get pico or nano models that are 180-plus frames per second on a Jetson Nano with 4 GB of RAM.

And again, based on the difficulty of the problem, if you’re doing something simple, like seeing oranges on a rack in a grocery store, compared to finding particulate matter in an IV bag on a manufacturing line, you can probably get away with a smaller model that still clears the floor of business utility while being more compute-efficient and delivering the results that you need. And I actually think “get away with” is probably the wrong framing. It’s actually an optimization. That might be your most optimal strategy because you’re able to deploy it at higher scale with perhaps less compute.

So, if you walk along the curve, the good news is that I think the intuition holds here. It’s what you would expect: harder problem, more data, bigger model. All those things follow what your expectations probably would be.

You mentioned distillation from foundational open-source models as a way that people are bootstrapping their way into datasets and then, obviously, fine-tuning downstream of that. And you mentioned this RF-DETR model, which—the RF stands for Roboflow, right? Maybe, if I understand correctly, there’s some—I don’t know if it’s on that model in particular or in other places—you’ve partnered with Meta. I’m interested to hear the story, and I’m also interested to hear the lay of the land in terms of who is producing the open-source models, why, and how.

In language, of course, there’s been a lot of talk lately about Chinese companies distilling from Claude, et cetera, and Anthropic trying to shut them down, which I think there are obvious business reasons for. There are also questions around what that means about the real strength that the Chinese companies have in terms of making their own models. How much should we discount what they’re able to produce? Does that explain why they’re so spiky?

I kind of wonder to what degree this is also happening, or understood to be happening—I’m not sure anybody really knows—on the vision side. I would say that, aside from Meta, it sure seems like the public perception is that the Chinese companies are leading in both vision tasks and also in image and video generation. Maybe they’re not fully leading, but they’re certainly category-leading in open-source image and video generation. I’m wondering: is there some distillation going on there, where they’re taking a shortcut, or do they just have tremendously better chops in that area?

If Meta were to have a change in strategy—and, obviously, they have had some changes in leadership—would the American side be kind of an empty bench? Are these projects so big that you guys, as Roboflow, could still dig deep and fund them on your own, or do you need a hyperscaler partner like Meta to really get to the scale that you need? I guess that’s the strategic and geopolitical question, and then you can dig into the partnership with Meta, too.

What should we expect? Are we just going to continue to get these great open-source models, like manna from heaven, or is that maybe more precarious or more of a moment in time than people may appreciate?

I think open-source AI—there are reasons to be concerned about its future relative to its past in terms of the number of open-source models we’re going to get. However, there are a lot of things that give optimism, too. In vision in particular, you bring up something that I think is under-discussed: in visual AI in particular, the US has almost never led, whereas in language, we have consistently been ahead, from closed models and open models alike.

There are a lot of reasons for that geopolitically, just as much as in terms of task emphasis and execution. Everything from the importance of manufacturing and vision in manufacturing to the importance of manufacturing in the Chinese economy—these are all trends that tell you why focusing on visual understanding as a domain is probably a high priority.

But, to name names of the folks that I think are in that mix, it’s the Alibaba Qwen team, who have done phenomenal work. They recently had Qwen-VL in the initial Qwen models. Qwen3-VL is world-class and competitive even with closed models in its vision-language reasoning and scene-understanding capability. The Qwen team has also had some recent leadership changes, or changes among leading researchers, so that might be tenuous.

The GLM team—we were talking earlier about their mixture-of-experts model, especially for OCR-specific tasks—has done impressive work, even surpassing what’s possible with closed models with its 9-billion-parameter model. And then there’s the DeepSeek team. If you remember, they published an OCR paper where the innovation was actually a data-processing technique for LLMs: they gave a model a screenshot of a page as a way to get more tokens versus just each individual word in the form of readable text.

The realization was that the degradation in understanding was much less than the compression that was achieved, so it was basically a way to give more tokens to scale up training. In the US, we’re not without folks doing incredible open-source work. You mentioned Meta, which publishes the Segment Anything family of models. I’d say SAM 3 is the best open-vocabulary model globally, and Meta is the publisher of it.

The Meta team, all the way back to FAIR and Yann LeCun starting efforts with Detectron2 and Faster R-CNN, and introducing DETR and the DINO family of models, has consistently advanced computer vision. One thing that people dunk on Meta about is their lack of language models. Again, I under-credit how good Meta has consistently been at visual AI in particular and at advancing computer vision.

If you think about their business, this makes sense as well: making sense of photos and images that people share on social, just as much as the future of glasses and so forth. You also have Microsoft with the Phi family of models, which are multimodal. The Allen Institute has Molmo, though if you want to talk about funding and turbulence, that’s a topical thing that’s taken place in the last little bit.

On the diffusion side, you have Mistral doing some work, and Black Forest Labs out of Europe. I think the other one that’s pretty exciting is NVIDIA. NVIDIA has put a ton of effort and investment into open-source AI. I think they have the most open-source model repositories now, just by count, if that’s your rough heuristic.

The Nemotron family of models and Cosmos Reason—I was talking with one of their directors of open source, Nader, recently about how much they’re investing in making those models increasingly multimodal. The Cosmos Reasoning team is doing great work to advance beyond just visual reasoning capability. So, there is this geopolitical race for sure, with everyone wanting to have the best models possible.

Now, you asked the question that is near and dear to me: where does Roboflow fit in this? You’re not a foundation-model company, you don’t have a data center the size of Manhattan like Meta, and you’re not the producer of GPUs like NVIDIA. So, where do you fit in this mix? This is actually something that gives me an intense amount of pride.

We published RF-DETR, and RF-DETR retook the state of the art for us in a very specific area of important tasks: real-time object detection and real-time instance segmentation. Before that, you had models like LW-DETR and the D-FINE family of models, which were tougher to fine-tune. Both were great work out of labs in China.

RF-DETR is the first real-time instance-segmentation transformer, as well as the fastest and most accurate for doing pixel-wise segmentation and detection. The bet we made, which initially I wasn’t sure would work and I’ve been delighted at how well it has worked, was to pick a narrow task and a small model that is useful on edge tasks, which, just as you and I have been discussing, I think is comparatively under-addressed.

So, we basically have this novel area where we know people need things on the edge, we know people want models to be theirs, we know that open-source AI is under attack, and we know that it’s incredibly important to give people models that can run in environments that they might not otherwise have. The way we did that is that it marries these themes.

We took a DINOv2 backbone, so a pretraining from the Meta family of models. They’ve since released DINOv3, but we used a DINOv2 backbone. We noticed that there had been improvements from models in the transformer family in accuracy, but not speed for detection-type tasks. Similarly, there were some transformer models that were faster, but not more accurate.

So we said: if we use a DINOv2 backbone, use all the benefits of pretraining, and use a shared-weights neural architecture search, or NAS, strategy, can we intelligently search for and find the most optimal speed-accuracy model from an Objects365 pretraining that then works downstream on COCO and user fine-tuning tasks, and attach a segmentation head and a detection head?

At the time when we started these experiments, it was soon after we'd raised our Series B. In total, we've raised about 63 million across all rounds, just to give you a sense of the size of the resources we have available at our disposal. So, not nothing, but also not all being spent on just this problem, of course. It also pales in comparison, perhaps, to the billions that can be spent on foundation models. Through the training runs, we realized that this technique had promise, and so we invested further in it.

We introduced the first detection model last April, the segmentation model in the fall, and we continue to invest in making the developer experience really high quality there. Critically, something I'm super proud of is that it's Apache 2.0, which means that even though there have been YOLO models that we support and folks can use, those are now not commercially permissible without a commercial license, which we're able to offer, which is awesome as a company. But I think there are places where people just want to build models that maybe they don't necessarily have commercial ambitions.

It's world-class at what it does. To give a sneak preview, we already know some ways that it can extend to perhaps other task types and even have yet more accuracy. The LW-DETR team in China has responded, but hasn't beaten back some of our work. It's this cool kind of global arms race where your tiny friends at Roboflow are putting the U.S. on the map in a pretty big way.

If Meta were to publish open source tomorrow, if NVIDIA started publishing open source tomorrow, just as I described to you, all of open source would take a hit. A lot of improvements come from taking the best ideas, experimenting, running ablations, smashing them together, and having smart minds—certainly smarter than me—think about how to solve these outstanding problems. So, I think it tells the story of what's going on in open-source vision, and it's something we're proud of just as much as the problems that are yet to be done.

Well, yeah, I'm interested to hear more. If I were going to do one double-click there, we're obviously entering, according to many—and I'm among them—the era of recursive self-improvement broadly, with AIs doing AI research. It sounds like you dabbled in that a bit with this architecture search. I wonder if, as you look back on that experience, you found anything surprising. Did it feel like a brute-force grind, or are there stories to tell about Eureka moments coming out of that architecture search that felt somehow qualitatively different from a brute-force grind through architecture space?

One thing that I think is really exciting is—I’ll go deeper on this idea of weight sharing in neural architecture search. A lot of the time, you're doing a very brute-force process: train a bunch of different models, compare the speed and accuracy of those models, and you're almost doing a grid search of different parameters that could help. An informed grid search, right? You're not going to do things that you would think are naive, but it is fairly naive guess-and-check: train this model, see its speed, evaluate it back and forth. There's still, of course, a degree of that.

We published a paper, so the details here are open for anyone to dive into as well. What we did with weight sharing in neural architecture search is, rather than train a separate model for every accuracy-latency configuration, we use weight sharing in NAS to basically train thousands of subnetwork configurations in parallel with a single training run. At each training step, one subnetwork is sampled by randomly checking parameters like patch size, the number of decoders, the number of queries, the input resolution, and the attention windowing. We use deformable attention in the model. At inference time, you can actually sample any of those subnets.

What that does is it doesn't just mean that we've introduced maybe one model. We've actually introduced a framework by which we can repeatedly produce open-source models as long as you can do NAS against the architecture. A NAS training run isn't as efficient as a single training run, but it's also not 7,000 times more inefficient despite having the ability to compare all of the different configurations. That was a huge freaking unlock for us, allowing us to use our compute budget efficiently and release models like this. That's one huge part of the story.

The other notable unlock was rewriting the deformable attention, which isn't supported in every inference engine. We've had to rewrite some support for it or wait, for example, for TensorRT in NVIDIA ecosystems to support it, and now it does. But that was a useful realization. I mentioned the DINOv2 backbone, and now DINOv3 is out—you can imagine what experiments we're running. So, yeah, the weight sharing in NAS is massive.

By the way, anyone can use NAS on their own dataset. No one has created a one-of-one model for your problem. The way NAS works is that it's going to train, create, and output a Pareto frontier that you can then pick from: where do I want to exist along the speed-accuracy trade-off? You can be anywhere along that curve within your available compute budget, and you can obviously just maximize accuracy or be lesser on speed.

When we saw that NAS worked on Objects365, we were interested in whether it worked on downstream tasks. Now we've actually rolled out the ability to run GPUs in the cloud that will do hosted NAS on any given dataset. To the theme of owning your own AI, if you NAS on your dataset, literally no other model architecture exists that is tailored to your dataset. There's some interesting crypto thing there of a one-of-one to give somebody, but that's outside my wheelhouse for sure.

It is the purest form of your model, because literally no other model would have landed on those optimizations for the dataset that someone wanted to train on. So NAS was the unsung hero that was a huge unlock for the efficiency gains we were able to see.

Yeah, that's really cool. One thing I noticed you have done—I don't know if you've done it exactly for this yet—but when people hear about this whole NAS, it sounds complicated, right? We're going to have a Pareto frontier's worth of models. I would imagine, channeling myself—I guess I'll just speak for myself—that sounds both awesome and complicated.

But I notice that for at least some things, you are following the trend that I'm seeing everywhere these days: here's a skill that you can just give to Claude Code and have it speed-run through the process of setting this up for you. So I'm interested in how easy it is these days to get started. If I'm sitting on some esoteric problem and maybe some small amount of data, and I'm thinking, “This Roboflow guy sounds like he's got some pretty cool techniques,” what's the path of least resistance to come out the other end of this tunnel, potentially not having done much work, and have my own one-of-one model that's got its own Pareto frontier of possibility and all that good stuff?

For NAS specifically, I give you the real stuff because I think the audience would want to dive deeper, and I'm a skeptical person myself, so I'm like, “Give me the real, real info. What's going on under the hood?” That's why I mentioned the paper is out there as well.

But as someone who builds products, we also have—I created the easy button where it's “Run NAS on my dataset,” and then, boom, we spin up a bunch of subnets on GPUs, kick off the training job, and show the results. What comes back for the user is their Pareto curve, and it's a 1-click press where you want to be along that curve. That's for the human user.

You mentioned the agent user, which is also an interesting place to spend some time. But the first thing I would note is that a core thesis of Roboflow—why we build things the way we do and how we approach things—is that you want to be very interoperable and allow someone to progressively reveal complexity, but set good defaults.

You can almost think about the products that we build that wrap RF-DETR, wrap our inference server, or use NAS. By all means, someone could set up their own infrastructure to do training and reimplement NAS, and it's all out there. It's open. The thesis is that by making it easier and simpler, you actually inspire and engender trust: that the benchmarks can be reproduced and that folks know where things come from.

Ease of use as a guiding philosophy means strong defaults can be set. I can also have a model trained for me or a dataset that gets curated. On the inference side, which we haven't spent even a ton of time talking about, we've made tons of investments into vision-specific inference. There are a lot of optimizations and assumptions you can make to use the GPU most efficiently for vision—for just the parts of the network that require it—versus, for example, a resize, where you can run that on a CPU.

There's tons of it, and all of that's open. Inference, if you pip install inference, is an open-source GitHub repository anyone can use. Also, if we provide that as good defaults in the service, and if we're worth our salt at all, then we should make products that are easy to use. That's for the human user.

Hm. Now, you mentioned something that's really exciting in general, that's certainly bigger than any one company, and that is agents becoming the biggest user. And what does that look like?

Like a lot of companies, we're leaning into the idea that if you expose CLIs—and maybe MCP; we might release a Workflow-specific MCP yet—certainly, we have lots of good CLIs, and there's this ongoing debate: is it an MCP or CLI future? At a minimum, there are CLIs for all the actions that take place in the platform that Claude Code, Codex, and so forth can take on behalf of a user, so that you can say, “Go optimize a model for me and make that easier for me.”

Now, something that we are investing in as well is a common trend you've seen among companies that build amazing infrastructure products. Here, I'll give a lot of credit to Vercel, where they've done an awesome job of making a common set of products for the front-end cloud and Chris [?] in the back-end cloud. And then they've layered on top of that a v0 agent, right? Their product that you can chat with, and it'll build a website and choose good infrastructure for your problem.

We've taken a lot of inspiration from that as a way to enable users similarly to chat with our Workflows AI agent. It's like, “Hey, I just want to say, count people crossing the line, or watch cars in the intersection, or whatever it is.” And with a lot of these problems, it's interesting: the hardest part is actually discerning what the user wants—what their intent is. Once you have a sense of what the intent is, then models can intelligently say, “Okay, you wanted to count cars. Nathan wanted to look at cars crossing the intersection, so is there a model that already knows cars? Probably SAM 3. Actually, cars are a class, probably RF-DETR, and it'll be more efficient, faster, and more compute-efficient. Great, let's grab that.”

And a pretrained model knows cars. “Okay, you said crossing the line. Can I ask the user? The intersection has multiple places you could have meant by ‘cross,’ so what intersection did you mean?” Increasingly, you can be in this future where it's like, if Joseph were sitting down with you side by side, helping you construct your problem, pick the model, and follow the architecture, can we expose that as an agent in a democratized way, for lack of a better term, so that anyone has access to it?

Folks who spend many hours a week thinking about these problems can be the guide and sherpa for building a given pipeline. In the scheme of things, it's ease of use with good defaults as a platform principle, with complexity revealed progressively. Secondly, it's agents using CLIs to basically use the same sort of easy-to-use stuff. And the third is a first-party agent, which we haven't released yet, but maybe by the time this comes out, folks will discover it, to guide them down that journey.

Those are some of the ways that I like to think about building products that balance giving someone the satisfaction and awareness that it's built on good primitives, while still being able to create products that are easy to use and allow folks to get to value quickly without needing to know everything about every subnet of a training run, for example.

A very particular question that comes to mind, that you might help me with—or maybe set my expectations on a little bit—is this: My company, Waymark, makes videos for small businesses, specifically focused on TV-quality advertising. A classic 30-second TV spot is really our bread and butter.

Customers have asked us from time to time, “Hey, could you help us with display ads, too?” Our customers—we partner with a lot of cable companies and media companies—are ultimately selling advertising inventory to small businesses. We're helping enable that with the creative solution.

They ask us about display ads as well, and now we're getting to the point where maybe we could add that. We can vibe-code all kinds of stuff much faster than we used to, certainly. But one challenge that I used to have a lot—and I'm not sure what its state will be today, as I'm digging into this—is aesthetic evaluation.

Way back in the day, there was really just one open-source model or one open-source dataset and a couple of open-source models that were trained on it, which seemed to do a halfway decent job of aesthetics. By halfway decent, I mean I could tell which was at the top and which was at the bottom of the distribution, but in the middle, it was very unclear which way I was headed a lot of the time.

Then there was one company that had one, too. I forget—oh gosh, what was the name of that? Everypixel, I think it was maybe called. These days, we typically go to foundation models for that, and we say, “What's suitable? What would make the business proud? How would you advise us, basically, on these available images to use?”

They work pretty well—definitely slow, definitely more than we'd like to spend in many cases to grind through a huge library of images that a small business might have. Is there anything in the small open-source world that would be able to tackle a problem like that, or is that still so esoteric that nobody's gotten around to building that foundation for me?

Aesthetics is a tough one for the reasons you described. The types of problems that models can recursively improve against are the ones you can benchmark. The second you can benchmark it, then you can scale a bunch of compute, and the bitter lesson takes hold. Aesthetics may be a little bit in the eye of the beholder: what's good, what's bad?

There are some places where, even if you just take diffusion models, some people like the way Midjourney looks more than they like the way ChatGPT looks, more than they like the way Gemini looks—or Imagen, I should say, when it creates examples. Maybe the model you're talking about is the LAION team's aesthetics predictor model that helped evaluate some of these things, because they also did some generative image stuff. They also released their aesthetics evaluator.

Mhm. That wasn't out when I was first really struggling with this problem. The timing was such that we had already moved to foundation models, but that was definitely the best purpose-built thing I think I've still seen to this day.

One thing that I'm sure you're aware of, and that your audience might find useful as a way to reason about this, is that in the context of display ads, there are some services—Facebook, for example—where you're not allowed to have text be more than X% of the display ad. They find that it just reduces the quality of the ad for the end user, whatever the reasons are.

For sure. Of course, that's a great example of the distinction between “Does this ad feel good, taste-wise?” versus rules-based: is there too much of this image that's text?

Then automation of taste, aesthetics, and preference—I think it's a great RLHF problem. If you have a given client, you know their brand guidelines and their style, and perhaps there's enough history of display ads they've run where you can get almost like a vibe-check model that has been tuned for what they've done.

With foundation models, like you said, perhaps you can do a few-shot approach: “These are the ways that this person commonly likes to do things. Is it similar?” Again, the big problem even with that approach is that so much of marketing is about being different. If you're adhering to the brand guidelines, you might be stylistically following what you should have done, but you might be failing the top-order task, which is to stand out from the noise.

Short answer: I don't have a great zero-shot aesthetics model for you beyond, I think, the things you're probably already doing. But the longer answer is that I think it's a great example of the conversation you and I have been having about what distinguishes a task where you can train your way—post-train your way—to victory with objective metrics, versus ones where it's a little more loosey-goosey to benchmark and therefore live outside the range where tossing compute at it gets better results.

Okay, so moving to the time we have left, let's talk about just frontiers—what's coming next in any number of different directions. There are, of course, new architectures that people tend to get excited about, myself included. Things like Mamba and state-space models more generally. At one point in time, there was an explosion of vision use cases there.

World models are obviously a big deal. I'm not really sure how to think about how they will relate to vision. We've got increasingly credible claims that people are going to start to scale up humanoid robots and put those into presumably factories first, but then businesses and homes not too far into the future, either.

What are the things that you're most excited about? What are the biggest questions that you have where you're kind of like, “If this works, it's going to be a game-changer, but I'm not sure if it's going to”? We're all about scanning horizons here. What are the horizons you are scanning?

There are things that I think are a continuation of trends that are working, and then some newer S-curves that we're starting to ride. Trends that we're continuing to ride are Transformers everywhere.

We talked about how the Vision Transformer was in 2020, and Attention Is All You Need was in 2017. So you've seen diffusion transformers—DiTs—and vision transformers—ViTs—continue to eat more and more tasks and achieve state-of-the-art accuracy.

RF-DETR is exactly that recipe applied to real time. That trend is known and going to continue. Another trend that's maybe more nascent is self-supervision, especially in the DINO family of models. DINOv3 kind of showed that you could have good latent understanding of things as a backbone without having large amounts of supervised, labeled data, and then you can use that image understanding downstream for tasks, whether that's detection, segmentation, captioning, or whatever.

Can you tell what the unsupervised trick is there? I always like to tell people the big unlock for language was that language itself is structured. If you just have a ton of language—predicting, given some text, what comes next—we've got lots for you to work with. Similarly with CLIP, right? It turns out there were billions of captioned images. What is the unsupervised unlock for DINO?

Okay, so in the DINO family of models, you have DINO, DINOv2, and DINOv3, and they're all riding on this trend of self-supervision. The DINOv3 model, I think, was trained at billion scale. I have to check the exact statistic, but I remember seeing that it was similar to the number of images on Roboflow Universe. I was like, “Huh, there's something there.”

So, with billion-scale images, the observation is that if you start to have a sufficient representation of given domains, then maybe, intuitively, if you think about a human, without being told what things are, you start to develop intuition for where and how structure should exist in a given scene. And that is understanding. If you know that the lamp is on top of the side table and that it's often in a bedroom, then you have an understanding of a given scene, and you can use that understanding again. It's a backbone, so you can attach a classification head to DINOv3. You can attach a segmentation head to DINOv3. But alone, it's just a backbone that has really rich latent understanding of scenes.

That's the unlock. If you think about just looking at a bunch of scenes, you're going to start to develop your own understanding. Maybe pattern matching is a crude way to think about it.

To be a bit more specific, does that involve some sort of masking-type thing? How is it creating a prediction task for itself that nobody needed to label data for?

In training, there are papers, so fortunately we can falsify and understand them. They use self-supervision techniques. Typically, you take a student-teacher model, and you have a bigger model that's the teacher that validates the output of the student. As you see the student continue to do well at predicting either patches or using Gram anchoring, you continue to scale up the student-teacher training recipe to larger amounts of data to understand more scenes.

The way that understanding happens actually isn't that dissimilar from the vision transformer itself, where you have patches. It's actually crazy that this works. Literally, these models—and there are different approaches—take patches of the image, and it feels very unintuitive. It still feels a little unintuitive to me. But if you have patches of an image, it's almost as though you can understand the rest of the image from individual patches, even if you treat those patches independently.

It reminds me—way back in the day, I used to do language stuff—of bag-of-words, where you would have a document and count the number of times each word occurs in a document, and you can start to get a sense of what that document is about. Is the same thing happening with understanding patches of a given image?

There are other techniques that use cross-attention and get a better understanding of where things are in a given image with respect to one another in the overall image. That's also why, by the way, earlier we were talking about the struggles of spatial reasoning. But the core unlock is that if you have a high number of images of various scenes and you run verifiable, falsifiable tasks—fill-in-the-blank, “What else would you expect to be here?”—or diffusion generation, and then you have a teacher that's able to validate the student's work, you have the recipe for a self-supervised loop. You can plug in more data and scale up.

That's what they did. They didn't release the dataset, but it was billion-plus—I need to check that. I think a billion-plus images were in the DINOv3 pretraining. It's actually really cool that that works, honestly, and that it's open and there's a good technical report for it.

Okay, sorry to take you down that rabbit hole. Let's pop back up to just more horizon scanning.

Yeah. What about JEPA-type things? Those are always hotly debated as to whether they're the inspired future that few can understand or if they're kind of beside the point. I still don't know where I come down on that myself, but feel free to opine on that or any other. What I'm most interested in is what horizons you think are the most important ones to be watching.

We were talking about ones that we're already riding—the known S-curve of transformer examples, self-supervision, and how patch embeddings work to create understanding. New S-curves that I'm excited we, as a society, are collectively starting to ride: one is world models.

Within that category, there are various techniques, like the V-JEPA technique and the World Labs techniques. The idea of a world model—and there are different labs with different approaches—but the underlying objective of a world model is: Can we understand and reason about scenes, places, and existence out in the world with a new architecture? If you think about that, it's—what's new? What's different? You're inherently multimodal by default.

Some models will think about this as next-scene prediction from video: Given that I've entered this, what's the next scene? Some approaches predict the next scene. Some will think about it a bit more like diffusion of a single viewpoint.

The so-what for world models that I'm interested in is whether world models give us true understanding—with physics, with spatial reasoning, with open-ended tasks—that we can just start to use. The answer is probably yes, but the more interesting answer is over what time horizon, and there I'm not sure.

I think what's interesting right now is that we're using world models. I would argue that maybe even Cosmos Reason is an example of a world model. You can use Cosmos Reason, at a minimum, for boring things like synthetic data generation, and at a maximum, perhaps you can use it to reason about something and navigate a given space.

So, world models are one category. I think about vision problems as read-write access, and I think about Roboflow, for what it's worth, as mostly read access. World models are a form of blending read-write access. Robotics is an example of write access to the real world. You are modifying and manipulating the real world with a robot.

Of course, that requires understanding. You have to have read access to have write access in a scene. World models are exciting because they're an example of potentially blending those two. Potentially, you get the understanding zero-shot or multi-shot. You could even argue that some of Sora 2 was the underpinnings of a world model to understand what's taking place. That's one trend.

To give an overview of another trend—and we can certainly go deeper into some of these, too, that we haven't mentioned—there are VLAs, vision-language-action models, which are incredibly popular in robotics. A vision-language-action task is that you provide an instruction, and a robot is typically able to act on that instruction. “Move my computer 50 cm to the left,” for example, might be an instruction you provide, and then a VLA would be able to act on that.

In that world, you have a number of emergent, younger startups that are thinking about this. You have NVIDIA's GR00T project working on it. Google's RT-2 is working on that set of VLA problems. Maybe another way to think about VLAs is that it's a new task type, a new paradigm, and we should expect the same things you and I were just discussing around different model sizes and different levels of generalizability.

VLAs, in some ways, will need to be edge-ready, because if you're going to run on an embedded device and have embedded intelligence, then you're going to need the thing to be at the edge and run in real time. I think VLAs are an emergent and exciting trend and perhaps still underindexed.

This isn't unique to language or unique to vision—it happens in language, too—but it's worth describing because, in visual understanding, there's inference-time scaling and reasoning in general. In a lot of ways, vision can be a tool call of a broader agentic system that wants to understand and describe how to do stuff in a scene.

For me, I find that I'm using Gemini 4 all the time as a replacement for instruction manuals. What does this button on the remote do? The pilot light in my water heater went out recently, and I'm like, “Okay, tell me about this specific model.” That's somewhat of a high-stakes task that one would want to proceed with caution.

I grew up as the son of a farmer, so if I wouldn't be allowed to figure out how to do that, I would probably be exiled from the will. Fortunately, my friends Gemini and ChatGPT and I were able to solve the problem. But that's a perfect example of where I'm using visual reasoning in the real world, but interacting with it through language.

In the post-training reasoning, there's probably a tool call there to do some search, figure out the water heater model, and figure out the instructions you're going to provide. All of that is in the category of having a big compute budget, and being able to do post-training and inference-time scaling to give better results. That's just going to continue. That's just getting going.

You can start to think about that as giving rise to visual agents. You can set one off to go do a task for you: organize my images for me, or figure out, in your case, perhaps, if there's one that's aesthetically able to do good categorization of things that are brand-aligned and not brand-aligned across categories of display ads that you want to do.

We can learn from coding agents. When you can let something run unencumbered over a long duration with a model, then we'll get similar benefits from long-running vision agents that can understand scenes and do things for us, with all the caveats of speed and latency included. Those are some of the trends—the hype-y ones—that I'm thinking about and paying attention to.

Again, what I try to do when I break these down is break them down into my normal-distribution bell curve of the world and figure out what's the impact, what's the implication, where can people use them, can they make them their own, and where they're going to be most useful. Broadly, I think the recording time of this episode is well timed because the vibes—the pendulum is swinging back to vision. You hear the rise of physical AI, of multimodality, the rise of hardware, and what's defensible in a world of SaaS always being rewritten and code generation being simpler.

That's putting more and more people into thinking about the real world and hardware, and ultimately cameras and getting things into those environments. To me, welcome—the water's warm. I've been here all along, the infrastructure's hot, so let it rip, and we're fortunate to be able to power a lot of that sort of stuff. Hype-wise, it's something that has me pretty excited about the amount of activity that's about to enter the space.

Those are some trends and maybe themes to track that I'm looking at. How about wearables as another one? That seems to bring a lot of these challenges together, right? Because if you're going to have something on your face, it can't be too heavy, it can't get too hot, but it has to understand what's going on around you well, or it's more annoying than it's valuable, right?

Totally. We started a partnership with Meta for the Segment Anything Model family, and now more general visual understanding. For example, when they launch SAM models, they're on Roboflow with day-one support, and now we're helping them understand where the model can be improved and where it can't, and doing some work like that.

For Christmas, my significant other got me the Oakley Metas. She was like, "If you're doing this awesome Meta work, you have to be dogfooding their stuff." It's my first pair of wearables that are mine. I've used ones like Spectacles, and I'm always tinkering with stuff. When the Apple Vision Pro came out, of course I gave that a run.

I'm pleasantly surprised. Wearables are going to inflect. There were 8 million pairs sold last year. By way of comparison, 60 million AirPods were sold, so a pretty good amount of volume moved. The Oakley ones in particular are targeted at active activities. I like to cycle, and you typically already have a pair of sunglasses on your face that are a bit bigger for cycling. I like to run.

They do bone conduction for music, and they understand the scene. The AI on board is not there. You have to have your phone with you, and they're offloading, presumably, some amount of the heavy lifting to the phone. You can say, "Hey, Meta," and then get some of the feedback. But again, like with many things in AI, it's the famous expression: this is the worst it'll ever be.

Now that it's useful enough to be in a form factor where this is a pair of glasses, I went on a ride Sunday with some friends. They didn't even know that they were glasses that had the ability to play music and capture media. Then they gave them a spin for the first time and were like, "Man, we need to get these for our next ride." That really gave me the sense that this technology has arrived.

The constraints are running on the edge and constraining the amount of power draw it's going to have. What's funny is that I'm a bit like Charlie Brown running up to his football and swinging and missing on AR. Roboflow actually started as building AR apps. Before we even had a company, in 2017, we made AR apps just for fun. I was like, "Oh, man, we've arrived." How wrong was I on the timing of that?

Then we came back at it in 2019 and made more AR apps. I think the big unlock is the form factor. You don't have to have the glass brick in your hand; you can have a different thing. So I'm pretty excited about wearables.

Snap also has its Spectacles. They were the first publicly traded company to mention Roboflow in an earnings statement, so they always have a special place for me. We did an integration with their Snap Spectacles for developers to create custom lenses for scenes you want to understand.

Actually, we had someone count the number of stop signs on their walk. It was a funny thing because they wanted to, I think, prove to their neighborhood that they were safe or something. There's something brewing there. The big change, certainly, is that the hardware has gotten good enough, and the consumer willingness to adopt is showing up in the numbers.

I hope—I think this will happen—I hope that ecosystem stays open, or becomes more open, I should say, so that anyone can publish apps. I don't have any inside information here, but based on what I shared, I would bet the strategy is that right now it's closed APIs because you want to curate the experience and have a high-quality first-user experience with the apps people can use.

But I would bet that the strategy will be to open that up, app-store-like, or maybe even Android-like, where anyone can sideload. I'm excited for that future, but I think the hardware-platform adoption precedes the software adoption, and we're just now starting the S-curve of the hardware adoption of wearables.

So if you had to zoom out—this is a big ask—but if you had to zoom out from all these various horizons that we've just been scanning and try to tell a story of how vision impacts life in general over the next few years, how do you think life changes? I do think it's hard to predict anything more than a few years out at this point.

Are we all going around with always-on cameras? Is that normalized? Do we all have a sort of 24/7 retrospective video of our lives, maybe subject to some times when we choose to pause it? Are there other unexpected changes to life that happen as these technologies get deployed that you think people are sleeping on?

I think AI's going to change everything, to a first approximation. But I'd be interested to hear your take on what particular role vision is going to play in that and how it will feel as we are actually living it.

Man, I would love to paint the optimistic future for you of what vision unlocks for us, step by step through everyone's day. From the moment you wake up, you have food that's been produced with higher quality and fewer pesticides because you didn't have to spray all parts of the field; you only had to spray where you saw weeds.

Maybe you had eggs for breakfast, and you want those eggs to have been visually assured to be safe, from the hens and all the way through the supply chain to your house and your grocery store. Maybe you grab your clothes out of the washer and dryer, which for some reason you still have to say "whites" or "colors," which is very obviously a silly vision problem ripe for the taking.

Your fridge auto-stocks itself because it saw you were low on eggs in the first place, and so you didn't even have to go and call the Instacart MCP. You automatically have the food in the fridge. You take your self-driving car to work.

There are zero accidents because all the cars are communicating with one another, and it's faster than you've ever been able to go because you're able not to worry about the unpredictability of someone else's actions, with networked systems talking to each other at a car intersection. You have Wi-Fi along the way, so you're able to spend more time with your family because your workday already started on the way to the office itself.

You're in the office, communicating with colleagues all across the globe, and you have perfect, pixel-perfect fusion representations of them in the room next to you. Remote work, same work, same place. It's just all the same in terms of what it feels like to collaborate, at least digitally.

There's going to be something in human connection still, but at least Zoom's representation has taken leaps and bounds. It's orders of magnitude better for meeting with other people. Then, I don't know, you get home that night, and you watch Thursday Night Football.

The stats are real time, and your fantasy team wins because you have the best algorithms to know who was going to play and who was going to score. You had your vision agent running in the background to do that better and faster than your friends.

You have a package that showed up at the right time—in fact, the same day—because all the vision systems in the factory and inventory made sure the product was made right and checked in at the right places. There was a bot that delivered it to your door, so it wasn't strewn about, and your Ring camera made sure there was no porch theft or whatever that might take it while you waited for it to arrive. This goes all the way to the moment you brush your teeth with a smart, silly but AI-enabled camera that's also doing cavity scans and making sure everything is right in your mouth when you go to bed. This future is not theoretical; every part of that chain involves things that Roboflow customers are working on in all parts of it.

Now, to give you something that's top of mind for folks with always-on cameras and what society is going to feel comfortable with, I want to give you some of my direct thoughts there as well. I think the transition—even now, early on in smartphone territory—made people uncomfortable because people always had cameras that could capture moments without others being aware that photos of them could have been captured. Frankly, even still, that's a real consideration: in public spaces, capturing photos or not capturing photos.

Over time, society—which is ultimately the judge of this—will determine whether it's willing to accept the new societal behavior based on whether the increase in quality of life is going to be better. I would take the bet that yes, because it will start with simple things. Think about my riding with my cycling glasses. They don't have a heads-up display yet, but I'd love to have turn-by-turn directions, and then pretty soon I'm used to having that small little display, and other folks are interested in that.

I do hope that to build technology companies, you have to inherently be optimistic because you're giving tools to people, and that means the tool we use is a reflection of what you think about humanity. So if you think humanity is inherently good, then you're able to amplify those attributes, and I do think humanity is inherently good, even if there are bad actors. I think the same thing will be true for glasses and consent. You can use prior technologies in pretty icky ways.

The internet can be used to communicate with friends or support a small business online just as much as it can be used to share photos that shouldn't be shared. The same thing could be true of the next generation of technologies. I have optimism that the benefits will continue to be things that folks will want to adopt.

The great news is, frankly, it's not up to me. It's like a jury of our peers deciding where it's going to be useful and not useful. On the governance front, I also think it's important that we have systems, society, and institutions that exist to govern the use of these things in a way that reflects the preferences of people around us. There's a reason privacy rights should continue to be strongly enforced and apply as the times change.

Search and seizure was written well before the existence of cars and homes, so what defines unlawful access and entry? We should have the same sort of means-tested laws applied to new technologies to define what is private and not private, public spaces and private spaces. Again, I remain optimistic that the principles we hold dear around having a right to privacy and a right to use things the way we want to, at least in the country in which I live, are going to be the way that future products are used and governed.

So that's how I think about it as a participant in the system, just as much as someone who enables this future. But the good news is, man, the world is going to get so much better. We have folks who are accelerating cancer research, cleaning up the world's oceans, removing pesticides from foods that we might produce, ensuring that electric vehicles are produced correctly, and ensuring that stuff shows up at the same time. I like to joke that we'll be able to power Santa Claus.

That future is happening now, and so I think those are all things where it won't be without bad actors and its own set of messy, sticky issues. There will be that case, that front-page story, and we as a society will need to respond and ensure the frameworks and rights that we hold dear continue to be in place, even as the tools that we have evolve and continue to be there. So that's how I think about that, and I think we have a responsibility to ensure that the future we want to live in is one that we help foster.

In a lot of ways, I like to show examples of vision and our results where the vast majority of things are all about improving quality of life, not about some of the bad implications or bad actors that folks might sometimes be concerned about. Those are some long-range thoughts, but a lot of folks ask me that question when building the company that we get to build. Hopefully that gives you some color on how we've thought about it.

Yeah, that's great. That could be a good place to leave it. If I was going to ask one more follow-up question, because I sometimes can't help myself, it would be: do you think there are technical solutions or rules that we could define in terms of technology properties that would really help?

Here I'm thinking about how we've had a lot of this discussion about very general-purpose models versus very specific models, and I am increasingly struck by this notion of narrow safety through narrowness, basically. From a bunch of different angles, I'm wondering right now whether there is a social contract to be had around AI: We want to, and we need to, and we all stand to benefit tremendously from solving very particular problems. But we also put ourselves at risk, perhaps, if we use fully general models everywhere to try to solve all these relatively narrow problems.

In the vision context, one example that I could imagine is, if you want to watch a public space for moments of violence or whatever, you could run that through a general-purpose model that can tell you anything. I think in some places we're identifying individuals by their gait and their facial structure and whatnot. But an alternative would be: let's have a very narrow violence detector model that doesn't really do much except sound an alarm when it has detected something that we want a higher-order response to.

I wonder if you have any thoughts on that. I don't know whether one could argue that maybe that sort of thing happens naturally, because cost and efficiency pull things in that direction. Somehow, I don't feel that comfortable betting on that, and I kind of think we might need a little bit more of a social contract or some sort of idea of a new right.

I'm always on the lookout for what new rights make sense in the AI world, and one of them might be to be classified by the smallest, narrowest, purpose-built model possible for the task at hand, as opposed to being processed by some general-purpose reasoner that could answer any and all questions about me. Anyway, I'd love to hear your thoughts on that.

Yeah, I spent some years in D.C. I was an intern in the Senate once upon a time. Thinking about some of the institutional questions that affect this stuff is something I've spent some time thinking about.

My general thought is that AI as a technology—and my firm belief in the importance and value of open source for freedom of use, discovery, and use-case proliferation—all stem from this idea of giving people the right to tinker, if you will, and use models where folks want to use them. What does AI change in terms of the societal rights that need to be in place? Where I come down is, I think that the outcomes that we want to have in society should continue to be enforced, and AI is a tool by which those outcomes can be realized or not realized.

In other words, to be really specific, we have regulations that prevent fraud. We have regulations that prevent forms of violence or regulate the actual outcome by which something happens. A scammer could use an LLM to make it really easy to impersonate someone else, and they should be prosecuted for committing a scam. They likely shouldn't be prosecuted for the size of model that they used. The model they were using was too big or too small for a given example or task.

So I think the idea of focusing on someone's minimally invasive use of a minimal model size gets into trouble because the capabilities advance quickly enough—or you have distillation—that you then end up with accidental corner cases where you cast too broad a net. That might stymie innovation and stymie adoption when, in fact, the goal was well-intended.

I'll give you another great example. One could very reasonably steelman the idea that AI in health care has such far-reaching implications that there ought to be some form of governance. If you're going to use AI for patient health, then you ought to have a governance body approve, inspect, or allow that type of use of AI in patient health. Someone could say, “That sounds like a very reasonable thought, a well-intended position.”

Then I think about users at Roboflow, like this user at UNC Chapel Hill who was using AI in their lab to automatically count the number of neutrophils that respond to a given experiment. Here you just have a lab postdoc student who's accelerating the rate at which they can experiment and doing the fairly menial, frustrating task of counting. There are hundreds of colonies of neutrophils that appear under this experiment, and the way the proteins react allows you to know if the experiment was good or bad and whether to do another round of treatment.

And all of a sudden, that person who's just using AI in a fairly harmless—in fact, quite useful—way would never endeavor to do that because it actually is touching patient health. And so you've put yourself in this accidental position where you've got something that's well-intended. I don't want to harm patient health, or I don't want to regulate a model of a given size or a given use case.

When in reality, probably the way to attach that is you should be liable if you use procedures or things that—there's plenty of this already in the medical system—you should be held accountable for practicing medicine correctly, in the way that ensures patient health is respected. So, I guess, to be succinct, I wouldn't think that a narrow model size nails the way that you and I probably would want this technology to unfold.

I do have optimism that types of regulations that inhibit misuse of any technology or of any behavior should be applied to AI, and that regulating at the tool level is ripe for accidental slowdown and deceleration of what I view to be the modern industrial revolution that's going to have consequential quality-of-life improvements in ways that we won't be able to fully forecast. And so, one of the best ways to do that is to let it flourish and stamp out the places where people engage in bad action.

So, that's a fairly general way. Of course, there are individual things to think about, but that's how I thought about where the field is today and where it shows a lot of promise.

Yeah, I think that makes a lot of sense as well. Another thing I obsess about all the time is: How do we avoid the nuclear outcome where we get all the weapons and don't get the energy? I'm certainly not wanting to stumble my way into that sort of scenario.

I think this has been great. Do you have anything else that I didn't ask you about that I should have, or anything else you want to leave people with before we break?

I don't think so. I really enjoyed the conversation. I appreciate the opportunity to chat about this, hear about some of the ways you've thought about visual AI, and almost get a refresh from CLIP in 2021 to visual AI in 2026. The rate at which this stuff moves means we could have a very different conversation 6 months from now about the same set of topics. So, it's just been fun to riff with you.

Likewise. Looking forward to it. Joseph Nelson, CEO of Roboflow, thank you for being part of The Cognitive Revolution.

Thanks for having me.