SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Okay, we're here in the remote studio with the grand return of the Roboflow, Latent Space, and SAM combo. Welcome to Joseph, my vision co-host, I guess.

Thanks. Great to be here.

Welcome back. We also have Nikhila Ravi, who's the lead on SAM—or I guess just SAM in general, right? We also have Pengchuan, who's a researcher on SAM.

Yeah, nice to meet you guys.

So, congrats on SAM 3's launch. The demo gets better every time you step it up, really amazingly. My general impression, or takeaway, when I tell people about SAM is that every time you have a new release, it's like once a year: you show up, drop a banger, drop the mic, and go work on the next year. You also add a dimension.

I was entirely— weirdly—not surprised when SAM 3 had the 3D thing, because I'm like, “Well, yeah, which is the next dimension to go? It's 3D.”

Actually, maybe just on that, I think that's a common misconception. We launched 3 separate models this time. It was SAM 3.

Correct.

SAM 3D Objects and SAM 3D Body.

Yes.

Those were 2 completely separate models. SAM 3 is just the image and video understanding model, which is on a DINOv3 backbone and is sped up.

Yeah. Sorry, I didn't mean to preface all this, but maybe, just to remind our audience—or for people new to the SAM series and the podcast we've done so far—each of you can go around and introduce your entry into computer vision or your relationship with SAM. Go ahead, Nikhila.

Okay, cool. Hi, everyone. I'm Nikhila. I'm a researcher at Meta, and I've been at Meta for 8.5 years, so I've really been through the evolution of the field in that time. I started working on a range of different problems in computer vision and worked briefly on 3D. We've got this library called PyTorch3D.

I really started working on the Segment Anything project around late 2021. So it's actually been almost 4 years since I've been working in this Segment Anything space. We started with SAM 1, then SAM 2 in July 2024, and now SAM 3.

It's been the culmination of a lot of work from a lot of people over the years. So, yeah, I'm really excited to be at this point and get to share it with all of you. I'll hand it over to Pengchuan.

Yeah. Hello, everyone. I'm Pengchuan. I'm a researcher on the SAM team. I've been working in the field of computer vision for nearly 9 years, starting in 2017, so I think it's a long time.

I worked at MSR for 5 years and then moved to Meta Reality Labs to work on egocentric foundation models for AI glasses for a while. Then, in 2023, I moved to the SAM team, and that time is exactly the start of SAM 3. Really, I think that's the lifetime experience I have on the SAM team.

I'm glad that SAM 3 is out, and I kind of achieved my original grand goal in computer vision: to reach human performance in detection, segmentation, and tracking in images and videos.

I'm Joseph, co-founder and CEO at Roboflow, where our mission is to make the world programmable. We think software should have the sense of sight, and models like SAM and others are critical to unlocking that capability.

Millions of developers and half the Fortune 100 build with Roboflow's tools and infrastructure to create and deploy models to production. We've been big believers in the Meta family of open-source models, all the way back to Mask R-CNN and Detectron2, and all the way to the present: SAM 1, SAM 2, and SAM 3.

The work that the Meta team does to advance state-of-the-art, open-source computer vision has been bedrock to enabling developers and enterprises globally to adopt AI. We've been big fans of the work, and I'm pleased to be joining you today to co-host the episode on SAM 3.

You guys shipped your own RF-DETR model, too.

Yeah, we've been doing some work to advance machine learning research, too. One example is RF-DETR, a Detection Transformer, which was born out of NeurIPS last year.

We had this observation that transformers had surpassed a lot of CNNs in vision tasks, but they hadn't been made to run in real time—as in, over 30 frames per second, for example, on a small T4, or, excuse me, a small edge device—and hundreds of frames per second on a T4.

We did some research and published RF-DETR, or Roboflow Detection Transformer, which we kind of joke is the greatest-of-all-time model for doing real-time segmentation and, obviously, detection on the edge. In RF-DETR, you have to have a fixed class list and need to know some of the objects that you want to segment ahead of time.

But for anyone running on constrained compute and on an edge device who wants an Apache 2.0 model to do that, RF-DETR and its family of models are key to fulfilling that mission and goal.

Yeah, amazing. I think we're going to go into a SAM 3 demo. Nikhila, you've prepped some things to show us, and there's nothing better than the creator of the tool showing it off.

To start with, what is SAM 3? SAM 3 is a model that can detect, segment, and track objects in images and videos using what we call concept prompts.

I'm going to start with a simple image example, and then we'll show you a video example. A concept can be anything that is a short text phrase. Here, for example, we can use something like “watering can,” and you can see the model predicts a mask for the watering can.

You can then refine the prompts using clicks or additional visual exemplars, which I'll show you in a different image. The idea of a concept prompt opens up the ability to find all instances of an object category without having to manually click on every single instance, as you would have had to do if you were using SAM 2 or SAM 1.

If the model misses any of the instances, you can add visual exemplars. A visual exemplar is also a way to describe a concept to the model. Here, I can add a positive box and show the model that this is also an instance of a flower that we want to detect.

This is just for images, but what's really cool is that you can now also do this in video. Here, I'll show you an example. Maybe this is a football match and you want to track all the players in white, for example. You can provide a concept prompt, and the model will find the objects in the first frame, then track and detect the new instances that appear later on in the video.

It's not just detecting on the first frame, but both tracking those detections and finding new instances that appear throughout the video. One of the things we love to do in our demos is show some real-world applications of this.

One idea is that you can use this for video editing or adding effects. Here is a really simple mask effect, but you can imagine that you might want to add a trail around the players and follow them around. Maybe you want to clone them, so you've got multiple players running around. You can also do background effects, such as spotlighting players.

These are just fun things you can do on top of the SAM 3 outputs. There are also some templates that are prepopulated with a text prompt and an effect. These are fun ways you can use the outputs.

Really, the crux of it is “Create from scratch,” where you can upload any image or video and try SAM 3 on that. We'll share the link so you can try it out as well.

One of the other demos that I have is a busy scene for labeling, which we can do later on, but just to give you a preview. If you wanted to find a tablecloth—and maybe back there there's an airplane—I'll do “airplane,” and you get the ability to start to define the confidence thresholds.

They do.

I don't know why “tablecloth” wasn't as good. I've used that one in the past. Maybe “table”? Yeah, cool.

Wow, look at that.

I think the other impressive thing you guys emphasized in your launch is also the latency. I don't know where this particular inference is running, but it says something like—

SAM 3 runs in 30 milliseconds on a single image if I want 100 detected objects on an H200.

Obviously, this is an H200, but it's also just impressively fast, and sometimes you can basically be real-time if you want.

Yeah, definitely. On images, it's really fast, and on video it scales with the number of objects. But for a limited number of objects, it's still really fast.

Also, even for video, if you can't afford many GPUs, we implemented a very good parallel inference algorithm. So even if you have a lot of objects to track, you can still get near-real-time tracking performance as long as you scale up the GPUs.

I'm reading in the paper: 10 objects on 2 H200s, 28 on 4 H200s, and 64 on 8 H200s or something like that.

I don't think there's an architecture there. I don't know if this is the parallelism demonstration that we're talking about.

In fact, when you try the demo, the video uses the parallel implementation of video grounding, so it's already in that fast mode. If you try it with a video with lots of objects, you can notice that it's actually not very slow, and you get the sense that we're doing multi-GPU inference.

Yeah, everyone should try it out.

Okay, amazing. This thing about concept segmentation—I feel like you had a prototypical version of this, and in your paper you really talk about generalizing it. What was the planning like for SAM 3 at the start? Was what we have today exactly what you planned for, or did it emerge as you discovered capabilities?

Maybe I could quickly talk about that. In SAM 1, we did have a proof of concept of text prompting, but that was just a very early exploration. It wasn't really built out, and it became the most highly requested feature since then. So, in SAM 3, we really wanted to do it properly and make it work in all different scenarios.

We had to really think about how to formulate the problem. It could have been that we took open-ended text input and made it work for all open-ended text, or we could have been more focused, which is what we chose to do, and really focus on these atomic visual concepts, like “yellow school bus” or “purple umbrella,” and nail the problem for these atomic visual concepts.

Pengchuan, maybe you want to talk a little bit about the benchmarks that existed previously and how we had to fully redefine the task and the benchmark that we wanted to solve.

Yeah. Maybe just to add to Nikhila's point, if you look at the size of these benchmarks, the previous benchmark Nikhila mentioned, LVIS, that everyone uses, has about 1.2K unique concepts. The benchmark that we created, which we're calling Segment Anything with Concepts, or SA-Co for short, has more than 200,000 unique concepts.

If you think about the natural language that people use, we don't just use 1,000 words. We have a very large vocabulary, and we really wanted to build a benchmark that could capture that diversity and size.

Yeah, it's really impressive and also very formulaic, or classic: every great model starts with a lot of data work. I think it's basically a scaled-up version of the same process for SAM 2.

Yeah, in some ways, I think the SAM 3 data engine really was a very novel and critical component. To your point, competitive advantage in AI is not just about the models, but really about the data—and maybe even more so, the data engine to generate that data. We put a lot of effort into SAM 3 specifically to automate that process.

One of the things that we're really impressed by is the diversity and depth, as well as the breadth, of uses that we see with models like SAM in production. Basically, when you think about computer vision, folks always think about dogs and cats and simple sorts of things. The reality is that computer vision is where AI meets the real world. Any sort of thing that needs to be seen and understood requires an understanding of that thing.

A model like SAM expanding the concepts from a few thousand closed-form concepts at most in a single model to tens of thousands of concepts means that you're going to see a huge acceleration in the number of fields and applications for the model.

So this is SAM 3, right? We've already seen and measured some of the impact of the SAM family of models, and we pulled some updated statistics on how impactful SAM is across the Roboflow community. I think Roboflow might maintain one of, if not the largest, hosted instances of SAM, and we've seen basically 106 million Smart Polygon-created examples that are powered by SAM 1, 2, or 3. We estimate that that's saved humanity collectively 100, maybe 130 years, depending on exactly how you want to calculate the time, just curating data.

Each of those use cases isn't just dogs and cats on the internet. We see medical labs across the world accelerating cancer research by doing things like counting and identifying neutrophils after a given experiment. We see folks using aerial imagery to help a drone navigate through the world, count and identify solar panels from above, or even do insurance estimates.

We've seen folks building underwater trash-cleanup robots. You can imagine an autonomous underwater bot navigating through the Pacific Ocean, identifying and grabbing plastics, and cleaning up the world's ecosystem. Relatedly, we've seen work with organizations like MBARI that are keeping track of species and identifying the impact of certain steps that are taken, or increasing the populations of given fish with underwater fish cameras.

We see folks in industrial settings doing work to produce electric vehicles or get products from point A to point B. At the time of recording, it's near Christmas, and it's a busy time for the holidays and for people giving gifts. That ends up being a really important time for making sure goods and services show up where they're supposed to be at the given point in time.

One of the statistics that we track is the frequency with which folks cite work like SAM, Roboflow, or the blogs that we publish. There are now a little over 2 research papers published every day citing some of the work across the Roboflow community. Those are folks publishing in Nature, ScienceDirect, and a fairly prestigious number of journals.

Each of those publications is someone's seminal work, often 6, 12, or 24 months of effort that's been accelerated by models like SAM. It's not an exaggeration to say that models like SAM are speeding up the rate at which we solve global hunger, find cures to cancer, or make sure critical medical products make their way to people all across the planet.

At the infrastructure level, we're thrilled and constantly surprised by the breadth and depth of adoption that we see from the community. In the first 5 days of SAM 3, there were about 8 million inferences running across all sorts of fields, and that's only increased since it was released. Then there was Thanksgiving, and now people are using it pretty heavily again. It's been incredibly encouraging to see both the depth of adoption and how much the community takes, uses, and relies on models like SAM in production.

Yeah. Maybe just to add to that from Meta's side, we don't usually get as much visibility into all of these real-world use cases. Being able to hear that from Roboflow and having these models available on the platform is so valuable for us, because we get to know how these models actually work in the real world, which is ultimately the best evaluation for a model. It's definitely awesome to hear about all these things that we're empowering.

Nikhila, you had this comment that the best evaluation for a model isn't necessarily a benchmark. What was it? If it works on real-world things? I think it's a really good sound bite.

Probably something like, “The best evaluation is whether it works in the real world.”

Yeah, true.

That's the ultimate goal for all of our models: SAM 1, SAM 2, and SAM 3. We want people to use them out of the box as much as possible. With language in SAM 3 specifically, there does need to be some domain adaptation in certain cases, but we've tried to make that easy.

Joseph, do you want to talk a little bit about the fine-tuning aspect?

I wanted to also endorse the real-world thing. I was happily surprised when I visited the CZI Imaging Institute in preparation for our podcast with Mark that they were using SAM to image human cells. They showed us how, in reality, all these sorts of masses are really undifferentiated, and it's hard for the human eye to track them.

This is actually a simpler example, where it's pretty clean. In reality, a lot of it is just gray mush, and you have to segment individual blobs out of it. They showed us how they were using SAM and fine-tuning SAM to do it. It's really complicated and also very meaningful for basic science research.

I should also mention that, in the paper, you can see what SA-Co's data distribution looks like: a lot of animals and, surprisingly, very few maps. I'm thinking, maybe there should be more maps. I'll say Hugging Face has been doing a lot here, as have other companies.

Yeah, this is actually one thing we get asked a lot: what's the minimum amount of data I need to fine-tune? Being able to do that with just 10 data points will hopefully unlock a lot more than we can do ourselves.

Yeah, I mean, the more the merrier. Obviously, this is where ablations are really helpful. You probably didn't have any fine-tuned ablations in here; I think this is all data- and model-training-oriented. But it's very clear.

I just have a cheeky, curious point: is there a ratio of negative examples to positive examples? In Nikhila's example, when you were demoing just now, you only selected positive examples. Obviously, there are going to be a lot more negative examples of a class than positive examples of a class.

So should there be some exchange ratio where negative examples contribute less than a positive example, or is that not the case for positive and negative examples?

I don't know that I've seen a golden ratio that works well or doesn't work well, but I can offer, anecdotally, that a single negative example goes a long way. A common place where fine-tuning is really helpful is data that's out of distribution that might have been impossible in production. One of my favorite fine-tuning examples is counting Waymos. There's not that much data that has Waymos labeled throughout the streets of San Francisco, but SAM 3 does a really good job of identifying a Waymo as a vehicle. If you prompt with Waymo, it doesn't find anything; if you prompt with vehicle, it finds and labels a Waymo as a vehicle, which is valid, but a Waymo is a specific type of vehicle, right?

From even just a 10-second video clip, you can actually start to have SAM 3 learn what should be seen as a Waymo versus what should be seen as a vehicle. Even on a single-image example, we see that SAM 3 starts to adapt because it takes the text and image prompt into account when it makes a subsequent inference. From 3 to 5 negative examples alongside positive examples, you start to see the model update its priors, if you will, for where it would predict things from what the user provided.

All this comes with caveats, right? Because when you talk about the visual world, the negative and positive examples could have been from a very different perspective or a very different type of object. Maybe you're labeling dog breeds and suddenly a new dog breed appears, or maybe you have a perspective where it's overhead and then suddenly you have a side-by-side view. So usually the best way is to have these things meet the real-world data and try.

But I'll offer the note that a small number of negative examples goes a really long way—small, like 3 to 5, not hundreds.

Yeah. The other place where negatives play a big role is whether it's in the image or not. One of the things that we did was really separate the problem into a recognition problem and a localization problem. First, can you answer the question, “Is this object or this concept in the image?” And then, if it's in the image, where is it in the image?

To really build in that capability, we had to annotate a lot of negative phrases in images—basically, a lot of phrases that don't exist in the image, in addition to the concepts that exist in the image with the corresponding mask pair. So, if you look at one of the tables in the paper that shows the training dataset distribution—I think it's Table 24—more than 70% of the annotations are these negative phrases that are not present in the image. We have to really train the model not to detect stuff that is not in the image.

Yeah, I think that the separation of localization and recognition is basically precision and recall, right? But in the vision domain, we basically add this presence token to the model, which explicitly separates the task of recognition and localization.

Basically, it simplifies the task, so the model doesn't have to try to do everything with just the proposals in the detector. It can have this global, learned token just for the recognition part.

Yeah. In general, I find that you guys did a lot of extra net-new work. You had a really nice chart in here about the yellow boxes being the new stuff. I forget where.

Yeah, the architecture diagram.

Yeah. I'm like, holy crap. Last time, it was like, you know, there's the memory stuff. This is SAM 2, and here there's all this. Obviously, it's hard to cover it all, but I wonder if there's any other interesting stories or tricks, like the presence token, that you might want to focus on.

Yeah, I mean, this is a nice diagram. I'm glad you brought it up because SAM 3 isn't just a version bump. It's an entirely new approach to segmentation. It's a new interface for segmentation, and it combines so many different tasks where previously you would have needed a task-specific model for each of these tasks: interactive segmentation, text prompting, open-vocabulary detection, and tracking. All of these tasks would have needed a separate model, so you really had to do a lot of work to bring it together.

One of the things we did was really decouple the detection component and the tracking component. So you can see we still preserve the tracking components from SAM 2, but the detector is separate. The reason we do this is, if you think about what a detector has to do and what a tracker has to do, the detector needs to be identity-agnostic. If you have a concept—dog—it needs to be able to find all instances of that dog, and it needs to have this representation of dog that is the same for all dogs.

But when you're tracking those dogs through the video, each dog needs to have a separate representation such that we're able to preserve the identities. There is this kind of task conflict that emerges between the detector and the tracker. We experimented a lot. We really tried to build a unified approach to do things, but what we found was that having a separate detector and tracker really worked.

We use a Perception Encoder as a shared visual backbone. It's a text- and image-aligned encoder. You can see the green boxes there; it says “from PE.” That's the Perception Encoder. It was also from our group in FAIR at the time. This was released earlier this year, in April.

This really brings together components from the entire FAIR and Meta ecosystem. We have Perception Encoder, a DETR-based detector, SAM 2, Llama, and our data engine.

Yeah, it's like any 3rd film in a trilogy: you always see the previous recurring characters come back.

Yeah. Well, if it works, you got to continue using it.

And to connect to something we discussed earlier, you mentioned that, in the video component, each object needs to be tracked independently. That's why the compute scales linearly with the number of classes, right? Because each of those instance types needs to be maintained.

It scales with the number of detected objects.

Yeah. So, for example, each dog that appears in the video—each one of those needs to be tracked independently. There was something else that you started to allude to in the paper that I was hoping we would spend some time discussing, and it's the interaction of SAM 3 and LLMs—Llama and others.

So, using SAM 3 to almost be like a tool call for LLMs, to give them better grounding and better visual understanding. There's a paper in the table where you describe the increase in performance. It's kind of alluding, I think, to maybe where things are going for using SAM 3 as a component part of multimodal architectures.

Do you want to describe a bit about what the introduction of that work was meant to showcase and how the interaction of SAM 3 and LLMs is envisioned to be important?

Yeah, maybe I can just do a quick intro, and I'll hand it over to Pengchuan to do the deep dive. But essentially, as I mentioned, SAM 3 constrains the text input to these atomic visual concepts, like “yellow school bus” or “yellow watering can.” Obviously, people want to interact with the model using natural language, and we want to enable that as well.

That really segues into being able to use SAM 3 as this visual agent for an LLM. So I'll hand it over to Pengchuan. Maybe you can explain the SAM 3 agent setup and then talk through some of the results that we got there.

Yeah. So, as Nikhila mentioned, the big picture is that SAM 3 is focused on these atomic concepts, but people definitely want to try much more complex phrases, like, “Could you locate the bigger character for me?” Or, for example, “What is the feature that distinguishes male and female in this picture?”

These are more complex language tasks. This is exactly something SAM 3 cannot do, but the SAM 3 agent is targeted at solving. In this case, you can see that it needs much more advanced language understanding and reasoning. SAM 3 currently does not have this capability because it has a small language encoder.

But we know that large language models have been trained on a lot of this data and have this word knowledge and reasoning capability. The SAM 3 agent is exactly using SAM 3 as the eyes for large language models to solve these complex visual grounding tasks.

Are there any insights or surprises that you have, other than, I guess, that SAM 3 is a very good tool? Is that the main conclusion?

If you go to Table 8 in the paper, as you describe this—if you don't mind.

Table 8. Okay.

Yeah. Here we go.

Yeah. To quickly reply to your question, I would say that, first, besides the fact that SAM 3 is really a good tool that provides the eyes for a language model, the other thing we definitely found is that SAM 3 is not perfect. It's not as robust as the human eye.

The LLM also helps to correct the SAM 3 errors. They have a synergy between each other, instead of just having the LLM provide the brain and SAM 3 provide the eye.

Interestingly, you use Llama 4. I saw there's a mix of Llama 3 and Llama 4 here, but it looks like it does best with Gemini 2.5, which makes sense given this comparable set of LLMs.

I think the baseline is also just: what extra addition does this add on top of just the MLLM? I would maybe want to do that ablation. Maybe you’ve already done it somewhere.

What do you mean by the additional thing?

Basically, without the tool call, there’s some native capability inside the MLLM itself.

That’s a really good question. In fact, one of our reviewers even asked that question. Without the language model—without the VLM—SAM only achieves about 30 on the validation set, if I remember correctly.

It’s also very intuitive. The test set has different subsets: short nouns, short phrases, and long phrases. The short phrases are very close to the SAM 3 training data—they’re atomic, short phrases—so there isn’t much complex reasoning. You’ll see that for short sentences, the SAM 3 model alone is very close to the SAM 3 agent, but for long ones, the gap is so large. That indicates that this is exactly the capability the natural-language model brings in.

Got it.

I can show an example here that might be insightful, too.

Go for it.

Even comparing SAM 3 and Gemini 3, let’s say that we just want to have them do an object-detection task. Here, we’re going to prompt with a speedometer and RPMs, and we’re going to ask for things like the indicator light, numbers, and needle. If we run SAM 3 head-to-head with Gemini 3 and Florence 2, almost as a baseline for where things have been, we can see each of the results.

First things first, you’ll note that the inference speed of SAM 3 is quite fast. This is just calling the Gemini 3 Pro API, so whatever hosted compute provides is what you get in terms of response time. The second thing you’ll note, in addition to speed, is the accuracy of the results. We might get a timeout error—let’s see.

Do you have Elo scores?

Of what scores?

Elo scores, like ELO. You had the arena, so I was wondering what the ELO was because you said you were blind-testing this.

That’s actually interesting because we had blind-tested SAM 3 before it was released. It wasn’t called SAM 3—it was just something for people to try and compare. I think we called it something like a potential SAM or SEG preview. We allowed users to vote, and they unanimously voted for what they didn’t know at the time was SAM 3. We actually got emails from people asking, “Where can I use that?” We just ignored them until the model came out.

With the responses here, you can see that the grounding capabilities of SAM 3 are ahead of even Gemini currently. Not only is it doing grounding, but if you look closely, you can actually see that it’s making segmentation masks, too, whereas Gemini 3 struggles with that and, by comparison, just does detection.

The other thing is the richness of the detections. Recall is high, as is precision. If we compare them here, Gemini does almost as well, but you can see that it misses some of the numbers and has some erroneous boxes that it predicts. It also doesn’t do segmentation; it just does detection for this task.

You can envision that, in the same way the SAM 3 paper introduces the idea of using SAM 3 in tandem with MLLMs, that will probably be the case pretty soon. Maybe the Google team will take some notes to improve Gemini and other series of models based on what SAM 3 demonstrates here. In other words, it’s not only faster, but it seems to be more comprehensive for concept segmentation.

I think the speed is actually a huge factor for many use cases. Even at Meta, we’re using SAM 3 for various product use cases, and fast inference speed is critical to enabling that. In many cases, you don’t even need an MLLM; it’s overkill to use an MLLM for some applications.

The other interesting thing is the Florence 2 results. Florence 2 is a little bit older of a model now, so maybe it’s not fair to put it head-to-head with the state of the art, but it is useful as a way to see how far we’ve come. By comparison, Florence 2 labels the entire region as a single class without detecting the individual numbers, indicator lights, and needle. Not only that, but it actually runs at about 3 times the speed of SAM 3.

SAM 3 is faster while doing a task the other models aren’t doing—segmentation—and it’s more accurate in both recall and precision for the things it’s intended to find. That really showcases the capabilities of the model.

I was a little surprised by this because it’s more like an OCR task—recognizing numbers is nearly OCR. We didn’t prioritize collecting OCR-heavy data, but it works. We knew that it roughly worked, but I was surprised that it worked so well.

That’s encouraging. Even on a task that wasn’t expressly prioritized, it still does a great job.

In fact, during our data-engine work, we intentionally did not sample OCR-heavy images.

On an easier one: glass mugs. SAM 3, Gemini 3, and Florence 2. SAM 3 loaded first, and, impressively, it even sees this glass mug in the corner. I think occlusion and partial objects are things SAM 3 does a great job with.

Gemini 3 struggles a bit with this one, maybe because of the opacity of the objects by comparison. Florence 2 does a good job finding one of the glass mugs. Again, this is another type of task that shows the power and versatility of the model.

Exhaustivity—finding every instance—is something we heavily prioritized, and it’s really built into the data-engine design. Joseph, do you want to talk about how we designed the data engine to scale exhaustivity? If a human had to annotate every single instance, it would take a really long time to annotate and verify, so we put a lot of effort into automating and speeding up that process. That allowed us to reach the data scale and diversity needed for a step change.

I think the data engine is definitely the critical component behind the SOTA performance we’re achieving now. Maybe we can go to the data-engine figure.

Page 5. Yeah, yeah, yeah. Here.

You can see that this is our annotation pipeline. We first source the images and generate the noun phrases. This is the input for the task: source images and generate noun phrases. For example, we generate captions and parse the captions to get the noun phrases. This gives us the input distribution.

Then we use the segmentation model in the loop to generate candidate masks. We can say, “That should be the candidate,” but it’s not perfect, especially in the beginning. The next step is verification. The model gives you these masks, and we first verify each mask to determine whether it’s good or not.

After filtering out all the bad masks, we have some good masks left, and we verify whether those good masks are exhaustive. Take your mug example: if the model doesn’t predict a partial mug, the exhaustivity check fails. When exhaustivity fails, we go to the next step, which is human manual correction. Human annotators manually annotate all the missing masks and make the data point exhaustive.

You can see that exhaustivity is a very big factor. We put it at the center of the data engine. If we ask a human annotator to annotate every mask from scratch, it takes a lot of time. In the beginning, each data point took more than 2 minutes to finish. With the model in the loop, that was reduced to about 45 seconds: the model proposes the masks, and the annotator only has to add the missing masks.

Another key innovation in this data engine is that we found the verification steps—verifying whether a mask is good and whether the good masks are exhaustive—can be done by AI, specifically by a multimodal model. We fine-tuned Llama 3.2 with human verification data and achieved superhuman performance on these 2 verification tasks. We no longer need humans for those 2 tasks, which further reduced the per-data-point annotation time to about 25 seconds.

You can see the journey of our data engine: from the original all-human process, which took about 2 minutes, to about 45 seconds, and finally to 25 seconds for each data point. That’s how we made the data engine highly efficient.

Did you maintain statistics on how many images were specifically hard? For example, did you track how many objects were very difficult or occluded, or how many images had a particularly difficult exhaustivity test? Or did you just bet that, at large scale, you would encompass occlusion and exhaustive cases?

We maintain that information, including exhaustivity and which cases are hard or easy. First, when a human annotates in our data engine, we know exactly which data points are exhaustive according to the model and which parts require human intervention.

In fact, we have that kind of metadata in our dataset. The second one is the better, more beautiful part: we have this kind of exhaustive AI annotator. Given a new data point, we can automatically decide whether it is a difficult data point or an easy data point with this AI annotator.

Yeah, I think the bootstrapping and annotation story was very strong last time around, and it’s even stronger this time. What are you going to do when you run out of humans? Next year, you’re going to have superhuman-level performance on everything, right? Like PCS and PVS. What then?

I’m not so optimistic about this. First, our current plan for the next project is a fully automated data engine without humans. That’s our dream. I would say that would be the perfect thing, but still, we need some useful information. There’s no free lunch: there’s something that no model can do well, and we need humans to inject that useful information.

I would say that practical minimal human intervention means humans only do the tasks that the model cannot do—the most difficult tasks. That’s the first part, the internal data engine. The second part is about human performance on this kind of PCS task. My feeling is that computer vision is going to enter the RLHF domain when we get to human performance.

You can see that language models, in the early age of language models, were not at human performance. SFT, or imitation learning, could really do the job and get to very good performance, but if you only do SFT and the SFT data is annotated by humans, then your performance is bounded by humans. You cannot get superhuman performance just through this kind of data-engine approach—using human-annotated data and then training on that.

You need to go to this RLHF domain, where humans really just tell you which of 2 points is better. This is exactly the philosophy: telling you which one is better is easier than constructing the data point from scratch. So you can get higher performance from human preference than from humans drawing from scratch. I hope that after SAM 3, we can see new research emerge in computer vision around how to go beyond human performance. SAM 3 is close to that, but I would say that a new learning paradigm is needed to go beyond human performance for SAM 3 tasks and for computer vision.

Yeah, just to add to that, we’re only talking about images. I think video is a whole other challenging beast, and getting to that fully automated data engine is something that we tried to do in SAM 2. We actually didn’t get to that fully automated approach.

In SAM 1, we did. The SA-1B dataset that we released was fully annotated automatically. We didn’t really get to that in SAM 2 for video, and in SAM 3 for video, I think there’s still a lot of room to push on this sort of pseudolabeling for video and really be able to get those same step changes we had on images.

What are the biggest changes needed to see the same step change in video that you’ve seen in images for the automated data pipeline?

Having a good video multimodal model is important. When we did SAM 3 earlier this year—or last year—you could see that image multimodal models were very good, but video multimodal models really became good or practical later this year. They became roughly okay at that stage, so we have a good base model to fine-tune on our data and get to human performance for this recognition or verification task.

We definitely need SAM 3-like effort on the perception side, but we also need multimodal language-model effort—a good foundation model on the vision-language side. I think it’s ready now.

Yeah, also, video annotation is just so much more time-intensive. To be able to annotate enough data to train a verifier, video mask annotation—we found it was very time-intensive. Maybe there are more efficient video annotation strategies. There’s a lot of exploration that could be done there, too.

Yeah. Spending a bit of time on video, I wanted to also talk about—obviously, last time we were focused a lot on memory attention. I think this time there was this sort of masklet thing that I wanted to get more ideas about, or just share the idea more generally. What was it called—the masklet detection score?

Masklet detection score. Exactly. It’s basically smoothing within a temporal window, which I think a lot of computer vision models don’t have. They could simply add it, and it would be a lot more stable when it comes to video. I don’t know why they don’t do it.

One big reason is the streaming requirement. When you want to gather information across the entire masklet, you need to wait for the masklet to end and then get the trajectory. That sacrifices some streaming capability, so the streaming requirement somehow limits the traditional methods for doing this.

I would say this is definitely beneficial. The reason is that I think even humans do this. You can imagine that when something just appears at the corner of the video—say, a hand appears at the corner of the window—you just don’t know whether it’s a man or a woman. Humans make mistakes, and SAM 3 will make these mistakes, too. But when you get more and more information and the person really enters the video fully, then you get to know whether it’s a man or a woman. So the idea here is to gather more information to really nail whether the concept you’re looking at is the concept you care about.

There’s a trade-off between latency and accuracy. If you care more about accuracy, then you can use the overall information across the entire masklet to get a more robust signal about the concept. But if you care about latency, then you need to make a decision at the very beginning, and you will sacrifice some accuracy.

I think also, in many video use cases—I think, Joseph, you were showing that Roboflow users care more about detecting the object rather than having unique identities. So, in some cases, maybe it isn’t required to preserve identities throughout the video, and you just want to do detection per frame, like the Roboflow Rapid examples you were sharing.

Yeah, there are cases where being able to count matters, and the objects are all going to be the same, so you don’t care as much about unique identities. You just want to know the full presence. Things like that matter. But there are other cases, like you mentioned, where maybe in sports you care about individual players versus just knowing that there are 11 players on the pitch.

One thing that might be useful to discuss with some of our time left is that we talked a little bit about how SAM 3 and VLMs will play nicely together, but there’s probably a broader discussion about how SAM 3 fits into the broader AI ecosystem and what bigger-picture trends it might fit into. Do you have some thoughts on what this represents about where things are headed?

Yeah, maybe I could say one point, and then, Peng, feel free to add one. As we mentioned before, SAM 3 isn’t just a version bump. We’re really having a unified model that can do many different tasks in the same unified architecture.

In the same way that LLMs can do many different tasks without needing a task-specific model, with SAM 3 we’re able to do image-promptable concept segmentation and video-promptable concept segmentation. We don’t need a specialist model for counting. We can do interactivity. So there are really multi-capability visual models that are on par with or better than single-task state-of-the-art models. That’s one place in which SAM 3 fits into the AI ecosystem.

In terms of VLMs, I don’t know if, Peng, you want to talk about the agent approach.

Yeah, definitely. You can see that SAM 3 is now really getting a big step change in vision. How it helps general AGI and fits into the general AGI or frontier-model landscape is very exciting for me.

We always have this example: give a six-fingered hand picture and ask how many fingers are in the picture. You can ask all the frontier models, including GPT-5, and you can imagine that with SAM 3, we can first detect how many fingers there are very robustly—six fingers—and then the multimodal model should know that this is a six-fingered hand instead of a five-fingered hand. The errors made by frontier models can be solved if we use SAM 3 as a tool.

But how is SAM 3 as a tool the end of the pipeline, or should SAM 3 be more naturally embedded into these foundation models? Should the frontier models have SAM 3 capability by themselves? I would say there are a lot of possibilities there. My picture is that now we have a very good brain with foundation models and a very good eye with SAM 3.

Now let’s see whether the eye is really working together natively with the brain, or whether it’s really a different kind of organ that needs to somehow work with the brain as a tool. I think this is a very exciting research area.

In your analogy, if you think about the visual cortex compared to the human brain, we have rods and cones in our eyes that do very fast detection—we joked, at a lizard-brain level, detecting simple stuff—and then you have your brain that reasons about some of the visual information that your eyes see. In your example of SAM 3 as a tool call versus SAM 3 as natively being a part of the multimodal model, which future do you think is more likely?

I think, at least, I want to bet on them running their work natively together. For simple or even intermediate-difficulty vision tasks—for example, counting fewer than 20 objects—I think this is like System 1 visual reasoning with our brain. Our brain should do that by itself.

But with very difficult tasks, you can imagine counting thousands of objects in a crowded picture. Then we might even need to draw something there. At that time, maybe we need an extra model for difficult tasks. This is a hybrid approach, but I’m more excited about the idea that, in most cases, it should be native.

So, for very simple System 1 questions—things like how many fingers are on a hand—that should be native. But for more complex things that are maybe long-running tasks and require long-running reasoning, there may be more of a tool-call approach.

Yeah, exactly. For example, in our SAM agents or in our AI annotator, we already demonstrate this approach. For simple cases, the model can do it by itself: “Okay, I can detect 10 people here.” Then the natural language model—or the AI annotator—can even know, “Okay, these 10 people are not exhaustive; there are more people there.” If you want to do it well, maybe you need to take more steps, such as calling an extra model.

You can see that this is a very native reasoning process for more advanced or complicated vision questions.

I have a related but slightly different question. SAM 3 is an incredibly powerful piece of work, and it’s open source. Is open source critical to achieving AGI?

Maybe I can comment on SAM specifically. In SAM 3, we leveraged many of the open-source contributions people had made on top of SAM 2. There were new datasets, new benchmarks, and new inference-time optimizations. We adopted a lot of the things that the community built on top of the models and datasets.

All those contributions helped make SAM 3. For the SAM series, we really benefited a lot from being very generous with what we open source and then leveraging what the community builds on top of that.

That’s just from the SAM perspective. I think it’s clear what the community brings and offers, and every time we do this, we always shout out to the community to try it on their use cases and report weird findings. If it doesn’t do what you’re trying to make it do, let’s talk about it and maybe implement it in the next version.

You already hinted at what might be coming for SAM 4, which is at least a little bit more document understanding and OCR work. Are there any other interesting directions? Obviously, there will be a lot more video work as well. What’s the talk of the town in the computer vision community? What would be really great or super obvious? Is next year going to be the year of something in particular?

Yeah, maybe I can talk about something first, and then I can add to it. First, definitely, I think that even if it’s not SAM 4, it could be SAM 3-something or SAM 3-point-something: small models. SAM currently only has 1 model and 1 size. We need a more efficient model that fits edge devices, and also a more efficient model for video.

Currently, the video model is not efficient. You either can’t achieve very good throughput, or you need GPUs to do that. So, first, small and efficient models. That’s 1 big thing. The second big thing is definitely video.

Roboflow can do that for you.

Yeah, yeah. The second thing is video. I would say that video is still far from human performance. There’s still a big gap from human performance, and a lot of research needs to be done there.

We need to figure out how to do end-to-end training with video. Right now, we have this decoupled approach, but we don’t end-to-end train the model, and we expect that it will definitely benefit from end-to-end training. On the video side, we also need to figure out how to scale up the data engine. We definitely need AI annotators for video. We’ve tried that, but I think that’s something worthwhile to pursue.

The third thing we’ve also discussed is how perception fits into AGI’s big landscape. Now we have the eye; how does the eye work with the brain to solve real reasoning tasks—not only output segmentation, but really answer questions like, “How many kids are here?” or even answer questions in general?

For example, in a biology lab, a robot might need to decide whether the liquid in a test tube is at the correct level. You can see that this involves perception, but it also involves reasoning. How to solve these more beneficial reasoning tasks with SAM is a very big direction.

On the robotics topic, it was exciting to hear from several friends who work at different robotics companies about how they’re immediately starting to use SAM 3. Especially for the video use case, robotics is probably one of the domains where improving video performance will have a lot of impact. To Pengchuan’s point, there’s still another step change to be achieved on video performance.

It’s also worth noting that we’re interviewing a bunch of robotics folks here, as well as Fei-Fei Li, who obviously started ImageNet. A lot of people are betting on explicit world models, and SAM is not one, for better or worse. I wonder when that crossover might happen. That’s an open question, if you guys want to discuss world models and where things are going based on community questions.

Similar to how Nikhila mentioned SAM 1, one of the almost obvious things that people wanted was open-vocabulary prompting. People were like, “Great, this model can see things, but I want to tell it what I want it to see.” Now, with the introduction of SAM 3, you have this text-based component, which feels like a key component of the ChatGPT era for vision arriving as a result.

What’s going to happen now is that you’ve provided people with an open text box and media, so you’re going to get all sorts of queries from people that maybe the model isn’t primed to perform particularly well on yet. For example, we were talking earlier about document understanding and document reasoning being areas where there are known improvements to be made. People will probably prompt it to try to OCR things, or they’ll want to do work with spatial reasoning: “Give me the object to the left of this other object,” or “Give me a sense of where things are in relation to one another.” That’s critical for robotics, because that’s how you navigate throughout the real world.

You’ll also have people asking for action recognition and vision-language-action models, or VLAs. These are the same types of tasks where people are used to providing open text prompts and getting, “Here’s the part of the scene where the player kicked the ball,” or, “The tennis player made the serve.” Those are interesting for understanding and synthesizing visual inputs.

Now that you’ve given people this open text box for media, there’s going to be a flood of the types of things users will want to try to do. Some of them SAM is already going to be really well adapted to do, and some of them it won’t be. I think that’s going to reveal the types of things that are obvious.

One of the things that we wanted to discuss was where to use SAM and how to discover how to build with SAM. In addition to the Meta team building a tremendous playground for interacting with images and video and applying effects, with a video emphasis, I think one of the things we’re pretty excited about with SAM 3 is how much it positively impacts each part of building a system for visual understanding.

So, for example, the very first step of historically aggregating and collecting a dataset because you think that there isn’t a model that understands the slice of the world you want to understand is where automating away a lot of labeling can exist. Basically, if you collected a bunch of data on something that’s already within SAM 3’s knowledge, then you can prompt SAM 3 to automatically label all that data for you.

We’ve actually made a bet on SAM 3 being a core part of auto-labeling at Roboflow, giving users a first pass. If you have a new image or a new video, you can start by providing just a text prompt and allow SAM 3 to find and automatically label those regions of interest for you downstream. I think there are areas for fine-tuning, too. Within a week of releasing SAM 3, MedSAM 3 came out for adapting SAM into medical contexts, and I think that’s a harbinger of what’s to come.

There will be lots of domain-specific adaptations of SAM in places where maybe there’s a specific ontology that someone wants to understand, or maybe the model just doesn’t have great awareness yet. We’re already beginning to see that with hundreds of fine-tunes that users are creating for various domains.

Then the last area is, “Okay, I’ve got my model. Now I want to use it.” One of the things we’re really proud of is being ready on launch day to showcase the infrastructure we’ve built to burst and scale infinitely large as folks have models they want to deploy and make readily available. Having an endpoint that serves either a fine-tuned model, a model as-is, or even a model that might be able to run on edge hardware as smaller models come out or as distillation becomes more common is also an awesome place where we’re seeing SAM 3 have an impact on each part of the computer vision lifecycle and pipeline.

That’s awesome. I think especially the impact on speeding up annotation—we’ve seen that consistently on Roboflow, and I’m really curious to see how the introduction of SAM 3 helps speed up that process even further. Just from playing around with it, it’s so much faster than having to manually annotate every single object. So, yeah, we’re really curious to see how that improves the experience.

One of the things we were pretty excited about is that we were able to build an entirely new product in the world of SAM 3, and we called it RAPID. Basically, there’s probably a model that already understands the objects in the world that you want to see. Here, I’m screen sharing an example: these are vehicles next to our office in San Francisco that go by. You can see a Waymo and other vehicles.

If I have this 10-second clip and the first thing I want to do is count cars and get a sense of each vehicle, what’s really awesome is that I can text-prompt it and say, “I want vehicles.” As I toggle through different frames in my video, SAM 3 already recognizes and understands those objects.

One thing that I think is really interesting is that there was a conversation earlier about how much you want to rely on a model versus the human’s output from the model for what you care about. For example, let’s pretend that in this scene, maybe the only cars we care about are the ones before the crosswalk and not the ones far in the distance. You might get people saying, “Hey, you know what? I actually want the objects that are most confident,” and move the slider down to get fewer objects.

Whereas others might say, “Hey, I want every single presence of a potential object in the scene,” which could even include reflections on the building. As computer vision approaches a world where we increasingly have models that can understand and improve themselves, and we rely on human output and human preference for the models, we’re going to get these funny scenarios where things aren’t immediately deterministic in terms of what a human cares about.

I think that’s where tooling fills a big gap. It’s also going to be interesting to see where users start to use and apply the models, and why you need this last-mile work to put the model in context for the domain that someone is trying to solve and tackle.

So, let me—since you’re here—this is one of those things where I’m not sure the concept of labeling concepts can scale, only because I don’t know if this slider between less and more is the way forward. Ultimately, I may need to tell you whether or not to include reflections, because sometimes the reflections are exactly what I want. Most of the time, they’re not going to be what I want.

I don’t know if some RLHF thing is going to solve any of that, because you just need more prompting. Just saying “vehicle” isn’t going to do it. I don’t know. Feel free to disagree.

You can imagine this kind of pipeline coming, for example, as was said: maybe the reflection is exactly what I want. Then you need some kind of iteration with the interface or the model to get finally what you need. You need to specify the concepts more clearly through multiple iterations.

Can a human not be involved in this iteration, but just models, and have them do it automatically? I think that’s definitely something interesting. You can imagine this workflow: I want reflections, and with the default threshold, maybe the model will get an output. Then another very strong perception model, or another kind of model—

A model like Gemini 3 could then ask—we could ask Gemini 3 whether there’s some reflection here. If it says yes, then we can automatically move the threshold lower and ask again and again to see whether the reflections are now included or not. Somehow, this process could possibly be done completely with AI.

I think Joseph is showing us the sort of Waymo annotation. Yeah, it’s nice. Now you have a Waymo model.

Yeah, I was just doing an example where maybe we want to find an object that’s not already represented in the training data.

Yeah, I think prompting could solve the problem of reflections, because maybe you could say “vehicles on the street.” But to your point, you would have to see that it’s a failure case, right? If I were just setting up a camera and saying “count cars,” I wouldn’t anticipate realizing that reflections could be a problem.

I think this is why, in some ways, a human in the loop is important, because identifying human intention—not necessarily human knowledge—is going to be important for a lot of last-mile use.

Maybe I want to echo what Joseph said. This is also my experience: different people have quite different definitions of even a visual concept. For example, for some dataset involving hands, some people would annotate just the palm as the hand, while some people would include the arm as part of the hand.

When we first tested SAM 3 on a customized dataset, we found that the performance wasn’t that good. When we finally looked into the performance, we found that users had different definitions or explanations of the concepts. Both explanations were valid.

In this case, you can see that you really need a human in the loop to do the few-shot fine-tuning or adapt to the user’s definition of the concept.

That’s exactly right. It’s not always deterministic what someone really wants, which is why I think that even if you have a fully comprehensive, omniscient model, putting the model into the context of what the user is trying to do is where a lot of tooling and infrastructure becomes really helpful.

I would say that it’s likely the benchmark will not saturate models. Maybe next year there will be a stronger model. But the benchmark is the one that I hope will guide the community to get better and better models, to get to the point where we measure human performance on the benchmark. I think maybe we’re the first ones to do that for this kind of segmentation and video grounding in the past. It’s been very difficult to measure human performance on this task. Hopefully, this benchmark can guide the community to achieve human performance for this task and even surpass human performance there.

We set out to be one of the best places, if not the best place, to build with SAM 3 and the SAM family of models. We’re eager to see what people build with SAM and computer vision models to move the whole field forward. We have infrastructure for everything from deploying SAM 3 zero-shot to making your own fine-tunes to automating labeling of data with SAM, and we continue to see the impact with each subsequent release expand the number of use cases and the amount of use and accelerate the time to value. So excited to see what folks can build on Roboflow with SAM.

Thank you all so much. This is really great coverage, great work, and, obviously, as always, it expands my mind as to what is possible with machine learning. We’re not at ASI or AGI yet, but every day we’re getting closer.

Awesome. Thank you so much.

Thank you. Thank you.