# Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs

Latent Space · 2026-11-16 · 98 min · https://www.youtube.com/watch?v=8cGfJOU3AlQ

## Transcript

swyx
Okay, finally, our BFL pod. Welcome, guys—Andreas and Dustin.

Andreas Blattmann
Great to be here. Thank you so much.

Dustin Podell
Thank you. Yeah, definitely. Thank you for having us. I know we’ve chatted about doing this for a long time.

swyx
Yes. We’ve had Anjney on a couple of times, and that’s where I met you, in Vienna, I think at one of the ICLRs or—

Andreas Blattmann
Was this the rooftop event, or what was it?

swyx
There was a year when there were 2 Viennas back to back.

Andreas Blattmann
Yeah, yeah, yeah. That was first ICLR, and then it was ICML.

swyx
Yeah, yeah. I don’t know how they do this thing. Anyway, at the time, you guys were just about to launch BFL. We had dinner, and you were thinking about what was next after Stability and all that. Anyway, congrats. How do you guys like to introduce yourselves? Maybe we’ll get a crisp self-intro up front.

Andreas Blattmann
I’m Andi. I’m a co-founder of Black Forest Labs. I’ve been developing diffusion models for, oh wow, I think 7 years now. I met Robin and Patrick way, way back in the day. I think it’s been 7 years since we met at Heidelberg. I’m originally from Germany. There, we started looking into diffusion modeling for images. We all had a physics background, in some sense, and this paradigm of mapping the general physical phenomenon of diffusion to machine learning and generative modeling had a kind of philosophical beauty to us. That’s why we got into it.

### The Origins of Diffusion

Then we developed a lot of the fundamental algorithms that are powering diffusion models today, for image generation, video generation, and also action prediction. I guess we’ll talk about this later.

swyx
We’ll talk about it. It’s not immediately obvious to people: image generation to action prediction.

Andreas Blattmann
Yeah, we can talk about that. And one thing I did want to comment on is that a lot of people don’t know the relationship. Black Forest is a forest in Germany?

swyx
It’s a mountain range, actually.

Andreas Blattmann
Ah. It’s a region in Germany, basically. Finally, Robin and I first met during our PhDs, and then we figured out that we were actually born, I don’t know, 10 miles away from each other. That was pretty fun, and eventually that led us to come back to Germany.

swyx
Beautiful. All of Europe looks like this. You can see the lab right in the middle there.

Andreas Blattmann
Yeah, yeah, yeah.

swyx
So you named it because you’re from here?

Andreas Blattmann
Exactly. We founded this research lab, and back in the day, we could basically pick where we wanted to do it. No one said, “That’s okay, you have to come to, say, San Francisco or whatever place.” So we decided we wanted to start this at the place where we were born, in Freiburg, which is at the foot of the Black Forest, basically, or a city in the Black Forest. That’s how it came about, and that’s how the company started there.

swyx
CompVis was also in Heidelberg?

Andreas Blattmann
CompVis was in Heidelberg. Exactly. That’s where we met. We then later moved on to Munich.

swyx
Which is where the team that did latent diffusion—

Andreas Blattmann
Through our PhDs, exactly. That is when we developed this latent diffusion algorithm.

swyx
Yeah. I really like your name, by the way, your podcast name, Latent Space.

Andreas Blattmann
Yeah. I’d actually be curious how that came together.

swyx
I can talk about it. Go ahead.

Andreas Blattmann
Yeah. We developed the latent diffusion algorithm at the university, and based on this research project, we scaled out the algorithm, which led to Stable Diffusion. Since then, it’s been a constant journey on a rocket ship, you could say, which is super fun. We’ve met amazing people like Dustin as well, who’s one of the oldest and most relevant researchers that we have. But we’ll learn more about that later.

swyx
We’ll gas him up later. I think latent diffusion, and generally VAEs as well, are part of the idea behind Latent Space. Even our logo is actually a VAE. It’s this idea that you can translate things into spaces and then explore things adjacent to what exists today, and then translate back. I thought that was such a cool insight for creativity and generation that I never really understood until I studied Stable Diffusion.

### Stable Diffusion Changed Everything

The launch of Stable Diffusion changed my life. I started Latent Space because of Stable Diffusion. I quit my job because of Stable Diffusion. I was like, “Okay, everything I know about AI is wrong because I can run this on my laptop.” If I was wrong about this, what else am I wrong about?

There’s another relationship with Terry Pratchett, which I don’t know if people have read.

swyx

Writer, right? Writer. He has this concept of L-space, which is library space. He has an orangutan who’s the librarian in Unseen University, which is wizard university. It’s basically a library with all possible books. So if you have a library with all possible books, you can theoretically go in there and find a book that’s right for you, and that is Latent Space.

Yeah. I originally called it L-space. One of my readers donated “Latent Space,” and then it became Latent Space. Amazing. Dustin, what was your journey into diffusion and AI in general?

Dustin Podell

Yeah. I came from a little bit of a different background than Andy and Robin and the team. I’m not from the German side, but I’ve still had a fantastic time working with these guys. It’s funny you mentioned that you quit your job when Stable Diffusion came out, because the original Stable Diffusion came out before I was working with these guys, and it was actually a cause for me to quit my own job at the time.

swyx
You caused mass unemployment. I’m actually curious about the number of people who changed their life direction when Stable Diffusion dropped. I’d be very curious to know the branching effects from that. We can theorize on it.

Andreas Blattmann
That’s actually super interesting, because here in SF, in the Bay, it had this huge impact, whereas in Germany, where we stayed back in the day, people didn’t really notice it because they were so far away from tech. There are a couple of people who’ve been involved in the open source community in Germany, but the majority of people didn’t really notice that drop in summer 2022, which was really interesting. Then you see what happens here.

swyx
Yeah.

Andreas Blattmann
That was really interesting to us.

Dustin Podell
To give a little background on me: at BFL, I’m one of the founding researchers who joined the larger team. I’m sure we’ll get into the history of how that all came to be. My background isn’t traditionally in the ML space at all. I was a software developer for around 10 years. Around 2020, I became very interested in open source AI. I saw the early work in GANs and other generative media, and I was tracking it and experimenting with it on the side. It got me quite excited.

I remember originally following the latent diffusion work and being quite excited about it. One of the things that tipped me off to think, “Okay, maybe this is something a little more serious,” was Midjourney V1. When that came out, I remember thinking, “Okay, this is actually going in a direction. I can see this being quite impactful.” Then the big moment that really changed everything for me, as we talked about, was the Stable Diffusion release. At the time, I was working at a software studio in Los Angeles. Stable Diffusion came out, and that day I turned in my resignation.

swyx
What was the plan?

Dustin Podell
The way I envisioned it at the time, I saw this thing that felt like magic. I thought the people working on it must be arcane wizards, and I needed to do whatever I could to understand what magic, runes, or other things they had imbued in it to make this happen and send it out into the world.

I wasn’t totally disconnected. At the time, I was in the Stable Diffusion Discord. I had friends in there, and I was interacting with the community.

swyx
Was it viewed as basically open source Midjourney, or did people not make the connection?

Dustin Podell
I was in both. I remember there were a lot of discussions around this, about how Stable Diffusion was having such an impact. There was a lot of discussion in the Midjourney Discord and the Stable Diffusion Discord around that time about the impact it was having. It’s been a long time, so I can’t remember the exact nature of it. I don’t want to speak on it.

swyx
Midjourney’s doing fine.

Dustin Podell
They’re doing great. But for me, it was more that I was very interested in the fundamental technology, and then the fact that I could run this at home. What I saw was this community of developers taking it home and developing on it. I was hacking away at it, meeting a bunch of really smart, great people, some of whom are at BFL now and have come along for the journey. That moment of community interaction, learning, and building together was really impactful for me.

Luckily, I was able to find a job opportunity that got me into Stability at the time. I didn’t start working with these guys right away. It took me a while to work my way up to that, and we can go into that history later if we want.

swyx
Stability had a bunch of different teams, right? There was Stable LM, Stable Music. There was—

Dustin Podell
Yeah, Harmonai, the audio team. The way I’d describe it is, if you grabbed a bunch of—

Andreas Blattmann
Discord was the space where a lot of this happened at the time. It was like a physical manifestation of a Discord. It was a melting pot of very different people who came together because they were interested in this emerging new technology. That’s one of the most exciting things about our team. Our founding team comes from very different backgrounds. We have this hardcore academic research branch, and then people from the open source community, like Dustin, who got into this space through a very different trajectory.

I think this is what makes our team so strong. There are rarely teams with this history, and with this kind of joint history together. It’s amazing.

swyx

We can go into more of your other co-founders first. I did want to say that I was also in that space. I was also very influenced by Stable Diffusion. I did the wrong thing: I went down the path of ComfyUI, where I build UIs for Stable Diffusion. You seem to have done the right thing and gone into research.

Dustin Podell

It’s funny you mentioned Comfy, because I used to work directly with him. Comfy joined somebody that I—

swyx

We’ve had him on the pod.

Dustin Podell

Yeah, and I think he moved here.

swyx

Yeah. He’s my neighbor. How do you become a researcher?

Dustin Podell

I’m someone who, when I get interested in something, is obsessive to the point where I can’t put it down until I pull it apart and figure out every aspect of it. I’d already been experimenting with pulling the models apart and talking to a lot of different people in the open source space, not just developers but researchers too. Remember the EleutherAI Discord back then? I met a lot of people I was learning from, and I was reading a lot of papers, trying to take it all in and understand it for myself.

Joining Stability was another major point for me. Originally, I didn’t join as a researcher. I joined as an engineer. But I spent all my free time with any researchers I was able to make friends with. Shout-out to Dr. Scott Hawley, who worked on the audio team and mentored me quite a lot. At some point, I was able to sneak my way onto the cluster and get access to some idle compute.

I treated it like I treat how I learn most things: open source hacking, trying things, reading papers, trying to implement different pieces of work, and seeing how models trained and learned. That was really it. There was no magic to it other than that I was excited and wanted to learn about it. I wanted to tear it apart until I understood how it worked.

swyx

This is open source. It is better.

Dustin Podell

Yeah. I think that’s generally how you learn best: trying things out and—

swyx

And he was also at Eleuther?

Dustin Podell

Yeah, affiliated, but not on the image generation side.

swyx

Back then, everything was very hackable, right?

The delta between open models and closed models was pretty drastic. People on the language model side were doing post-training. That’s where a lot of the new guys started.

You can look at early Stable Diffusion and where we’ve gotten to today. It’s pretty crazy. You both had pretty impactful roles: you did SDXL, and you did Latent Diffusion. We’ll get through all the history, but for people who haven’t followed the space closely, where are we today? What’s the state of things? I know FLUX.3 Video is out, and FLUX.3 Action is now 7B for robotics. What’s the latest in this space with you guys?

Andreas Blattmann

We started as an image generation lab when we started Black Forest Labs, as a kind of legacy of Stable Diffusion and the like. That was our focus for the first year, but then we realized that image generation is nice and this technology can unlock so much more.

### From Images to Visual Intelligence

That’s when we slowly moved into what we call visual intelligence. We’re trying to get our intuitive intelligence into models: the kind we use when we catch a ball and understand physics without knowing how to describe it mathematically, or when we grab a bottle and know exactly what to do because we’ve done it many times throughout our childhood.

We want to transform that knowledge and that form of intelligence, which is very different from language model intelligence, into models. Learning from pixels—which means not only images, but also videos, and maybe audio too. “Natural signals” might be a better term. We realized we had the algorithm to do this, which is diffusion or flow matching. So we decided to extend from image generation to all natural representations. That led to FLUX.3: one model that learns multimodally from images, videos, audio, and actions. We can talk later about how it all comes together.

To put it in terms of Daniel Kahneman, if you’ve read Thinking, Fast and Slow, it’s a kind of System 1 intelligence, not System 2 intelligence. Language modeling may be System 2, but for now we’re focused on System 1, which is extremely important for intuitive tasks and real-world interaction. That’s why, at a high level, it’s also so good for robotics. That’s been the journey, and it’s what we’re focused on now. In the end, it’s about combining both systems into the very broad kind of intelligence that humans also have.

swyx

I want to get into the history and all of that, but just to recap for people at home: you’re already making money as a generative media company, right?

Dustin Podell

Exactly.

swyx

I don’t know what numbers you’ve announced. Maybe you can recap. You raised $300 million—

Dustin Podell

You raised a ridiculous amount of money.

swyx

Yeah, multi—

Dustin Podell

All in all, I think we’ve raised $450 million so far.

swyx

Oh my God.

Dustin Podell

Way more than Stability.

swyx

Any revenue milestones you’ve announced?

Dustin Podell

We haven’t announced any. The last number published, in December, was $100 million in revenue. We’ve crossed that since.

swyx

Clearly you’ve crossed that if you raised $450 million.

Dustin Podell

Yeah, exactly.

swyx

Anyway, I’m just saying you didn’t have to pivot.

Andreas Blattmann

We’re not pivoting. We’re just extending. This model is trained on everything; it’s always been the plan. To create artwork, you also need this visual intelligence. To create a video like the one we’re seeing here, you need to know how physics plays out. The model needs to understand that the arm of this capybara isn’t just falling away; it has to follow some law described by physics. It doesn’t describe the law itself; it describes the output. Similarly, humans don’t understand physics, but we understand that if we let a ball fall, it falls downward.

swyx

There are different concepts of a world model, and that’s something we can get into. Where can you learn? This is the Bitter Lesson version, where you learn from a bunch of input sources, or you physically structure the 3D model and learn from that.

Andreas Blattmann

No 3D model at all. We learn from natural data. We also have a bit of triangulation from our eyes: we have two eyes, which gives us a little understanding of 3D perception. But even if we closed one eye, we could still learn a lot about the world. These models are basically doing that: learning from natural, 2D video data.

That data has a lot of 3D understanding in it. If you want to generate a pan around an object, you need to hallucinate how it looks from behind. If I look at you, I have a pretty good understanding of what your back looks like because I’ve seen it before. That gives you an implicit 3D understanding, even though the models are trained on 2D videos.

swyx

I’m trying to steelman the counterargument. This is the Gary Marcus version: “LLMs don’t learn anything,” or whoever the famous critics of this approach are. The idea is that a model only sees inputs and has never had personal experience. You can model my back because you’re also human and can model other humans. Maybe that analogy isn’t exactly right, because I’m sympathetic to your side of the story. But I think that’s how people criticize this approach to world modeling: you can get closer and closer, but you’ll never quite get something fundamental about the thing because you never modeled the thing. You modeled the most likely constraints around it.

Dustin Podell

I think to some degree you could argue that we operate under the same principle. We’re embodied and constantly taking in data, and obviously we regularly update. But we’re still just taking in filtered signals from the world. There are all sorts of wacky things about how our eyes process information. Part of our brain has to hallucinate some of it because we have a blind spot in the center, among other things.

Fundamentally, I don’t think there’s that much difference between us taking in natural data and building a model of the world that we can act on, and these models learning from data. The signals are different: they’re turned into pixels or waveforms and processed in another way. But the information is still there for the model to take in and build a representation of the world that it can operate with.

swyx

Yeah.

Andreas Blattmann

I was actually arguing about this with Lukas Beyer. He’s with Meta now, I think, formerly at Google and OpenAI. He created bits. He’s in Zurich, which is pretty close to Freiburg, and he comes over from time to time.

We were debating human learning. He was saying that we don’t need interaction, whereas I think interaction is important for higher forms of intelligence. First, there’s observation. That’s also how we humans learn as babies: we lie around and observe things, and from that alone we learn so much. That maps exactly to how we’re currently training these models. They observe a lot of 2D videos and audio, and then they learn from it.

I think the next step will be closing the loop and interacting with the world. That’s also how we learn about the world. We can tap on wood and notice a sound occurs. We can manipulate the world to understand it better. That’s how I think humans mainly learn—at least how I personally learned.

swyx

And he was actually saying he thinks the majority is observation, which I found very interesting. If that were true, then we just need to do more of what we're currently doing, and that could unlock an insane amount of intelligence.

I wonder how much of this is emergence with good priors. We've seen that adding new modalities to language models doesn't give you the full benefit of doubling your performance just because you've added vision. But if you have good priors, is the benefit small or big? It's interesting to see.

I think there's a lot of technical stuff that'll be interesting to walk through, like Gaussian splatting and the 3D data World Labs did. What are the trade-offs between video generation output and other approaches? But I think something fun for listeners would be the history. You worked on very early Stable Diffusion, SDXL, and latent diffusion even before that.

Dustin Podell

Video diffusion as well.

swyx

Video diffusion. There are a lot of buzzwords here: LCM, LoRA, flow matching, old prompting. Can you teach us everything?

Dustin Podell

What's interesting? How big are these, scaling-wise?

swyx

I seriously... No, so I've actually built a tech tree. I wanted to build this game like Civilization, but the tech tree is AI papers.

Dustin Podell

Oh, okay.

swyx

You spend research points to get things. These are the tech trees: first it's GANs, then latent diffusion happened. There are a lot of core points beyond that.

Dustin Podell

I'd play that game.

swyx

So, anyway, just to pitch the game: you role-play as a new lab, and you have to spend points on GPUs or researchers. Then you have to collect data, and there are all these little things you can research.

Dustin Podell

Fine-tune.

swyx

Come on, whip it up. Let's see. We're in the vibe-coding world now. I want to play it tomorrow.

Dustin Podell

Game balance is really hard. I think it's an AGI-hard problem, because once they find a way to win, they just always win that way, and it's not interesting as a game. You need multiple ways to win, and high-risk ways where you probably lose but maybe you win.

swyx

Yeah. Mode collapse, classic problem in GANs, by the way.

I think one thing I'd be interested in teaching our audience is: if you've heard a lot of terms, what are the key things that have really impacted the field? You've lived through a lot of this. I'd also say this is a great way for people who want to get into this space to learn good interview questions if you're hiring. You should know most of these things.

There are some points where different paths have popped up, like autoregressive image generation versus diffusion, two backbones or hourglass diffusion. What are the key things people should know? I'll leave it pretty open. First would be SDXL, maybe. What were the things we did in SDXL?

Dustin Podell

The refiner.

swyx

Oh, no, please, not the refiner. Why that reaction?

### The SDXL Refiner

Dustin Podell

SDXL was a big piece of work in trying to make the models feel good to use for the first time. Stable Diffusion 1 and 2 were very exciting and fun to use, but for anyone who was in the space at the time, between SD1 and SDXL, we saw a ton of work in the open source community on how to fine-tune these models, make them more aesthetic, and make them multi-purpose.

The original Stable Diffusion models could only do square images. SDXL was the first attempt to release a foundation model with a lot of these general improvements built in. It involved large-scale, multi-aspect training. At the time, this was before packing. We had an efficient way to do bucket-style training, on all sorts of different image resolutions at a large scale.

This was the first time we did multiple actual training stages, taking the model from pre-training to developing post-training sets. A lot of data work went into SDXL. I'll give a little backstory on the refiner and some things that didn't quite make it in.

The final release model fundamentally works like the original Stable Diffusion models. It was a latent diffusion model: it took in text, had a noisy latent, took the text context, generated imagery in latent space, and then a VAE decoded it. The refiner was another full diffusion model. Its purpose was to take an image that wasn't rough, but where SDXL wasn't the best at high-frequency detail: the fine details, like skin texture and the texture of grass.

swyx

Yeah, upscaling.

Dustin Podell

It wasn't an upscaler because it didn't actually change the resolution, but it upscaled in terms of detail.

swyx

It's a refiner.

Dustin Podell

It refined it. The reason I thought it was janky is that the whole team that worked on this was fantastic. I don't want to take all the credit; there were a lot of fantastic people who worked on this. We did a lot of work to push the detail quality of the model while trying to keep its generality, and we just didn't quite get there.

Then there was a side track of work looking at other methods. This is where the refiner came into being. Here's a little bit of the secret lore: there was a version of SDXL that was never released. It had a different architecture and didn't work quite as well, but it was trained pretty far along, even through the post-training stages. The refiner was that model. It's a full diffusion model.

Andreas Blattmann

I actually completely forgot about that.

Dustin Podell

The model had a little bit better performance on high-frequency details. It just had very poor structure. The idea was: what if we took a different model for a different job? We did a lot of work to push how well it could refine high-frequency detail, and then we made a pipeline. Together, this became a more performant system.

Andreas Blattmann

Which is actually interesting. It was one of the early pioneering works that looked into what properties are learned at which part of the diffusion trajectory. The structural things are learned closer to the noise, and the fine, high-resolution, high-frequency details come further down, closer to the clean image.

Today, there are a bunch of different works that analyze this. You regenerate from low frequency to high frequency as you go from noise to a clean image along this trajectory. But that wasn't really known then. It was informally known, maybe a bit. This was one of the first works to exploit it and say, "I dissect this trajectory, and I have two experts: one for the structure and the next for the texture."

Dustin Podell

Yeah, and right now we have modulation, which conditions on the time step to account for that in the architecture and stuff like that. But that was one of the first works to make this distinction. And, as you see, it worked quite well.

I guess the reason I still hold some frustration is that, at the time, I wanted this one model to be so good at everything.

Andreas Blattmann

It was a cop-out.

Dustin Podell

Maybe it was a little bit of a cop-out. Nowadays, I take a bit of a different view, because we see systems of models do incredible work together.

Andreas Blattmann

But at the time, I remember feeling this old frustration from back then.

swyx

Well, it’s also very fair, because you should always go with the simplest solution that works. If you need to understand at what point you need a machine learning approach, it’s just not always clear.

Andreas Blattmann

A machine learning approach. Yeah, exactly.

swyx

When it comes to scaling, whatever the simplest solution is that works, you should go with a bit of Occam’s razor in statistics.

Well, Runway was also doing Gen-1 and Gen-2. Anastasios said on this podcast that Gen-2 was a pipeline mix.

Andreas Blattmann

Oh, okay.

swyx

Like when they—

Andreas Blattmann

No, if it works, it works.

swyx

They just didn’t admit it, whereas yours was public, so you had to admit it.

Andreas Blattmann

Yeah. I think if you go back to the time, people coming into the space today haven’t seen 200 or 500 steps of diffusion. As you say, you have a background, and then minutes later you have little fine details coming in. It was also very tinkerable. This came out as a 3B model that works on 8 GB of RAM.

As much as you could follow the bitter lesson and do the general approach, if you’re not a lab and you’re just doing this at home, you have one or two GPUs, so you’re going to do pipeline stuff.

swyx

Yeah. That’s how you can see the differences right there.

### Bringing Diffusion to Video

Andreas Blattmann

Yeah. I’ve been working on video for quite a while. Actually, before Stable Diffusion, I was working on video generation for NVIDIA. I had a paper there, “Align Your Latents,” where we basically mapped the latent diffusion algorithm to video. That was the natural next thing to do.

It was very exciting to see things moving. You can imagine the compression reasoning behind why Stable Diffusion was so efficient compared to other models at the time. That also holds for video, where you have this temporal dimension that you can compress. So that made a ton of sense.

I worked on this first for NVIDIA, and then, when we joined Stability, after SDXL, we worked on Stable Video Diffusion. That was the first application of this latent diffusion approach to video. Importantly, that work was about investigating the effects of data filtering, which I felt was underappreciated in research, and still is.

When we think about what these models are doing, effectively, we’re estimating a probability distribution, and the dataset gives rise to that probability distribution. If you have the perfect algorithm, the best algorithm in the world that learns anything you want, but your data is very bad—if you have blurry videos or images on average, and not a lot of motion in the videos—your model will learn that, no matter how good your algorithm is. What comes out of the model will be very bad.

The data is one of the most important levers you have when you want to train these models. In the research field, people didn’t talk much about data because a lot of the research community worked on benchmarks and tweaked the algorithms. But with frontier models, we know by now that data is extremely important. We wanted to share more context on that. We investigated how much data—

swyx

You have this trade-off: quantity versus quality, basically. Where’s the sweet spot? Is there a chart we should pull up?

Andreas Blattmann

Coming, coming. The most exciting experiment we did wasn’t the data one. If you go down a bit, we showed that pretraining on plain 2D videos lets you fine-tune an explicit 3D model extremely quickly. If you go to the last paragraph of the paper, here are the 3D views of the model.

What we did was take this video model that we trained, and then we had, I think, 1,000 fine-tuning steps on 3D data, and we got a 3D model out of it. That comes back to the question of what you actually need to understand the world in 3D. Only 1,000 steps of fine-tuning shows that this video model, although trained only on plain video, actually had an insane amount of 3D understanding. You just needed a bit of focus to get a 3D model out of it.

That was one of the early experiments where we saw that you can get world understanding through training on plain video. From a research perspective, that was the most interesting experiment in this paper. It’s also a pretty short paper. There’s a section here where you explain the pipeline: image pretraining, video pretraining, video fine-tuning, and then 1,000 steps to make it 3D.

swyx

Crazy. At the time, I don’t remember what I did, but there was this temporal consistency problem. If you just stretched out diffusion and tried to stitch frames, it wouldn’t work. I had a very old hackathon project where I forget which method we used, but we fixed the temporal consistency and scaled it out to 15 or 20 seconds. It took forever, but it was a very hacky solution because you could do this in an afternoon.

Critically, you need this kind of learning over the temporal dimension. If you just stitch together a very fast diffusion model, you get flickering. Even if you fix the seed and do a ton of work on the effects, you realize that today everyone is using transformers. In the transformer suite, you need attention over the temporal dimension to learn from it and understand what consistent motion looks like.

As far as I understand, we asked this about the Grok Imagine part as well: there’s no compression on the video side that’s useful, right? You just have frame, frame, frame.

Andreas Blattmann

Exactly.

swyx

Which is the simplest version.

Andreas Blattmann

If you look at the most advanced MP4 codecs, they have very smart compression. But that is lossy, obviously, and you need to connect the frames and extract what consistent motion looks like.

swyx

I don’t think any video model exposes how far back you look in temporal attention, because sometimes you need to look back longer, or it needs to be more adaptive. For example, if something gets occluded and then comes out again, you want it to be the same. But how long is it occluded?

Andreas Blattmann

That principally depends on the problem you want to solve. If you want to create a long-memory model that can create virtual worlds, then you need to find a way to look back a very, very long time. That’s super memory-hungry, and I think it’s one of the most exciting research problems to work on these days. Long context is extremely important.

swyx

I don’t know if there’s more there before we cut over to Black Forest Labs. Anything else in the Stable world?

Andreas Blattmann

There’s SD3.

swyx

SD3. That was the last paper you did before you—

### SD3 and Flow Matching

Andreas Blattmann

Yeah, SD3 was a pretty big paper. It was a culmination of our work at Stability, I would say.

swyx

Yeah, absolutely. This is your graduation thesis.

Andreas Blattmann

If you want, yes. This paper was really about understanding which part of the trajectory from data to noise the model learns which properties, and where we have to spend most of our training computation to maximize the model’s performance.

Obviously, it was also the transition from classic diffusion to flow matching, with a slightly different paradigm. We call it rectified flow.

swyx

This was also the first time we used diffusion transformers, right? With the—

Andreas Blattmann

Yeah, and then we took it a bit further.

swyx

A lot of changes, going from a convolutional architecture to a transformer architecture.

Andreas Blattmann

Changing the probabilistic model from diffusion to flow matching is much simpler mathematically, which is super nice.

swyx

I think you should actually define some of those terms for people who haven’t heard them. Latent diffusion, flow matching—what are they? How did you come up with these things?

Dustin Podell

Well, he didn’t come up with flow matching.

Andreas Blattmann

Exactly. We didn’t come up with flow matching. We just saw this algorithm as something that is more general because, with diffusion, one of the 2 distributions you’re translating between always needs to be a standard normal. That doesn’t hold for flow matching.

Instead of seeing the problem of transforming one probability distribution into another as the solution to a stochastic differential equation, you see it as the solution to an ordinary differential equation. That simplifies everything mathematically. It also makes fast sampling more feasible. Once we’ve trained the model, we want to make as few steps as possible to sample as fast as possible.

Some people would say flow matching is actually a special case of diffusion. Others might say it’s a generalization of diffusion, because you can translate between 2 distributions no matter how they’re shaped. You don’t need one of them to be a standard normal.

In general, you can see it as a model that learns a vector field that translates between 2 distributions. Once the model has learned that vector field, you can integrate it numerically, which is as easy as solving an ordinary differential equation with a forward Euler step. It’s similar in many respects, but mathematically easier and more general in the kinds of probability distributions it can translate between. That’s why we went with it.

A third reason is that it was easier to derive a fast model from flow matching. It was much simpler and more scalable, much easier to work with, and it had this nice performance benefit as well.

swyx

I don’t know. The diffusion space got quite complicated. Are we satisfied there’s no loss? Is it lossy in acceptable ways?

Andreas Blattmann

Lossy? Do you mean—

swyx

No, loss compared to full diffusion.

Andreas Blattmann

No.

swyx

All right. So then it’s good. It’s a strict improvement.

Andreas Blattmann

Exactly. It is an improvement, definitely.

swyx

It’s effectively just drawing a line between noise and data.

Dustin Podell

Yeah. It kind of took this complex space of how you map between these 2 distributions, as Andy was saying, and simplified it down to the simplest thing: drawing a line.

swyx

Because you’re trying to take a shortcut. What if it’s a bumpy distribution?

Andreas Blattmann

Yeah, that’s the thing. One of the most important parts of the paper, on the probabilistic modeling side, was looking at the trajectory from noise to image and asking which parts really matter and which don’t.

We train this in steps. We always train on one step from data to noise, and then we approximate this direction in flow matching with the model. If you sample the steps between data and noise uniformly, you might spend a lot of time optimizing for things that don’t matter.

One central part of the paper was asking where we should spend our compute. Which parts of the trajectory from data to noise carry the most important information? That’s where you should spend most of your compute during training, oversampling those parts and undersampling the less important ones.

It’s a further exploration of something we talked about with SDXL, which was the refiner. We spent time training the refiner at the end of the trajectory to learn high-frequency details. This was more of an exploration of the entire space of where you should spend time to make the best general model to learn the data.

swyx

Amazing. One thing I picked up with SD3 was text in images, which you guys showed off a lot. There was a time when images just couldn’t do text, then they did more and more, and now it’s, “Here’s our nice typography.” Can we attribute that to the diffusion transformer?

Andreas Blattmann

I don’t know if I could say it was the diffusion transformer.

Dustin Podell

I think it really was just better modeling—

swyx

Better data?

Dustin Podell

It was better modeling and better data. As much as I want to attribute it to some fancy thing, it really was just that it can model the world better. It’s a data thing.

Andreas Blattmann

It’s also about what you’re training on. For everything you train on, you need a caption. If your caption doesn’t describe the text that’s visualized somewhere, your model won’t learn it.

Sometimes we spend a lot of time talking about the nitty-gritty details of an algorithm. In the end, the better lesson is—

swyx

Right.

Andreas Blattmann

—more easy data—

swyx

Yeah.

Andreas Blattmann

—is almost always the most effective. You can obviously scale in a smart way, and if you spend tons of compute, you should spend it as efficiently as possible and only where it matters. That’s also at the heart of this paper.

swyx

I think prompting also got a lot easier at this time.

Dustin Podell

We used to have the 4K—

swyx

Prompt extenders. The 4K upscale, Ultra—

Dustin Podell

Our solutions were massive, with all the lists.

swyx

8K.

Andreas Blattmann

Yes, yes, yes.

swyx

Greg. Who was the Greg guy?

Dustin Podell

Greg Rutkowski.

swyx

Oh, Greg Rutkowski, yes. I wonder how he’s doing. I hope he’s doing well.

Dustin Podell

I hope he’s doing well.

swyx

And for context, this was when DALL·E 3, Midjourney, and Ideogram V1 existed, and nothing was open. Anything you’d see at this quality was either old Stable Diffusion or a closed portal.

Then you made the big break to Black Forest. What was that whole period like? You had to leave and start a lab from scratch.

### Founding Black Forest Labs

Andreas Blattmann

It was probably one of the most exciting times we all had. When you’re in stealth, you’re a small team. We were 13 people when we started.

Dustin Podell

14, yeah.

Andreas Blattmann

We first went to France and had a fun little week in a chalet. We spent the entire week hacking and hanging out.

swyx

I remember that. It was fun.

Andreas Blattmann

Dustin figured out some really cool stuff to make our models much more consistent. I remember that. It was such a fun week.

Dustin Podell

That was a magical week. That really was our founding offsite.

Andreas Blattmann

That’s when we started training Flux.1, which 3 months later resulted in this nice company launch with a model that I think blew pretty much everyone away. It blew us away too.

swyx

Oh, yeah. Commercially, what this directly unlocked, after having incorporated the company for only 3 months and then releasing this model, was received really well. That was amazing.

Andreas Blattmann
Yeah.

swyx
That was a lot of fun. Did you know that when you started this? You had to get all the clusters together and all the data. You had to start from scratch.

Andreas Blattmann
We had to start from scratch, yes, but we knew what to do. Obviously, we had a lot of experience.

swyx
You’d just done SD3. Did you have a hypothesis about what you’d do differently?

Andreas Blattmann
We’d obviously gathered a lot of knowledge over all that time, even before Stability. That all culminated, and we also had a team that was all in one place. We were very focused, which had never been the case before, because people were working from very different parts of one company. Also, as Dustin said, Dustin wasn’t actually on our team before.

We had a super nice team together and spent a lot of time discussing how we’d design the learning process for this model. That happened at the offsite, and then we just kicked off. That resulted in this model.

Yeah. Was it discussions of what FLUX.1 would be, or was it broader, about where things go in a few years? How much of this vision was part of it?

Dustin Podell

I guess I can jump in. One of the reasons the company was started was that there were some frustrations at Stability. We saw the impact our models had made out in the world—not just in the open-source world, but also in companies that were started, run, and quite successfully using our models. We knew there was a lot of demand and potential in what our models could do.

Unfortunately, I think we also felt at Stability that we couldn’t fully capture the value of what the models could be. We believed in our team’s ability to do the research, build these models, and get them out into the world. This would be our shot to correct the inefficiencies or mistakes that had kept us from capturing the value of the models themselves.

That was a big part of why we thought we could move forward. We had a track record of doing this and making an impact, and we figured we could continue and build a successful company out of it. I think it’s gone pretty—

Andreas Blattmann

I think it was a good thesis.

Dustin Podell

Yeah, it’s gone pretty well. I think it was a good thesis. So—

swyx

As I’ve said before.

Andreas Blattmann

Yeah.

swyx

Well, what use cases were you seeing? A lot of people went straight to Hollywood. “That’s our goal.” Imad used to say, “We will make Game of Thrones Season 8 again.” I’m trying to remember the name of this company, but this might even have been before SDXL. For a week or two, it went viral: anyone could take a picture of themselves, and it—

Dustin Podell

Yeah, the older version of Stable Diffusion, right? The older version. It used the old DreamBooth-style fine-tuning, which was extremely inefficient.

swyx

But it blew up. Everyone was using it, and everyone was excited about it. The models could make so many things, and we saw a lot of potential in how people wanted to use them. We saw where the models were going. At the time, they couldn’t do typography. We knew typography would be important, and we eventually got there with SD3 and beyond. There wasn’t just one use case. We saw this as a general technology with a lot of potential that would continue to grow, and that potential is now expanding much more into the physical world.

Andreas Blattmann

This was already the potential we saw.

Dustin Podell

Yeah, absolutely.

Andreas Blattmann

BFL wasn’t founded to be an image-generation lab. The FLUX movement was so good that we capitalized on it, too, because it gave us a kickstart. But when we pitched the company to investors for the Series C, it was clear we wanted to work on video. We see that as—

swyx

Yeah, I think we shared this on a podcast back in 2023, when we were still at Stability. We were saying video modeling is actually the central means to understanding how our world works. As I said earlier, we humans first observe the world, then start interacting with it, and through that we learn so much. That was always the thesis.

Then we had to take this detour through image generation, which was very successful. You always face the issue of where to spend your time. Unfortunately, the day only has 24 hours. But FLUX.3 is really where we saw this going for a long time, and what we wanted to be doing for a long time. That’s why we’re so excited to have the model out there, able to create videos, images, and audio, and act as a robot brain in real manufacturing workflows.

There’s a lot of stuff still coming this year that we have to announce on the physical AI side. As Dustin said, our goal was always to create the most general technology. We see that happening now, which is why we’re so excited.

Yeah, I think you also took the right paths to stay focused. If I’m not mistaken, FLUX.1 launched with xAI. It was their image-generation model.

Dustin Podell

That’s right.

swyx

Good partnership. That’s a good way to solve the question of whether to build distribution yourselves. Even with the latest FLUX.3 Max, post-training is with fal. They do a lot of inference distillation. They’re good at that, right? It’s interesting.

But it seems like you’re expanding now. What’s the latest one with robotics?

Andreas Blattmann

Flux Reaction.

swyx

Yeah. Before we get there, there’s one more I want to cover.

Andreas Blattmann

Yeah, that’s okay.

swyx

Image editing in context.

Andreas Blattmann

Sure.

swyx

There’s also a really nice history on the website.

Andreas Blattmann

Yeah.

swyx

It’s weird that we’re now at the stage of the company where things just happen, and I’m like, “Oh, we have this.” I was kind of sad FLUX.1 wasn’t on there to view, since it was the original one.

Andreas Blattmann

Yeah, it’s fine.

swyx

Pro looked good. Even just the visuals here—maybe someone will clip the progression from original Stable Diffusion into XL and through all the papers.

Andreas Blattmann

You see the progression, right?

swyx

Yeah. But one thing I want to cover at some point: I no longer personally see the progression. You show me state-of-the-art image and video models, and a lot of Chinese labs are putting them out. It’s hard to evaluate. I know people care. I’ve talked to Summit at BFL about how people in the industry aggressively A/B test, and there’s a lot they care about in evaluations. But there’s also a level where it all looks very realistic to me.

Andreas Blattmann

The rest is a prompting skill issue.

swyx

I think there are a lot of little nitpicks. There were issues with hands, and there are still issues with text sometimes.

Andreas Blattmann

There are still hand issues?

swyx

No, there were hand issues. There are issues with perception and layout. I’m redesigning my living room, and it’s horrible.

### The Case for Image Editing

Andreas Blattmann

Controllability is a huge thing. I think controllability is the thing we need to solve at large scale. There are so many ways you can control these models. Maybe that’s a good segue into FLUX Kontext, because that was a pain point we saw for a long time. People were training so many LoRAs just to get character consistency into models.

swyx

Yes. Shout-out to Replicate, with all the LoRAs back then.

Andreas Blattmann

Exactly. Then we had this amazing pitch from Dustin. It was at Northside, in France, I think. It was another cabin. A lot of things happen in cabins. I think we should go to the cabin.

swyx

You should do more.

Andreas Blattmann

Then we had this pitch from Dustin, who saw the potential: “I need to get this context. I need to be able to prompt not only with text, but also with images for image editing.”

Dustin Podell

Yeah. I originally—

swyx

That made pre-hype as well. I’ve heard the story that FLUX Kontext was you and one or two other people.

Dustin Podell

For a long time, it was basically my side project. Then I pulled one of our data guys into it a little and said—

Dustin Podell
“Come, we’re building this thing.” For me, as Andy was saying, it originally started as a pitch of mine at a cabin in France. I like to do these presentations every offsite, or every time we get together, where I try to look at the state of the field, where I think it’s going, and what we aren’t doing yet. What’s something these models could do that’s novel and useful, but that we might not be looking at right now, or that isn’t solved?

I did a presentation on what I called the “omni model”—the most overloaded word in the world at this point. There was a lot of research out there at the time on other editing models. If we remember back, there was the old Prompt-to-Prompt, and then there’s—why can’t I remember the name of this? The pitch was essentially that these models are clearly learning deep relationships about the world, and that’s going to keep coming up. It’s about these relationships. A Coke can is sitting on top of a table; this bottle is sitting here. You prompt the model, “I want a photo of a dog wearing a hat,” and it needs to know that the hat is on the dog. It needs to have this general understanding.

My theory at the time was that since the model already has all this spatial knowledge, if we build the right data and train an already pretrained model the right way, we should be able to evoke general image editing with very little training, capturing these prebuilt relationships: when things are there, when they’re not there, and how these spatial relationships inside an image work.

Then it became a three- to four-month project. I don’t remember the timeline. I was here in our San Francisco office with the one data guy I stole, building out early test training datasets. I remember the moment I saw that it was clearly working quite well. We made an effectively tiny dataset of very high-quality labels describing simple relationships in an image, and that was really it. It wasn’t a way you would prompt a modern edit-instruct model; it was just literally describing these relationships.

I trained the model for maybe six hours, maybe four hours. It was a very short training run. All of a sudden, I saw what felt like clear generality coming out of it. I think it was a photography dataset, just photos of stuff, but even from that I was seeing that I could put a logo on a shirt. I could do things like that. It wasn’t good, but the model was clearly learning to attend between these 2 concepts and use these pretrained relationships.

Then I took that to our next offsite, showed the results, and it became a more company-wide project for a couple of months. We really put effort in and brought people in. It was mostly a dataset project. At the end, there was a lot of very nice distillation work by Axel that sped it up. Then we were able to put this model out into the world, and I was quite proud of it.

swyx  
That was the first model to really do character-consistent editing at high speed, right? That made it accessible to so many people. For most people, it’s very common to generate something, chat with it, and change it. Before this, image generation took quite a while, and you needed to recreate the image. You couldn’t just change the parent to be blue. You wouldn’t get the same character.

And you’d need to mask something you didn’t like and fill in the blank.

Dustin Podell  
Yeah, I just wanted to get away from the slot machine, essentially. Everyone had to do the slot machine.

swyx  
Yes. Prompt and prompt and prompt, and it was very clear that iterative editing was a much more effective way to do this.

Dustin Podell  
My crappy ComfyUI was that: it just slopped out 10 different versions, and you could pick.

swyx  
I still slop out versions. I’m changing things, living in my “generate 4, generate 6” routine, redoing something. Just make 10 iterations, and eventually one is good enough. My favorite is that you pray to RNG Jesus because it’s a random number.

Dustin Podell  
To me, this concept of what a foundation model is is something like this: you know all these little workflows you do to do this and that, and they all use different tools? We’re going to replace that with one model.

swyx  
And that was where Flux.1 Kontext came from. This was a very clear “aha” moment for me.

Dustin Podell  
Yeah.

swyx  
I never tried Prompt-to-Prompt, but to me the closest thing to Flux Kontext is Qwen Image. The Qwen folks are doing really good work. They had a Photoshop layer model.

Dustin Podell  
Yeah, it was incredible. They’re doing pretty cool stuff over there.

Andreas Blattmann  
It’s interesting how much of this just comes back to there being a lot of good data.

Dustin Podell  
Yeah, it was targeted good data. I don’t know how to describe whether it was a good concept, but yeah.

swyx  
One thing I’m surprised by: Japan had this whole law where they would protect you if you trained on copyrighted material in the country. Everyone was like, “Ooh, Japan.” It’s interesting—nothing happened.

Andreas Blattmann  
I always wanted to visit that office. I was sad I never did.

swyx  
On your omni point, it was an interesting time. Gemini is obviously the latest omni model. ImageBind, which I didn’t realize was from 2023, was from Meta. They threw in embeddings for heat maps and depth.

Dustin Podell  
Heat maps.

swyx  
Depth. That’s a big one. I think this work has progressed; there’s definitely newer work on it.

Andreas Blattmann  
You can think of it as just another color, right? It’s RGBA, whatever that is.

swyx  
Yeah, yeah.

Dustin Podell  
The fun thing was that it was all in, all out, so you can go from text to heat map if you want. You can go from IMU to depth. It’s a big market for that.

Andreas Blattmann  
Interestingly, there was no video out, which is the central thing we need to understand the world better. I think that’s always been at the core of what we’ve been doing.

swyx  
Okay. Before we leave the data or IP discussion, is there a particular kind of data that everyone in the industry knows is valuable? If I have a data vendor listening, how do they sell to you? How do they approach you?

Andreas Blattmann  
It depends heavily on what type of problem you want to solve. If I train a robot, I obviously need a lot of egocentric data. That’s not a secret, right? That’s just how this thing will be.

swyx  
Egocentric, or—

Dustin Podell  
There’s also a benefit for you guys, right? On the language model side, there are a lot of people doing RL environments. Is there a parallel where people should come to you with that kind of stuff?

swyx  
I’m trying to get you vendors.

Andreas Blattmann  
We have so many data vendors approaching us, actually.

swyx  
Too many?

Andreas Blattmann  
Yeah.

swyx  
Okay. What’s good versus bad?

Andreas Blattmann  
Again, it depends on which stage of training you’re at and what your effective goal is. There’s a trade-off between the scale of the data and its quality. For every training run, for large-scale pretraining, you have to find that sweet spot. Depending on the architecture you’re using, that might look very different. That’s one of the central pretraining experiments, or pre-scaling experiments, that you need to do. You need to do a ladder to understand at scale how much data you actually need, and to predict how the quality should look. Where is the sweet spot between these 2 dimensions of data quantity and data quality?

swyx  
Any other major jumps we want to call out? Flux.1 Kontext was a big one.

swyx

Flux.2, Flux Tools, Action—we talked a bit about those. Any other things?

Dustin Podell

I mean, I think the 3 sizes are very important: the Dev models and Schnell.

swyx

Schnell?

Dustin Podell

Schnell means “fast” in German.

swyx

I think you probably covered that in a blog post. But I think having open source, closed source, and a range of performance makes sense.

Dustin Podell

Yeah, absolutely.

swyx

If there’s anything to track, the one thing that made me quite sad about Flux Kontext is that it was only single-image editing. It was clear there was a huge need for multi-image. People want to put in many different images and relate them—

Dustin Podell

References is a better word here.

swyx

Yeah, many different references. I think Grok is king right now, though I don’t know the current state of it. I know Midjourney does quite a bit with style books, where you can build up references.

Dustin Podell

They’ve always been very good at style reference, but this is more general. Flux Kontext would only take one image, so you could only iterate with that. If you had a bad photo of one person, that was all you had.

swyx

But you could have 3 different photos of the same person.

Dustin Podell

It was actually general enough. I remember it was quite funny: people were stitching a bunch of images into one image to emulate it. With Flux.2, we were finally able to spend the time to build out the data and do larger-scale training for handling, I think it was 10 reference images at the time. That gave you much more context to work with.

And, to a large extent, that’s also a data problem. You need the data to have all these references.

swyx

Is there a practical limit? Let’s say the industry standard is 10 or 18 references. That’s it, right? I don’t need 100.

Dustin Podell

Obviously, you can show crazy numbers for how many characters or images you can get into one image, or video. But then you need agent work to get me 100 references.

swyx

Which is the easy thing, right? It’s agent work. But then I have to check it. I have to review 100 images to see if that’s actually what I wanted.

Dustin Podell

It’s also a hard creativity problem. Where should I put these 100 images?

swyx

Composite them.

Dustin Podell

Without over-indexing on the number, anything over 10 references is hard for a human to follow. If we’re creating a video, I can put in 50 people. But in the most important movies, how many central characters do we have? Five or 6 maximum, unless you’re doing Avengers.

swyx

Exactly.

Dustin Podell

At some point, that approaches the limit of what we humans can process in a 2-minute clip. This also connects to the evals you mentioned earlier. Building evals for larger contextual models becomes a much bigger process. We ended up releasing an open eval set called ContextBench, built off the internal benchmark we use to evaluate context. As you scale up the number of references—

swyx

I haven’t seen this.

Dustin Podell

Please look up ContextBench. I’ll pull it up. It was a lot of work, but it was fun to put together.

swyx

“Make the cat very fat.”

Dustin Podell

I remember sitting in the back room of our San Francisco office with a few of the other guys. Tim Dawborn and I were laughing.

swyx

Do you just take some edibles and go, “Dude—”

Dustin Podell

We were sitting there late at night, literally just pulling images and typing in the stupidest things.

swyx

It works, right?

Andreas Blattmann

But you need to make one step to make it enjoyable, and you need to decide what the ground truth is.

Dustin Podell

This is just a benchmark, basically. It’s just inputs.

Andreas Blattmann

Right, but how do you evaluate the output?

Dustin Podell

That’s the interesting thing. You ask people. This is a public benchmark to give everyone some arena for improvement.

swyx

I see. So this is half of an eval.

Dustin Podell

This is the eval. You have 2 models you want to compare, and then you judge.

swyx

Arena style, yeah. So it’s not a validation set in that sense. It’s just an eval benchmark.

Dustin Podell

This is just me sitting in a room. We removed some of it. Genuinely, 80% of this is that. It’s also heavily influenced by our Italy offsite.

swyx

Why don’t you do Mad Libs, a combinatorial explosion that’s more structured than just random inspiration?

Dustin Podell

We definitely do more things like that now. We had to filter some things out of the public set because there were some very dumb things in the internal set.

I spend a lot of time looking at how people actually use our models. I’m someone who really likes to use our models. I think it’s very important that you test and play—really just play with your models. I came from Discord and open source communities, where you see people doing the stupidest things, but people are having fun with it. Besides covering actual use cases, including production use cases, I want to make sure the models are fun to use.

swyx

Good vibe eval—vibe bench.

Andreas Blattmann

I underappreciated this 3 years ago. Yes, it’s memes, and it’s a Midjourney subscription or whatever, but it actually becomes very real work.

swyx

Because they do it for fun, then they think, “Oh, I can do this at work.”

Andreas Blattmann

Exactly. When you automatically evaluate these models, there are a bunch of human evals everyone is doing in the field, but every evaluation has blind spots. You should combine as many independent evaluations as possible to get the best picture of how the model is performing.

That doesn’t remove the burden from you to sample the model yourself. It’s actually super fun to try to understand where the strengths and weaknesses are. Constantly sampling and checking the model is a core part of developing it. Is it where I’d expect it to be?

swyx

At some level, it’s what you guys care about too, right? These are your use cases. If you want something to get better, make a little benchmark and see how things perform.

Dustin Podell

Absolutely. I think it’s very important that the people building the models actually want to use them.

swyx

There’s a big distinction between that and a microphone brand trying to use it for product photos—a very niche use case they can min-max on.

Dustin Podell

It’s very general.

swyx

One thing I do question is Midjourney’s well-known loop: on Discord, everyone sees other people prompting. You get 4 options, you choose one, and that’s feedback for your next model. How come you didn’t do any of that?

Dustin Podell

We didn’t do it publicly at scale.

swyx

But isn’t it valuable?

Dustin Podell

It can be valuable, but only up to a certain extent. If you cross certain limits, the signal you get is pretty weak. In the early days, we saw the insane potential of that tuning on Midjourney’s models, like V5. But once your base model reaches a certain level of strength, you don’t get those super good examples where you can clearly see the model improving from this. There’s also a lot of noise.

Andreas Blattmann

There are a lot of other ways you can do this. I had this debate with someone: back then, it was very useful because prompt adherence wasn’t great. It’s pretty judgmental to decide which one follows the prompt better when neither of them does it well. You get a lot of preference tuning because there’s very high signal: people prefer this one.

Now, back to my earlier point: if I take the top 5 models and tell them to remove a bed, they’re all going to look the same to me. There isn’t that much signal in the preference, right? They’re all filling it in pretty well because they understand this task very well. So that side of the evaluation is kind of gone for me. It all looks fine.

swyx

I was going to jump in and say that was actually one of my favorite parts of moving into Kontext. Finally, it wasn’t just, “Did it actually do the task?” I really enjoyed getting to these very clear, concrete tasks.

### Autoregressive or Diffusion

One more part of the text tree, and then we want to go into robotics in action. Autoregressive image generation seems like a big trend, with both Gemini and ChatGPT doing it. Is that as big an impact in your mind as it is in ours? It seemed like a big change because then you could edit conversationally. It’s the next obvious step from Kontext: instead of editing with an image and a text prompt, you’re working conversationally with a model.

Andreas Blattmann

I’d say that’s not a feature of autoregressive models alone. It’s based on the data you train on. You can get language capabilities into a diffusion model as well, which we know is working. In the long term, this is where we believe things will go, and where we’re working toward getting more modalities in, including language. So it’s not a question of the algorithm; it’s a question of what you focus on. I think both approaches are extremely exciting and have a lot of strengths and shortcomings.

I’d say autoregressive models are much more data-efficient. You can train them better on data because, for diffusion models, you have this trajectory from data to noise. Every training example effectively gives rise to infinitely many training examples, because you can pick any possible point to denoise from along the trajectory between data and noise. Obviously, the model would overfit if you trained only on that one example. Theoretically, one data point gives rise to infinitely many training examples. Autoregressive models, on the other hand, can train on all the tokens in parallel and get the maximum gradient from each training example.

That’s the training side, where I’d say autoregressive models are more data-efficient. At sampling time, though, autoregressive models have to generate one token after another because they’re iterative in the data direction. Diffusion models can distill 50 steps down to 4 and speed things up a lot. That isn’t possible in the same way with autoregressive models, because they have to generate token after token.

swyx

You make it sound like autoregressive generation is just worse, then.

Andreas Blattmann

Autoregressive generation has a limitation in that respect, but diffusion has a limitation in terms of data efficiency and training. I like both approaches. We’ve also been working on autoregressive models for a long time, and both are super exciting.

swyx

So, on data efficiency, autoregressive models win?

Andreas Blattmann

Yeah.

swyx

Okay. So, scale. One of the other big things you hear from people who’ve worked on both ChatGPT and Imagen is adding in-loop RL, so models can act agentically: generate something, reason about it, edit the image, and continue that workflow. They can do that too, and this is separate from pipelining diffusion generation, right? It’s all the same model doing it, trained at the data level, so you can scale that. Is there a counter to this in diffusion?

Andreas Blattmann

There’s no inherent limitation that would prevent a diffusion-based model from doing the exact same process and all of these things.

swyx

That’s really blowing my mind. I had the wrong idea. Nano Banana happened and I thought, “Holy shit, autoregressive is a thing.” But you’re saying it’s not. Am I wrong that Nano Banana is diffusion?

Andreas Blattmann

I think it is diffusion-based. I don’t have insider knowledge, so I might be wrong, but I’m pretty sure it has the language component to do autoregressive things. I think what we’d call it is a Transfusion model. It can essentially do both. There’s no inherent limitation saying you have to do it one way or another. These are explorations.

swyx

So it’s not settled consensus. I thought there was a wave of people shifting from diffusion to autoregression for Imagen 2.

Andreas Blattmann

There’s a lot of exciting research going on in that area right now. We’ve seen models work extremely well on that front. The same is happening with flow matching, which is also improving. I’d say the debate isn’t settled yet. There’s significant research on both fronts right now.

swyx

That’s exciting. The same is happening for robotics, too. We have VLAs versus VAPs, which is super interesting. The last thing is diffusion for text. There are benefits, of course, like speed. We see that clearly with text. People can play around with Diffusion Gemma, too. Will these models ever reach parity? Is it just a matter of investment in money and data?

Andreas Blattmann

I’d say a lot of it is data. If it doesn’t work, why doesn’t it work? My simple answer is that we reason left to right; we don’t reason by diffusing into vision. At the same time, I think people like to focus on the step-by-step nature of tokens in autoregressive models, but they can also parallelize. We can take in a whole book at the same time, too, which those models can do.

swyx

Yeah, you can fake it in. I totally get it. It’s a valid question: how come diffusion for text never took off? It’s slowly getting slightly better, but how?

Andreas Blattmann

I think it’s a matter of which approach came earlier and became the standard for language models. As you said, if it’s more data-efficient and the world has been doing a lot of scaling, that gives it an advantage. That doesn’t mean there aren’t pockets of use cases where you can get really good results at a certain size. Diffusion Gemma is a good example: it’s pretty good, meets a bar, and can run really fast. If you want that, great.

On the autoregressive language front, I think it really comes down to the fact that it was simple, scalable, and there early. It’s clearly been a very successful trajectory. A lot of time, money, and research have gone into making it work. There’s a lot of exciting research on the diffusion front now, and I’d separate the idea of diffusion language into 2 camps. A lot of people don’t realize this: there are discrete diffusion language models, which I almost don’t like to call diffusion models. Diffusion Gemma and the Mercury models are in this camp. They’re more like masking-style models, where they jump from a mask to a token, but they’re still in the discrete space.

Now there’s a big wave of research on continuous diffusion language models. Those are actually operating in a denoising space. I don’t want to call them flow-matching models, but they’re essentially operating that way.

swyx

So, more like in— This was used for audio, if I’m not mistaken.

Andreas Blattmann

Audio generation, continuous diffusion. Continuous diffusion. I mean, well, diffusion has also been used for—

swyx

It’s for everything, yeah: visual modeling, audio modeling. My last fun question there is: do we have a good way to do GRPO-style RL on pure diffusion text models?

Andreas Blattmann

Yeah, we’re doing this, actually. For FLUX.3, we did this, and it has had tremendous gains.

swyx

So, FLUX Action is part of the very recent launches and work you’re doing. How do you approach this? What’s the feedback and impact so far?

### FLUX Action and Robotics

Andreas Blattmann

Basically, you pre-train on videos, images, audio, everything together, and then you have this very general model that understands the world. Then you can take a bit of action prediction data. Basically, you add predictions in the state space of this robot, which is a parameterization of the robot arm. Each joint is described by a translation and a rotation: 6 degrees of freedom, basically. Then you predict that together with a video.

Since the model can already reason about the video and what it needs to create to fulfill a certain task in video space, it’s simple to add this new dimension, this state space, and predict that as well. Given a chunk of earlier video, you condition on that and enter an instruction. The model creates the next chunk of actions to fulfill that action.

This works extremely well. We can do it in the real world, with real-world robots, and in a simulation. Computer use is also a mapping of this to the digital world, and we can make the model play games, right, and stuff like that.

So this is all enabled by this single FLUX.3 model. You also see that it’s extremely capable because we have this very general pre-training, which isn’t limited to classic robot data. It uses all kinds of data that gives the model an inherent understanding of the world, and you see that in its performance.

swyx  
Being a diffusion model is also pretty speedy, as we see here, with these distilled versions, right? Would you call this a world model or a world action model? Do you have a term you like to use?

Andreas Blattmann  
It’s a world action model. I don’t like to use the term “world model.” It’s so overloaded.

swyx  
I’m in the same boat. It feels like a homonym at this point.

Andreas Blattmann  
Let’s call this a visually intelligent model because it can do all these basic things, right?

swyx  
I guess the distinction I see in these models is that you’re not just predicting video or the world; you’re also predicting action states, right?

Andreas Blattmann  
Yeah. One is what’s happening in the world, what the state is, but also what the next action to predict is.

swyx  
Exactly. So it’s sort of two outputs, right?

Andreas Blattmann  
Yeah.

swyx  
Is there a target frame rate that you aim for as an industry standard? Because you don’t have to predict that far out, is my idea.

Andreas Blattmann  
Do you mean in terms of the real-time factor? It’s 10 FPS, I think, what the model currently runs at. It’s a very small model.

swyx  
And you’re projecting out one frame, 5 frames, whatever, right?

Andreas Blattmann  
I would have to look it up. I think it’s a small chunk of frames, somewhere between 9 and 21 frames, that it predicts. Then the model is conditioned on the previous generations again, and on what the camera perceives when it runs on the robot. Then you predict further. So it’s a bit of a chunk. It’s not running autoregressively in time, so to say, but it’s chunk prediction.

swyx  
What I’m driving at is that this video prediction side is very much for predicting the results of what you do. So probably you need some kind of long-range planner outside that model to coordinate the actions.

Andreas Blattmann  
I mean, this is why we were discussing the language side earlier.

swyx  
Yeah. So as it runs right now, there’s no high-level planner involved or something?

Andreas Blattmann  
Yeah. This is just a model running on a loop.

swyx  
Some of the interesting things here: this is what we did with Dyna Robotics.

Andreas Blattmann  
Yeah.

swyx  
Dyna has one, Runway has one. I guess let’s talk about the fine-tuning side. If I want to take your general model and fine-tune it for a new robot with a new task, something out of distribution that it hasn’t seen—say, my own home robotics, where I can program in how to use a humanoid arm—what’s that process like? How long does it take? How important is fine-tunability?

Andreas Blattmann  
Since this model is so general and the pre-training was so good, you need a bunch of hours. I think we showed one example where we compared it with Pi0.5, which was also open source. It took 8 times less data to train on it. That was for an industry application. We needed 4 hours of data to get to a success rate that was good enough for this basic application.

swyx  
Is that something you want to help improve right now?

Andreas Blattmann  
Yeah. We go back to the context thought. We shouldn’t be doing fine-tuning. What we’re targeting and working on right now, extremely carefully and with a lot of focus, is in-context learning for robotics.

You should be showing a robot an action one time, and then it should be able to do it. This would truly be superhuman intelligence. If you look at how long it takes us as children to learn to walk, or to use our thumb correctly to grab this can, for instance, it takes weeks or even months. Our vision is that an autonomous robot should be able to observe any action once and then be able to do it. That would be clearly superhuman visual intelligence, and that’s what we want to reach.

swyx  
What’s the standard now? How many demonstrations do we need?

Andreas Blattmann  
Around 4 hours of fine-tuning, at least.

swyx  
Four hours, yeah.

Andreas Blattmann  
Even 4 hours is pretty good. I’d say 2027 will be the year we crack this task: single-shot learning.

swyx  
“Mid-training” is an interesting one here, too. Is this like going from manufacturing to, say, medical stuff, like filling pill capsules? Is that where—

Andreas Blattmann  
The model is trained quite generally. Even before fine-tuning, it might be able to handle many of the tasks you throw at it. This version might be able to handle many tasks. Fine-tuning is really just nailing it.

swyx  
In the robotics world, you want a certain level of accuracy you expect out of it.

Andreas Blattmann  
Exactly. This is just standard plain large-scale mid-training. Then we move on to the fine-tuning stage, where we add the high-quality robotics data and get a further bump. But you see that even with mid-training, the model can already do these tasks to a somewhat good extent.

swyx  
I guess the other interesting thing is that it’s 7B, right? Is that mostly because these things are deployed on device?

Andreas Blattmann  
Exactly. You want a model that’s as small as possible because it needs to act on device if you’re adding this to a manufacturer, and it needs to run in real time.

swyx  
And this is on the backbone of FLUX.3?

Andreas Blattmann  
Yes.

swyx  
Is the image and audio stuff all derived from this?

Andreas Blattmann  
All of it. This is derived from one model, basically. We have this multimodal model. That’s really important to understand. It was trained on a huge corpus of images, videos, and audio, and from that we can derive everything we released this summer: FLUX.3 Video, FLUX.3 Video Edit, FLUX.3 Action, FLUX.3 Image, and FLUX.3 Image Client. We also derived an upscaler from it. We have a really nice 4K video upscaler, which is super fast, too. I can only recommend it. It’s really good.

This is all powered by this one model, and it shows how training these very general models compounds over time, because you can derive all these applications from them.

swyx  
Is there a difference between a video upscaler and an image upscaler? Isn’t it—

Andreas Blattmann  
For a video upscaler, we talked about temporal consistency. If you had an image scaler with no temporal understanding, you’d get this flickering again. Especially in the high frequencies, you don’t see it much when you compare two images side by side. But if you switch between them, you notice it. One of the basic tasks everyone who looks at a lot of images does is opening two images in two tabs.

swyx  
You do notice, yeah.

Andreas Blattmann  
And then switching left and right, you see every single difference. That would amount to flickering in temporal space if you didn’t have this temporal component in the video upscaler.

swyx  
Got it. And that’s what makes the difference from an image upscaler. Any notes on real robot evals and how to scale this? I see “sponge in bowl” three times. At some level, if you look at our DynaPod, there are literally people testing robots one-off because you have to test at real scale. But come on, two out of three runs on “sponge in bowl”—just restart and do it again. You can be three out of three, right?

Andreas Blattmann  
It’s a hard problem, and I think one thing that’s constantly being worked on is closing the sim-to-real gap. This will get better and better over time. But to be honest, evals are painful no matter where you do them.

They’re also painful when you look at content creation evals. Ask three different groups of voters what a good image looks like, and you’ll get three different answers. You get wars over it. We have wars over it internally. Evals are just hard for generative modeling.

swyx  
Is the commercial side already developed? How developed is it?

Andreas Blattmann  
We’ll have more of these coming soon. I can’t tell too much, but there will soon be the first BFL-powered robots live in production.

swyx  
And it’s open, right? 2027 will be the year of robotics. Say more on predictions. Where do we go from this in 2027? How fast do things pick up?

Andreas Blattmann  
Single-shot learning. In-context learning is a big thing we’ll see first. We’ll see the first commercial applications of these things in real manufacturing. That’s why it’s good that we have a German base, because there’s a ton of industry data and lots of potential for partnerships. People are really hungry for robotics, too. There are good synergies that we want to use to make 2027 the year of robotics. I’m very excited about it.

The research is also moving very quickly, so I suspect robotics will improve very quickly.

swyx  
What about the fact that robotics has physical manufacturing constraints? You can scale up tech and models, but that’s pretty hard to affect. Predictions for 2027: where are we realistically on robotics? Can I buy something off the shelf that can single-shot learn?

Andreas Blattmann  
I suspect that later in 2027, people will be able to get some quite interesting robotics in their house. Whether it’ll be able to do fine motor tasks and all that, I don’t know. This is entirely predictive; I have no baseline for it. But I suspect next year there might be some interesting, potentially useful robotics in your house, at least for a niche of tech people like us—not the average person.

swyx  
And for manufacturing?

Speaker 1
I think 2027 will be a year for manufacturing.

swyx
Because manufacturing is more restricted than the long tail of autonomous driving or consumer robotics, where ideally you’d want it to do anything you want.

Speaker 1
There’s a very good chance we’ll see a lot of applications unlocked by robotics in 2027. That’s what we’re working toward, basically.

swyx
Awesome. Exciting. It seems like a very fun place. You guys have robot labs, an image and video model, and you’re building the future. You’re just missing the flying car.

You mentioned Germany a lot, but you also have a pretty big San Francisco office. I didn’t know there was a big European side.

Speaker 1
There’s a substantial San Francisco presence. It’s called Black Forest Labs. We invite everyone to come over to our place at 410 Townsend Street. We’re not a European company; we’re a company with roots both in Europe and the US, and we’re acting globally.

swyx
That’s fun. We got you on because you’re visiting town, and we thought, “Okay, we should have you on.” How can people help you? Are you hiring?

Speaker 1
I think just supporting us in marketing and talking about our models. Help us get our name out here in the US. In the community, we’re well known. In broader society, not that much. There are the big players, and that’s what we need to change in 2027: really getting our name out, because we have extremely strong technology. We have a long history of building the core pillars that everyone is using. It’s time to get our name out to a broader audience.

swyx
The tech community knows you, but beyond that, not so much. Tell your mom about FLUX.

I’m a marketing guy, and I help companies with marketing all the time. You guys are very different, because generative media in general has a mixed reputation with the broader population. Normally, I’d suggest a TikTok or Instagram marketing campaign, but you’d get killed because people hate AI so much. It’s interesting to find the niches where this isn’t regarded as AI. It’s just a normal thing you can do with tech. The moment you say it’s AI, the population will have pitchforks. Do you have any response, thoughts, or explanations about that?

Speaker 1
I think it’s important to emphasize that we take this responsibly. Our models aren’t creating avalanches of slop. They live in real use cases and professional workflows, and they help make professional people’s lives much easier and increase productivity so much.

swyx
It’s Photoshop.

Speaker 1
Exactly. It’s Photoshop. It’s crazy how much more productive you are. That’s really the goal on the content side. It’s not about putting people out of work at all. It’s about making everyone’s life so much easier.

swyx
I wouldn’t say that. If I worked on your team, this would be my go-to-market problem right now: how do we get people to go viral the way Wispr Flow does a little bit? People say, “I understand how to use this. This is not AI. It’s just Wispr Flow.” Or it’s Granola, same thing.

Generative media hasn’t had that crossover, except for Nano Banana, and whenever ChatGPT had the image generation Studio Ghibli moment. Those are the moments where people say, “This is a fun trend I want to hop on.”

Speaker 1
There were moments that hit well. Your point about seeing yourself when you’re older—that had a huge moment. People would look at themselves in their 80s and send it to their family. There have been moments, but nothing super predictable. It’s very hard to predict one of these viral moments. Nano Banana was their internal name, and they kept it because people liked the name and it stuck.

One big use case we saw a ton of was image restoration, which we didn’t necessarily expect. I saw amazing examples of people taking old, degraded family photos and restoring, colorizing, and fixing them up, while keeping a very high level of similarity to the original. That was one of my favorite moments: people showing their families these restored photos and seeing their reactions. I really enjoyed that positive moment. But, like I said, so often we can’t predict what the thing is going to be. Even back in the Stable Diffusion days, it was always surprising what the trend was online. They just emerge.

swyx
You just never know. They emerge.

Speaker 1
You cannot plan them. Once you have them, you need to exploit them, obviously. You should always be asking, “What’s the next big thing? What’s the next capability that models haven’t been able to do?” Then try to bake that into your model.

Speaker 1

Absolutely, yeah. Making bets. Some of them will work, and some others will not. But it’s extremely important to always have this mindset: “Hey, what’s the next thing we should be working on that will bring a lot of joy or productivity increases to people?”

swyx

Yeah. Well, that is the domain of research. You guys are doing an amazing job, so thank you so much for sharing your story. I hope more people come to know you as a team, because they’ve seen a lot of the model buzz. But I feel like this is also the first time I’ve heard the full story of BFL.

Speaker 1

Yeah. No, thank you so much for having us.

swyx

Thank you so much.

Speaker 1

It’s an amazing presentation. It’s fun to finally have gotten to do this.

swyx

Yeah. I mean, it’s a year in the making. I bought a bunch of MicroDucks, and I’m looking forward to them running around.

Speaker 1

Oh, do you have one? I want to get my hands on one. I feel like I missed the window when these were available.

swyx

You have connections. I’m sure I can—

Speaker 1

Whoever out there can get me a MicroDuck, get me a MicroDuck!

swyx

I’ll get you a MicroDuck.

Speaker 1

Okay, perfect.

swyx

I bought a bunch because I know this is going to be a hot gift for Christmas.

Speaker 1

Fantastic.

swyx

But, yeah, robotics in 2027—that’s one year away. We’ll see where this all lands. I’m very optimistic about this approach, and I think you guys have explained it very well.

Speaker 1

Fantastic. Yeah. Amazing. Awesome. Thank you.

swyx

Cool. Thanks so much.

Speaker 1

All right. Thank you.
