Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs
swyxAndreas BlattmannDustin Podell
- Black Forest Labs is extending a commercial image business into a broader visual-intelligence platform. The company reports $450 million raised and revenue above the $100 million figure published in December; the conversation does not define that figure as annual recurring revenue. Image generation supplied the commercial kickstart, but video and physical-world understanding were part of the original ambition: “we’re not pivoting,” “we’re just extending.”
- The central bet is that learning natural signals produces knowledge that transfers across products and into robotics. Blattmann describes FLUX.3 as one model learning multimodally from images, videos, audio and actions, supplying a backbone for generation, editing, upscaling and action prediction. He calls the target “System 1 intelligence”; an earlier video model’s conversion into a 3D model with roughly 1,000 fine-tuning steps helped convince him that ordinary video contains substantial reusable world knowledge.
- Controllable editing matters more commercially than another improvement in attractive samples. FLUX Kontext emerged from a small relational dataset and an initial experiment lasting perhaps four to six hours, then became a larger data and distillation project. Podell wanted to escape “the slot machine” of repeated generation; FLUX.2 subsequently expanded the workflow from one reference image to approximately 10.
- Data design and compute allocation repeatedly explain gains that look architectural from outside. Better typography required captions that actually described visible text, while SD3 concentrated training on the informative portions of the noise-to-image trajectory. The host invokes “the simplest solution that works”; the guests emphasize choosing useful data and spending computation where it matters.
- Conversational image editing does not establish autoregression as the winning architecture. Blattmann argues that autoregression uses training data more efficiently, while diffusion offers sampling advantages such as distilling 50 steps into four. Conversation, reasoning and iterative editing are not exclusive to either approach; Blattmann says “the debate isn’t settled yet.”
- BFL’s robotics proposition combines a small deployable model with less task-specific data. The 7B action model derives from FLUX.3, predicts video and robot actions together, and reportedly runs around 10 FPS. For one industrial application, Blattmann cites roughly four hours of data and “eight times less data” than an open-source comparator, while acknowledging that the next objective is learning from demonstrations without fine-tuning.
- The 2027 forecast is strongest in manufacturing and much more speculative in homes. Blattmann expects progress toward learning an action from one demonstration and calls 2027 “the year of robotics,” with production deployments approaching. His suggestion of useful home robots later in 2027 comes with “entirely predictive” and “I have no baseline for this”; physical deployment constraints and weak real-world evaluation remain unresolved.
1. Open weights recruited researchers as well as users
Blattmann traces the work to a physics-influenced group at CompVis in Heidelberg, later Munich. Diffusion’s connection between a physical process and generative modeling had “a philosophical kind of beauty.” The university’s latent-diffusion research became the foundation for Stable Diffusion; BFL’s Freiburg location subsequently reflected the founders’ roots near Germany’s Black Forest.
Podell’s route began with roughly 10 years in software development, followed by experiments with open-source generative models. Midjourney V1 made the direction feel consequential; Stable Diffusion made it accessible enough that he resigned from his Los Angeles software-studio job on release day. He thought the people working on it must be “arcane wizards” and wanted to understand their machinery.
Becoming a researcher was an accumulation of practical work rather than a formal transition. Podell joined Stability as an engineer, learned from researchers including Scott Hawley, who worked on the audio team, read papers, implemented experiments and obtained idle cluster access. His method was to “tear it apart until I understood how it worked,” supported by Discord communities including EleutherAI.
Swyx describes the same release as the reason he quit his job and started Latent Space: “everything I know about AI is wrong because I can run this on my laptop.” Blattmann recalls a striking geographic contrast: the release visibly redirected careers in the Bay Area, while much of the surrounding population in Germany barely noticed.
2. Image revenue supports a broader visual-intelligence ambition
BFL reports $450 million in total funding. The last published revenue figure, released in December, was $100 million, and Blattmann says the company has since crossed it. The hosts’ point is that the broader research program follows an already substantial commercial business; the guests insist that extending into other modalities was always the plan.
Blattmann distinguishes intuitive physical competence from explicit reasoning: catching a ball or grasping a bottle does not require writing down the governing equations. His Kahneman analogy places BFL’s work in “System 1 intelligence,” with language modeling perhaps corresponding to System 2. Ultimately, he expects the systems to combine rather than remain separate kinds of intelligence.
Blattmann describes FLUX.3 as one model that learns multimodally from images, videos, audio and actions. Generating convincing media already requires some physical understanding: a character’s arm must move plausibly. The model learns the observable consequences of physical laws rather than explicitly describing those laws mathematically.
3. Observation may teach the world before interaction completes the lesson
BFL’s approach does not require an explicit 3D model underneath the generator. Blattmann argues that 2D video contains enough information to develop implicit 3D understanding: generating a camera move around an object requires predicting its unseen side. Humans similarly infer someone’s back from prior experience, even when viewing them with only one eye.
A host steelmans the objection: a model has never personally inhabited the world, so perhaps it only approximates the constraints around an object without understanding the object itself. Podell’s response is that people also build representations from filtered sensory signals. Human vision even fills in missing information; direct access to an unmediated world is not the relevant distinction.
Blattmann nevertheless considers interaction important for higher intelligence. Babies initially observe, then manipulate their surroundings and discover correlations: knocking on wood produces a sound. Observation supplies substantial knowledge, but acting on the world can reveal relationships that passive viewing alone might not establish. He presents closing that loop as a subsequent research step.
He recalls disagreeing with Lukas Beyer, who was then at Meta, formerly at Google and OpenAI, and lived in Zurich. Beyer argued that interaction was unnecessary, while Blattmann considers it important for higher forms of intelligence. If observation alone carries most of the learning, as Swyx says Beyer suggested, scaling the current process could unlock far more intelligence. The disagreement remains open.
4. SDXL’s refiner rescued a model that understood texture better than structure
Podell frames SDXL as an effort to make the models feel good to use. Earlier releases inspired extensive community work on aesthetics and usability; SDXL incorporated broader improvements directly. These included large-scale training across aspect ratios and resolutions, efficient bucket-style training, and distinct pre-training and post-training stages supported by substantial data work.
The refiner was not an upscaler: it improved fine detail without changing image resolution. Its origin was an unreleased SDXL candidate with a different architecture, trained well into post-training. That candidate performed poorly on structure but better on high-frequency details such as skin and grass. The team repurposed its strength instead of discarding the entire model.
The resulting division exploited the denoising trajectory. Structure emerges nearer the noisy end; texture and other fine details emerge nearer the clean image. One model handled composition and another finished the surface detail: “different model for a different job.” Blattmann describes this as an early practical exploitation of a distinction that subsequent research examined more systematically.
Podell admits he initially regarded the extra model as something of a “cop-out”: he wanted one model to do everything. He now appreciates what systems of models can accomplish. The host frames the choice as using the simplest solution that works. The hosts also recall SDXL’s accessibility: it was a 3B model that worked on 8 GB of RAM, making practical experimentation and pipeline construction possible outside large labs.
5. Video exposed both the value of data filtering and the limits of frame stitching
Before Stable Diffusion, Blattmann was already working on video generation at NVIDIA. Align Your Latents adapted latent diffusion to moving images, where compression could exploit the temporal dimension as well as space. Stable Video Diffusion followed at Stability after SDXL, with a particular emphasis on the effects of filtering training data.
His explanation starts with the distribution being learned: blurry videos and clips with little motion produce a model that generates blurry, static-looking outputs, however good the algorithm. Research culture had often emphasized benchmark algorithms while under-discussing data. The hosts frame the issue as a quantity-versus-quality trade-off; Blattmann describes investigating data filtering.
The paper’s most exciting result for Blattmann was a transfer experiment. After training on ordinary 2D video, approximately 1,000 fine-tuning steps on 3D data produced a 3D model. He treats the speed of adaptation as evidence that substantial spatial understanding was already present in the video representation.
Temporal consistency requires learning across time, not merely stitching independently generated frames—even with fixed seeds and elaborate processing. Transformers need temporal attention to learn consistent motion. Longer-lived virtual worlds create a harder memory problem: an object disappearing behind an obstruction should remain the same when it returns, potentially requiring very long and memory-intensive context.
6. SD3 simplified the mathematics and redirected the training budget
SD3 combined several changes: convolutional architectures gave way to transformers, and classic diffusion gave way to flow matching, specifically rectified flow. Blattmann stresses that the team did not invent flow matching. They adopted it because the formulation was mathematically simpler, scalable and well suited to deriving faster samplers. When Swyx asks whether that loses performance compared with full diffusion, Blattmann says no.
In Blattmann’s explanation, classic diffusion uses a standard-normal endpoint, whereas flow matching can connect more general distributions. The model learns a vector field that transports one distribution into another; sampling then becomes numerical integration of an ordinary differential equation, potentially using simple Euler steps. Podell summarizes the simplification as effectively “drawing a line” between noise and data.
Another central contribution concerned where training computation goes. Uniformly sampling points along the data-to-noise trajectory spends effort on regions that may contribute little to final quality. SD3 investigated which regions carry important information, then oversampled those and undersampled less useful ones. This extended SDXL’s structure-versus-texture insight into a broader allocation problem across the trajectory.
The typography improvement receives the deliberately unglamorous explanation: “better modeling and better data.” A caption that omits text visible in an image does not teach the desired relationship between words and pixels. Better foundations also reduced reliance on elaborate prompting rituals. The guests resist attributing every visible capability jump to a single fashionable architectural component.
7. BFL’s founding thesis included capturing the value Stability left behind
The founding team numbered approximately 13–14 people. An offsite in France brought researchers who had previously occupied different parts of Stability into a focused group, discussing the learning process and hacking together. FLUX.1 training began around that period; roughly three months after incorporation, the company launched with a model whose commercial reception surprised even its creators.
Podell’s frustration at Stability was not a lack of evidence that the models mattered. Open-source users and successful businesses were already building on them. The gap was the ability to “capture the value of the models themselves.” BFL combined confidence in the team’s research record with a belief that a more focused organization could correct those commercial inefficiencies.
Blattmann says video-based world understanding appeared in the original investor pitch and in discussions as early as 2023. Image generation became a successful detour and commercial kickstart. The hosts also highlight specialization through partnerships: FLUX.1 launched with xAI, while they cite fal’s later post-training and distillation work on FLUX.3 Max as another way to maintain focus.
8. FLUX Kontext elicited editing skills already latent in spatial knowledge
Podell pitched the idea at another French offsite. A model that understands “a dog wearing a hat” already represents relationships between objects and their positions. His hypothesis was that carefully chosen data could expose those relationships for general editing, rather than requiring a separate specialized workflow for every modification or consistent character.
The work initially remained a side project involving Podell and one data colleague. They built a tiny, carefully labeled photography dataset describing simple relationships, rather than modern editing instructions. After perhaps four to six hours of training, the model showed unexpected generality—including placing a logo on a shirt—from that narrow starting point.
The early result was not polished: “it wasn’t good.” What mattered was evidence that the model could attend between concepts and reuse pretrained relationships. After Podell demonstrated the results at another offsite, the work became a broader company project for a couple of months. “It was mostly a data project,” followed by distillation work by Axel that sped it up.
The intended workflow change was iterative control. Instead of repeatedly prompting until a lucky image appeared, users could preserve a character and ask for a particular change. “I wanted to get away from the slot machine,” Podell says. The hosts connect this to a broader foundation-model pattern: absorbing many previously separate tools and fragile workflows into one general capability.
9. Multiple references expanded context without making bigger counts the objective
Podell’s principal disappointment with FLUX Kontext was its single-image input limit. Users wanted several views of a person or multiple references with distinct roles. Some worked around the restriction by stitching several images into one. FLUX.2 addressed the underlying requirement with larger-scale training and data supporting, at the time, 10 reference images.
That extension was again substantially a data problem: the model needed examples teaching it to relate multiple references correctly. Blattmann resists turning the capability into a contest over extreme counts. Gathering 100 references is one task; reviewing them and deciding how they should compose into a coherent image or video is another, harder creative burden.
Podell’s comparison is narrative rather than architectural: major movies generally revolve around perhaps five or six central characters, with Swyx immediately offering Avengers as an exception. More than 10 references can already become difficult for a person to follow. The point is a human-context constraint, not a categorical claim that the model cannot accept more.
10. Evaluation needs concrete tasks, human judgment and people who use the model
ContextBench grew out of BFL’s internal editing benchmark. Its prompts include requests such as “make the cat very fat,” reflecting late-night experimentation as well as production use cases. Podell wants models that are enjoyable to use because he came from communities where apparently silly experiments were often the first contact with a useful capability.
The hosts press on what makes this an evaluation rather than merely a prompt collection: where is the ground truth? The answer is comparative human judgment between model outputs, rather than a conventional validation set with one correct answer. A host also proposes more structured combinatorial generation; Podell says the team now does more work along those lines.
Blattmann argues for combining independent evaluation methods because each has blind spots. Automated and human assessments do not remove the need for researchers to sample their own models throughout development. Podell puts the product requirement plainly: the people building a model should actually want to use it, including for the things ordinary users find entertaining.
A host challenges BFL’s lack of a large public preference loop like Midjourney’s. Blattmann says such feedback can be valuable, but becomes noisier once baseline quality improves. His example is asking several leading models to remove a bed: all may now do it adequately. Swyx says editing’s clearer question is whether the model actually did the task.
11. Autoregression wins on data efficiency while diffusion offers a different speed lever
The hosts initially associate conversational editing with autoregressive image models. Blattmann separates the interface from the algorithm: language capabilities, conversation and iterative editing can also be trained into diffusion-based systems. The limitation is what developers have enabled through data and research focus, rather than an inherent prohibition on those behaviors.
His training-efficiency argument favors autoregression. An autoregressive model can train on all the tokens in an example in parallel, extracting a broad gradient signal. A diffusion example supplies a continuous range of possible noise levels, but each training instance samples a particular point along that path. He therefore calls autoregressive models more data-efficient.
Sampling reverses part of the trade-off. Autoregressive generation must proceed token by token, while a diffusion sampler can be distilled from 50 steps to four. A host pushes back that this makes autoregression sound simply worse; Blattmann reiterates that each has a limitation on a different side of the training-and-inference equation, and BFL has explored both.
Swyx revises his interpretation of Nano Banana as decisive evidence for autoregression. Blattmann thinks it is diffusion-based, but says he lacks insider knowledge and might be wrong; he suspects it may combine autoregressive language with diffusion imagery, which he calls a “Transfusion model.” Their larger conclusion survives that uncertainty: reasoning and editing loops are possible across approaches, and the debate is not settled.
12. “Diffusion language” contains two materially different research programs
Asked why diffusion text has lagged, Blattmann points to autoregression’s early arrival, simplicity, scalability and accumulated research investment. He says people reason left to right, while Swyx notes that models can take in a whole book at once. Human experience is therefore an imperfect architectural argument.
Blattmann distinguishes discrete diffusion language models, including Diffusion Gemma and Mercury, from continuous diffusion research. The former operate through masking and discrete token replacement; the latter work in a continuous denoising space. He is uncomfortable treating both as equivalent merely because they share the diffusion label, and regards the continuous direction as an active research wave.
The hosts ask whether GRPO-style reinforcement learning can work with pure diffusion text models. Blattmann answers that BFL is doing this and says they applied it to FLUX.3, with “tremendous gains.” The exchange does not provide a separate pure-text benchmark or parity result.
13. FLUX.3 action predicts the robot’s movement alongside the visible outcome
The action model begins with broad pre-training across images, video and audio, then adds robot-state prediction. Blattmann describes a state representation involving translations and rotations, with six degrees of freedom. The model predicts this together with video, using an instruction and earlier observations to generate the next chunk of actions needed for a task.
BFL’s claim is that general visual knowledge makes the additional action dimension easier to learn than starting from robot data alone. Blattmann describes applications to real robots, simulation, games and potentially computer use. He prefers “visually intelligent model” to the overloaded “world model,” while accepting “world action model” for a system that predicts both outcomes and actions.
The deployed model is 7B parameters because manufacturing use requires small, fast, on-device operation. Blattmann says it currently runs around 10 FPS. He would need to check the exact prediction horizon, estimating between 9 and 21 frames. Execution uses chunks and renewed camera observations; the model is running on a loop without a separate high-level planner.
The same FLUX.3 model underlies image and video generation, video editing, action capabilities and a fast 4K video upscaler. Video upscaling still needs temporal understanding: independent image enhancement would introduce flicker, especially in fine detail. Blattmann presents the expanding product family as the compounding benefit of training one general model rather than separately developing every application.
14. Four hours of task data is progress; one demonstration remains the objective
For one industrial application, Blattmann reports obtaining an adequate success rate with approximately four hours of data—“eight times less data” than an open-source comparison model. The example is task-specific, rather than a universal sample-efficiency result. Broad mid-training already supplies some task competence; high-quality robotics fine-tuning pushes performance toward the accuracy a particular deployment requires.
The hosts challenge the premise that every new robot or task should need fine-tuning at all. Blattmann agrees: BFL is concentrating on in-context learning, where observing an action once would let a robot perform it. He calls that a potentially superhuman capability and says, “I’d say 2027 will be the year we crack this task.”
Real-world evaluation remains awkward. A host questions how informative two successes in three “sponge in bowl” trials can be when restarting might yield three successes. Blattmann points to work closing the simulation-to-real gap, but offers no complete evaluation solution. His broader admission is that generative-model evaluation is painful—even internal groups disagree sharply about what constitutes a good image.
15. Manufacturing carries the stronger 2027 deployment thesis
Blattmann says the first BFL-powered robots will soon be live in production, while declining to disclose commercial details. Germany provides industrial relationships, relevant data and prospective manufacturing partners. His “2027 will be the year of robotics” call combines anticipated learning improvements with the expectation of initial commercial applications in real factories.
Manufacturing is the more tractable setting because tasks and environments are relatively constrained. Blattmann contrasts that with consumer robotics, where a person ideally wants a machine capable of doing almost anything. He gives manufacturing a “very good chance” of unlocking many applications in 2027; the hosts nevertheless emphasize that model improvements do not remove physical manufacturing constraints.
Blattmann’s household forecast is more tentative: later in 2027, technically inclined early adopters might get interesting, potentially useful home robots. He does not know whether they will handle fine motor tasks, and does not predict average consumers buying them broadly. His qualification deserves preservation: “this is entirely predictive” and “I have no baseline for this.”
16. Useful workflows still have to cross the AI branding barrier
BFL wants broader recognition beyond the technical community, particularly in the US. Blattmann emphasizes the company’s roots in both Europe and America and its global ambitions. The remaining marketing challenge is that users may recognize the models and their technical lineage without knowing the company behind them—a gap he wants to address in 2027.
Swyx argues that generative media faces unusual public resistance: labeling a useful workflow “AI” can itself invite hostility. His opening assessment is that conversational editing could replace routine Canva and Photoshop workflows, while he does not expect it to replace the top end of the market. Adoption depends on a capability people immediately understand and want.
Blattmann responds with professional productivity: these models already serve real workflows and make work easier, rather than existing to generate “avalanches of slop.” He also says the goal is not putting people out of work. Swyx resists that reassurance and returns to the concrete go-to-market question: how does useful functionality achieve a mainstream crossover?
Podell’s strongest unexpected example is restoring degraded family photographs—repairing and colorizing them while preserving a high level of similarity to the original. The family reactions mattered to him personally. Viral uses cannot reliably be scheduled: “they emerge,” and “you cannot plan them.” The research response is to keep betting on new capabilities that might deliver either joy or productivity.
Full transcript
Okay, finally, our BFL pod. Welcome, guys—Andreas and Dustin.
Great to be here. Thank you so much.
Thank you. Yeah, definitely. Thank you for having us. I know we’ve chatted about doing this for a long time.
Yes. We’ve had Anjney on a couple of times, and that’s where I met you, in Vienna, I think at one of the ICLRs or—
Was this the rooftop event, or what was it?
There was a year when there were 2 Viennas back to back.
Yeah, yeah, yeah. That was first ICLR, and then it was ICML.
Yeah, yeah. I don’t know how they do this thing. Anyway, at the time, you guys were just about to launch BFL. We had dinner, and you were thinking about what was next after Stability and all that. Anyway, congrats. How do you guys like to introduce yourselves? Maybe we’ll get a crisp self-intro up front.
I’m Andi. I’m a co-founder of Black Forest Labs. I’ve been developing diffusion models for, oh wow, I think 7 years now. I met Robin and Patrick way, way back in the day. I think it’s been 7 years since we met at Heidelberg. I’m originally from Germany. There, we started looking into diffusion modeling for images. We all had a physics background, in some sense, and this paradigm of mapping the general physical phenomenon of diffusion to machine learning and generative modeling had a kind of philosophical beauty to us. That’s why we got into it.
1. The Origins of Diffusion
Then we developed a lot of the fundamental algorithms that are powering diffusion models today, for image generation, video generation, and also action prediction. I guess we’ll talk about this later.
We’ll talk about it. It’s not immediately obvious to people: image generation to action prediction.
Yeah, we can talk about that. And one thing I did want to comment on is that a lot of people don’t know the relationship. Black Forest is a forest in Germany?
It’s a mountain range, actually.
Ah. It’s a region in Germany, basically. Finally, Robin and I first met during our PhDs, and then we figured out that we were actually born, I don’t know, 10 miles away from each other. That was pretty fun, and eventually that led us to come back to Germany.
Beautiful. All of Europe looks like this. You can see the lab right in the middle there.
Yeah, yeah, yeah.
So you named it because you’re from here?
Exactly. We founded this research lab, and back in the day, we could basically pick where we wanted to do it. No one said, “That’s okay, you have to come to, say, San Francisco or whatever place.” So we decided we wanted to start this at the place where we were born, in Freiburg, which is at the foot of the Black Forest, basically, or a city in the Black Forest. That’s how it came about, and that’s how the company started there.
CompVis was also in Heidelberg?
CompVis was in Heidelberg. Exactly. That’s where we met. We then later moved on to Munich.
Which is where the team that did latent diffusion—
Through our PhDs, exactly. That is when we developed this latent diffusion algorithm.
Yeah. I really like your name, by the way, your podcast name, Latent Space.
Yeah. I’d actually be curious how that came together.
I can talk about it. Go ahead.
Yeah. We developed the latent diffusion algorithm at the university, and based on this research project, we scaled out the algorithm, which led to Stable Diffusion. Since then, it’s been a constant journey on a rocket ship, you could say, which is super fun. We’ve met amazing people like Dustin as well, who’s one of the oldest and most relevant researchers that we have. But we’ll learn more about that later.
We’ll gas him up later. I think latent diffusion, and generally VAEs as well, are part of the idea behind Latent Space. Even our logo is actually a VAE. It’s this idea that you can translate things into spaces and then explore things adjacent to what exists today, and then translate back. I thought that was such a cool insight for creativity and generation that I never really understood until I studied Stable Diffusion.
2. Stable Diffusion Changed Everything
The launch of Stable Diffusion changed my life. I started Latent Space because of Stable Diffusion. I quit my job because of Stable Diffusion. I was like, “Okay, everything I know about AI is wrong because I can run this on my laptop.” If I was wrong about this, what else am I wrong about?
There’s another relationship with Terry Pratchett, which I don’t know if people have read.
Writer, right? Writer. He has this concept of L-space, which is library space. He has an orangutan who’s the librarian in Unseen University, which is wizard university. It’s basically a library with all possible books. So if you have a library with all possible books, you can theoretically go in there and find a book that’s right for you, and that is Latent Space.
Yeah. I originally called it L-space. One of my readers donated “Latent Space,” and then it became Latent Space. Amazing. Dustin, what was your journey into diffusion and AI in general?
Yeah. I came from a little bit of a different background than Andy and Robin and the team. I’m not from the German side, but I’ve still had a fantastic time working with these guys. It’s funny you mentioned that you quit your job when Stable Diffusion came out, because the original Stable Diffusion came out before I was working with these guys, and it was actually a cause for me to quit my own job at the time.
You caused mass unemployment. I’m actually curious about the number of people who changed their life direction when Stable Diffusion dropped. I’d be very curious to know the branching effects from that. We can theorize on it.
That’s actually super interesting, because here in SF, in the Bay, it had this huge impact, whereas in Germany, where we stayed back in the day, people didn’t really notice it because they were so far away from tech. There are a couple of people who’ve been involved in the open source community in Germany, but the majority of people didn’t really notice that drop in summer 2022, which was really interesting. Then you see what happens here.
Yeah.
That was really interesting to us.
To give a little background on me: at BFL, I’m one of the founding researchers who joined the larger team. I’m sure we’ll get into the history of how that all came to be. My background isn’t traditionally in the ML space at all. I was a software developer for around 10 years. Around 2020, I became very interested in open source AI. I saw the early work in GANs and other generative media, and I was tracking it and experimenting with it on the side. It got me quite excited.
I remember originally following the latent diffusion work and being quite excited about it. One of the things that tipped me off to think, “Okay, maybe this is something a little more serious,” was Midjourney V1. When that came out, I remember thinking, “Okay, this is actually going in a direction. I can see this being quite impactful.” Then the big moment that really changed everything for me, as we talked about, was the Stable Diffusion release. At the time, I was working at a software studio in Los Angeles. Stable Diffusion came out, and that day I turned in my resignation.
What was the plan?
The way I envisioned it at the time, I saw this thing that felt like magic. I thought the people working on it must be arcane wizards, and I needed to do whatever I could to understand what magic, runes, or other things they had imbued in it to make this happen and send it out into the world.
I wasn’t totally disconnected. At the time, I was in the Stable Diffusion Discord. I had friends in there, and I was interacting with the community.
Was it viewed as basically open source Midjourney, or did people not make the connection?
I was in both. I remember there were a lot of discussions around this, about how Stable Diffusion was having such an impact. There was a lot of discussion in the Midjourney Discord and the Stable Diffusion Discord around that time about the impact it was having. It’s been a long time, so I can’t remember the exact nature of it. I don’t want to speak on it.
Midjourney’s doing fine.
They’re doing great. But for me, it was more that I was very interested in the fundamental technology, and then the fact that I could run this at home. What I saw was this community of developers taking it home and developing on it. I was hacking away at it, meeting a bunch of really smart, great people, some of whom are at BFL now and have come along for the journey. That moment of community interaction, learning, and building together was really impactful for me.
Luckily, I was able to find a job opportunity that got me into Stability at the time. I didn’t start working with these guys right away. It took me a while to work my way up to that, and we can go into that history later if we want.
Stability had a bunch of different teams, right? There was Stable LM, Stable Music. There was—
Yeah, Harmonai, the audio team. The way I’d describe it is, if you grabbed a bunch of—
Discord was the space where a lot of this happened at the time. It was like a physical manifestation of a Discord. It was a melting pot of very different people who came together because they were interested in this emerging new technology. That’s one of the most exciting things about our team. Our founding team comes from very different backgrounds. We have this hardcore academic research branch, and then people from the open source community, like Dustin, who got into this space through a very different trajectory.
I think this is what makes our team so strong. There are rarely teams with this history, and with this kind of joint history together. It’s amazing.
We can go into more of your other co-founders first. I did want to say that I was also in that space. I was also very influenced by Stable Diffusion. I did the wrong thing: I went down the path of ComfyUI, where I build UIs for Stable Diffusion. You seem to have done the right thing and gone into research.
It’s funny you mentioned Comfy, because I used to work directly with him. Comfy joined somebody that I—
We’ve had him on the pod.
Yeah, and I think he moved here.
Yeah. He’s my neighbor. How do you become a researcher?
I’m someone who, when I get interested in something, is obsessive to the point where I can’t put it down until I pull it apart and figure out every aspect of it. I’d already been experimenting with pulling the models apart and talking to a lot of different people in the open source space, not just developers but researchers too. Remember the EleutherAI Discord back then? I met a lot of people I was learning from, and I was reading a lot of papers, trying to take it all in and understand it for myself.
Joining Stability was another major point for me. Originally, I didn’t join as a researcher. I joined as an engineer. But I spent all my free time with any researchers I was able to make friends with. Shout-out to Dr. Scott Hawley, who worked on the audio team and mentored me quite a lot. At some point, I was able to sneak my way onto the cluster and get access to some idle compute.
I treated it like I treat how I learn most things: open source hacking, trying things, reading papers, trying to implement different pieces of work, and seeing how models trained and learned. That was really it. There was no magic to it other than that I was excited and wanted to learn about it. I wanted to tear it apart until I understood how it worked.
This is open source. It is better.
Yeah. I think that’s generally how you learn best: trying things out and—
And he was also at Eleuther?
Yeah, affiliated, but not on the image generation side.
Back then, everything was very hackable, right?
The delta between open models and closed models was pretty drastic. People on the language model side were doing post-training. That’s where a lot of the new guys started.
You can look at early Stable Diffusion and where we’ve gotten to today. It’s pretty crazy. You both had pretty impactful roles: you did SDXL, and you did Latent Diffusion. We’ll get through all the history, but for people who haven’t followed the space closely, where are we today? What’s the state of things? I know FLUX.3 Video is out, and FLUX.3 Action is now 7B for robotics. What’s the latest in this space with you guys?
We started as an image generation lab when we started Black Forest Labs, as a kind of legacy of Stable Diffusion and the like. That was our focus for the first year, but then we realized that image generation is nice and this technology can unlock so much more.
3. From Images to Visual Intelligence
That’s when we slowly moved into what we call visual intelligence. We’re trying to get our intuitive intelligence into models: the kind we use when we catch a ball and understand physics without knowing how to describe it mathematically, or when we grab a bottle and know exactly what to do because we’ve done it many times throughout our childhood.
We want to transform that knowledge and that form of intelligence, which is very different from language model intelligence, into models. Learning from pixels—which means not only images, but also videos, and maybe audio too. “Natural signals” might be a better term. We realized we had the algorithm to do this, which is diffusion or flow matching. So we decided to extend from image generation to all natural representations. That led to FLUX.3: one model that learns multimodally from images, videos, audio, and actions. We can talk later about how it all comes together.
To put it in terms of Daniel Kahneman, if you’ve read Thinking, Fast and Slow, it’s a kind of System 1 intelligence, not System 2 intelligence. Language modeling may be System 2, but for now we’re focused on System 1, which is extremely important for intuitive tasks and real-world interaction. That’s why, at a high level, it’s also so good for robotics. That’s been the journey, and it’s what we’re focused on now. In the end, it’s about combining both systems into the very broad kind of intelligence that humans also have.
I want to get into the history and all of that, but just to recap for people at home: you’re already making money as a generative media company, right?
Exactly.
I don’t know what numbers you’ve announced. Maybe you can recap. You raised $300 million—
You raised a ridiculous amount of money.
Yeah, multi—
All in all, I think we’ve raised $450 million so far.
Oh my God.
Way more than Stability.
Any revenue milestones you’ve announced?
We haven’t announced any. The last number published, in December, was $100 million in revenue. We’ve crossed that since.
Clearly you’ve crossed that if you raised $450 million.
Yeah, exactly.
Anyway, I’m just saying you didn’t have to pivot.
We’re not pivoting. We’re just extending. This model is trained on everything; it’s always been the plan. To create artwork, you also need this visual intelligence. To create a video like the one we’re seeing here, you need to know how physics plays out. The model needs to understand that the arm of this capybara isn’t just falling away; it has to follow some law described by physics. It doesn’t describe the law itself; it describes the output. Similarly, humans don’t understand physics, but we understand that if we let a ball fall, it falls downward.
There are different concepts of a world model, and that’s something we can get into. Where can you learn? This is the Bitter Lesson version, where you learn from a bunch of input sources, or you physically structure the 3D model and learn from that.
No 3D model at all. We learn from natural data. We also have a bit of triangulation from our eyes: we have two eyes, which gives us a little understanding of 3D perception. But even if we closed one eye, we could still learn a lot about the world. These models are basically doing that: learning from natural, 2D video data.
That data has a lot of 3D understanding in it. If you want to generate a pan around an object, you need to hallucinate how it looks from behind. If I look at you, I have a pretty good understanding of what your back looks like because I’ve seen it before. That gives you an implicit 3D understanding, even though the models are trained on 2D videos.
I’m trying to steelman the counterargument. This is the Gary Marcus version: “LLMs don’t learn anything,” or whoever the famous critics of this approach are. The idea is that a model only sees inputs and has never had personal experience. You can model my back because you’re also human and can model other humans. Maybe that analogy isn’t exactly right, because I’m sympathetic to your side of the story. But I think that’s how people criticize this approach to world modeling: you can get closer and closer, but you’ll never quite get something fundamental about the thing because you never modeled the thing. You modeled the most likely constraints around it.
I think to some degree you could argue that we operate under the same principle. We’re embodied and constantly taking in data, and obviously we regularly update. But we’re still just taking in filtered signals from the world. There are all sorts of wacky things about how our eyes process information. Part of our brain has to hallucinate some of it because we have a blind spot in the center, among other things.
Fundamentally, I don’t think there’s that much difference between us taking in natural data and building a model of the world that we can act on, and these models learning from data. The signals are different: they’re turned into pixels or waveforms and processed in another way. But the information is still there for the model to take in and build a representation of the world that it can operate with.
Yeah.
I was actually arguing about this with Lukas Beyer. He’s with Meta now, I think, formerly at Google and OpenAI. He created bits. He’s in Zurich, which is pretty close to Freiburg, and he comes over from time to time.
We were debating human learning. He was saying that we don’t need interaction, whereas I think interaction is important for higher forms of intelligence. First, there’s observation. That’s also how we humans learn as babies: we lie around and observe things, and from that alone we learn so much. That maps exactly to how we’re currently training these models. They observe a lot of 2D videos and audio, and then they learn from it.
I think the next step will be closing the loop and interacting with the world. That’s also how we learn about the world. We can tap on wood and notice a sound occurs. We can manipulate the world to understand it better. That’s how I think humans mainly learn—at least how I personally learned.
And he was actually saying he thinks the majority is observation, which I found very interesting. If that were true, then we just need to do more of what we're currently doing, and that could unlock an insane amount of intelligence.
I wonder how much of this is emergence with good priors. We've seen that adding new modalities to language models doesn't give you the full benefit of doubling your performance just because you've added vision. But if you have good priors, is the benefit small or big? It's interesting to see.
I think there's a lot of technical stuff that'll be interesting to walk through, like Gaussian splatting and the 3D data World Labs did. What are the trade-offs between video generation output and other approaches? But I think something fun for listeners would be the history. You worked on very early Stable Diffusion, SDXL, and latent diffusion even before that.
Video diffusion as well.
Video diffusion. There are a lot of buzzwords here: LCM, LoRA, flow matching, old prompting. Can you teach us everything?
What's interesting? How big are these, scaling-wise?
I seriously... No, so I've actually built a tech tree. I wanted to build this game like Civilization, but the tech tree is AI papers.
Oh, okay.
You spend research points to get things. These are the tech trees: first it's GANs, then latent diffusion happened. There are a lot of core points beyond that.
I'd play that game.
So, anyway, just to pitch the game: you role-play as a new lab, and you have to spend points on GPUs or researchers. Then you have to collect data, and there are all these little things you can research.
Fine-tune.
Come on, whip it up. Let's see. We're in the vibe-coding world now. I want to play it tomorrow.
Game balance is really hard. I think it's an AGI-hard problem, because once they find a way to win, they just always win that way, and it's not interesting as a game. You need multiple ways to win, and high-risk ways where you probably lose but maybe you win.
Yeah. Mode collapse, classic problem in GANs, by the way.
I think one thing I'd be interested in teaching our audience is: if you've heard a lot of terms, what are the key things that have really impacted the field? You've lived through a lot of this. I'd also say this is a great way for people who want to get into this space to learn good interview questions if you're hiring. You should know most of these things.
There are some points where different paths have popped up, like autoregressive image generation versus diffusion, two backbones or hourglass diffusion. What are the key things people should know? I'll leave it pretty open. First would be SDXL, maybe. What were the things we did in SDXL?
The refiner.
Oh, no, please, not the refiner. Why that reaction?
4. The SDXL Refiner
SDXL was a big piece of work in trying to make the models feel good to use for the first time. Stable Diffusion 1 and 2 were very exciting and fun to use, but for anyone who was in the space at the time, between SD1 and SDXL, we saw a ton of work in the open source community on how to fine-tune these models, make them more aesthetic, and make them multi-purpose.
The original Stable Diffusion models could only do square images. SDXL was the first attempt to release a foundation model with a lot of these general improvements built in. It involved large-scale, multi-aspect training. At the time, this was before packing. We had an efficient way to do bucket-style training, on all sorts of different image resolutions at a large scale.
This was the first time we did multiple actual training stages, taking the model from pre-training to developing post-training sets. A lot of data work went into SDXL. I'll give a little backstory on the refiner and some things that didn't quite make it in.
The final release model fundamentally works like the original Stable Diffusion models. It was a latent diffusion model: it took in text, had a noisy latent, took the text context, generated imagery in latent space, and then a VAE decoded it. The refiner was another full diffusion model. Its purpose was to take an image that wasn't rough, but where SDXL wasn't the best at high-frequency detail: the fine details, like skin texture and the texture of grass.
Yeah, upscaling.
It wasn't an upscaler because it didn't actually change the resolution, but it upscaled in terms of detail.
It's a refiner.
It refined it. The reason I thought it was janky is that the whole team that worked on this was fantastic. I don't want to take all the credit; there were a lot of fantastic people who worked on this. We did a lot of work to push the detail quality of the model while trying to keep its generality, and we just didn't quite get there.
Then there was a side track of work looking at other methods. This is where the refiner came into being. Here's a little bit of the secret lore: there was a version of SDXL that was never released. It had a different architecture and didn't work quite as well, but it was trained pretty far along, even through the post-training stages. The refiner was that model. It's a full diffusion model.
I actually completely forgot about that.
The model had a little bit better performance on high-frequency details. It just had very poor structure. The idea was: what if we took a different model for a different job? We did a lot of work to push how well it could refine high-frequency detail, and then we made a pipeline. Together, this became a more performant system.
Which is actually interesting. It was one of the early pioneering works that looked into what properties are learned at which part of the diffusion trajectory. The structural things are learned closer to the noise, and the fine, high-resolution, high-frequency details come further down, closer to the clean image.
Today, there are a bunch of different works that analyze this. You regenerate from low frequency to high frequency as you go from noise to a clean image along this trajectory. But that wasn't really known then. It was informally known, maybe a bit. This was one of the first works to exploit it and say, "I dissect this trajectory, and I have two experts: one for the structure and the next for the texture."
Yeah, and right now we have modulation, which conditions on the time step to account for that in the architecture and stuff like that. But that was one of the first works to make this distinction. And, as you see, it worked quite well.
I guess the reason I still hold some frustration is that, at the time, I wanted this one model to be so good at everything.
It was a cop-out.
Maybe it was a little bit of a cop-out. Nowadays, I take a bit of a different view, because we see systems of models do incredible work together.
But at the time, I remember feeling this old frustration from back then.
Well, it’s also very fair, because you should always go with the simplest solution that works. If you need to understand at what point you need a machine learning approach, it’s just not always clear.
A machine learning approach. Yeah, exactly.
When it comes to scaling, whatever the simplest solution is that works, you should go with a bit of Occam’s razor in statistics.
Well, Runway was also doing Gen-1 and Gen-2. Anastasios said on this podcast that Gen-2 was a pipeline mix.
Oh, okay.
Like when they—
No, if it works, it works.
They just didn’t admit it, whereas yours was public, so you had to admit it.
Yeah. I think if you go back to the time, people coming into the space today haven’t seen 200 or 500 steps of diffusion. As you say, you have a background, and then minutes later you have little fine details coming in. It was also very tinkerable. This came out as a 3B model that works on 8 GB of RAM.
As much as you could follow the bitter lesson and do the general approach, if you’re not a lab and you’re just doing this at home, you have one or two GPUs, so you’re going to do pipeline stuff.
Yeah. That’s how you can see the differences right there.
5. Bringing Diffusion to Video
Yeah. I’ve been working on video for quite a while. Actually, before Stable Diffusion, I was working on video generation for NVIDIA. I had a paper there, “Align Your Latents,” where we basically mapped the latent diffusion algorithm to video. That was the natural next thing to do.
It was very exciting to see things moving. You can imagine the compression reasoning behind why Stable Diffusion was so efficient compared to other models at the time. That also holds for video, where you have this temporal dimension that you can compress. So that made a ton of sense.
I worked on this first for NVIDIA, and then, when we joined Stability, after SDXL, we worked on Stable Video Diffusion. That was the first application of this latent diffusion approach to video. Importantly, that work was about investigating the effects of data filtering, which I felt was underappreciated in research, and still is.
When we think about what these models are doing, effectively, we’re estimating a probability distribution, and the dataset gives rise to that probability distribution. If you have the perfect algorithm, the best algorithm in the world that learns anything you want, but your data is very bad—if you have blurry videos or images on average, and not a lot of motion in the videos—your model will learn that, no matter how good your algorithm is. What comes out of the model will be very bad.
The data is one of the most important levers you have when you want to train these models. In the research field, people didn’t talk much about data because a lot of the research community worked on benchmarks and tweaked the algorithms. But with frontier models, we know by now that data is extremely important. We wanted to share more context on that. We investigated how much data—
You have this trade-off: quantity versus quality, basically. Where’s the sweet spot? Is there a chart we should pull up?
Coming, coming. The most exciting experiment we did wasn’t the data one. If you go down a bit, we showed that pretraining on plain 2D videos lets you fine-tune an explicit 3D model extremely quickly. If you go to the last paragraph of the paper, here are the 3D views of the model.
What we did was take this video model that we trained, and then we had, I think, 1,000 fine-tuning steps on 3D data, and we got a 3D model out of it. That comes back to the question of what you actually need to understand the world in 3D. Only 1,000 steps of fine-tuning shows that this video model, although trained only on plain video, actually had an insane amount of 3D understanding. You just needed a bit of focus to get a 3D model out of it.
That was one of the early experiments where we saw that you can get world understanding through training on plain video. From a research perspective, that was the most interesting experiment in this paper. It’s also a pretty short paper. There’s a section here where you explain the pipeline: image pretraining, video pretraining, video fine-tuning, and then 1,000 steps to make it 3D.
Crazy. At the time, I don’t remember what I did, but there was this temporal consistency problem. If you just stretched out diffusion and tried to stitch frames, it wouldn’t work. I had a very old hackathon project where I forget which method we used, but we fixed the temporal consistency and scaled it out to 15 or 20 seconds. It took forever, but it was a very hacky solution because you could do this in an afternoon.
Critically, you need this kind of learning over the temporal dimension. If you just stitch together a very fast diffusion model, you get flickering. Even if you fix the seed and do a ton of work on the effects, you realize that today everyone is using transformers. In the transformer suite, you need attention over the temporal dimension to learn from it and understand what consistent motion looks like.
As far as I understand, we asked this about the Grok Imagine part as well: there’s no compression on the video side that’s useful, right? You just have frame, frame, frame.
Exactly.
Which is the simplest version.
If you look at the most advanced MP4 codecs, they have very smart compression. But that is lossy, obviously, and you need to connect the frames and extract what consistent motion looks like.
I don’t think any video model exposes how far back you look in temporal attention, because sometimes you need to look back longer, or it needs to be more adaptive. For example, if something gets occluded and then comes out again, you want it to be the same. But how long is it occluded?
That principally depends on the problem you want to solve. If you want to create a long-memory model that can create virtual worlds, then you need to find a way to look back a very, very long time. That’s super memory-hungry, and I think it’s one of the most exciting research problems to work on these days. Long context is extremely important.
I don’t know if there’s more there before we cut over to Black Forest Labs. Anything else in the Stable world?
There’s SD3.
SD3. That was the last paper you did before you—
6. SD3 and Flow Matching
Yeah, SD3 was a pretty big paper. It was a culmination of our work at Stability, I would say.
Yeah, absolutely. This is your graduation thesis.
If you want, yes. This paper was really about understanding which part of the trajectory from data to noise the model learns which properties, and where we have to spend most of our training computation to maximize the model’s performance.
Obviously, it was also the transition from classic diffusion to flow matching, with a slightly different paradigm. We call it rectified flow.
This was also the first time we used diffusion transformers, right? With the—
Yeah, and then we took it a bit further.
A lot of changes, going from a convolutional architecture to a transformer architecture.
Changing the probabilistic model from diffusion to flow matching is much simpler mathematically, which is super nice.
I think you should actually define some of those terms for people who haven’t heard them. Latent diffusion, flow matching—what are they? How did you come up with these things?
Well, he didn’t come up with flow matching.
Exactly. We didn’t come up with flow matching. We just saw this algorithm as something that is more general because, with diffusion, one of the 2 distributions you’re translating between always needs to be a standard normal. That doesn’t hold for flow matching.
Instead of seeing the problem of transforming one probability distribution into another as the solution to a stochastic differential equation, you see it as the solution to an ordinary differential equation. That simplifies everything mathematically. It also makes fast sampling more feasible. Once we’ve trained the model, we want to make as few steps as possible to sample as fast as possible.
Some people would say flow matching is actually a special case of diffusion. Others might say it’s a generalization of diffusion, because you can translate between 2 distributions no matter how they’re shaped. You don’t need one of them to be a standard normal.
In general, you can see it as a model that learns a vector field that translates between 2 distributions. Once the model has learned that vector field, you can integrate it numerically, which is as easy as solving an ordinary differential equation with a forward Euler step. It’s similar in many respects, but mathematically easier and more general in the kinds of probability distributions it can translate between. That’s why we went with it.
A third reason is that it was easier to derive a fast model from flow matching. It was much simpler and more scalable, much easier to work with, and it had this nice performance benefit as well.
I don’t know. The diffusion space got quite complicated. Are we satisfied there’s no loss? Is it lossy in acceptable ways?
Lossy? Do you mean—
No, loss compared to full diffusion.
No.
All right. So then it’s good. It’s a strict improvement.
Exactly. It is an improvement, definitely.
It’s effectively just drawing a line between noise and data.
Yeah. It kind of took this complex space of how you map between these 2 distributions, as Andy was saying, and simplified it down to the simplest thing: drawing a line.
Because you’re trying to take a shortcut. What if it’s a bumpy distribution?
Yeah, that’s the thing. One of the most important parts of the paper, on the probabilistic modeling side, was looking at the trajectory from noise to image and asking which parts really matter and which don’t.
We train this in steps. We always train on one step from data to noise, and then we approximate this direction in flow matching with the model. If you sample the steps between data and noise uniformly, you might spend a lot of time optimizing for things that don’t matter.
One central part of the paper was asking where we should spend our compute. Which parts of the trajectory from data to noise carry the most important information? That’s where you should spend most of your compute during training, oversampling those parts and undersampling the less important ones.
It’s a further exploration of something we talked about with SDXL, which was the refiner. We spent time training the refiner at the end of the trajectory to learn high-frequency details. This was more of an exploration of the entire space of where you should spend time to make the best general model to learn the data.
Amazing. One thing I picked up with SD3 was text in images, which you guys showed off a lot. There was a time when images just couldn’t do text, then they did more and more, and now it’s, “Here’s our nice typography.” Can we attribute that to the diffusion transformer?
I don’t know if I could say it was the diffusion transformer.
I think it really was just better modeling—
Better data?
It was better modeling and better data. As much as I want to attribute it to some fancy thing, it really was just that it can model the world better. It’s a data thing.
It’s also about what you’re training on. For everything you train on, you need a caption. If your caption doesn’t describe the text that’s visualized somewhere, your model won’t learn it.
Sometimes we spend a lot of time talking about the nitty-gritty details of an algorithm. In the end, the better lesson is—
Right.
—more easy data—
Yeah.
—is almost always the most effective. You can obviously scale in a smart way, and if you spend tons of compute, you should spend it as efficiently as possible and only where it matters. That’s also at the heart of this paper.
I think prompting also got a lot easier at this time.
We used to have the 4K—
Prompt extenders. The 4K upscale, Ultra—
Our solutions were massive, with all the lists.
8K.
Yes, yes, yes.
Greg. Who was the Greg guy?
Greg Rutkowski.
Oh, Greg Rutkowski, yes. I wonder how he’s doing. I hope he’s doing well.
I hope he’s doing well.
And for context, this was when DALL·E 3, Midjourney, and Ideogram V1 existed, and nothing was open. Anything you’d see at this quality was either old Stable Diffusion or a closed portal.
Then you made the big break to Black Forest. What was that whole period like? You had to leave and start a lab from scratch.
7. Founding Black Forest Labs
It was probably one of the most exciting times we all had. When you’re in stealth, you’re a small team. We were 13 people when we started.
14, yeah.
We first went to France and had a fun little week in a chalet. We spent the entire week hacking and hanging out.
I remember that. It was fun.
Dustin figured out some really cool stuff to make our models much more consistent. I remember that. It was such a fun week.
That was a magical week. That really was our founding offsite.
That’s when we started training Flux.1, which 3 months later resulted in this nice company launch with a model that I think blew pretty much everyone away. It blew us away too.
Oh, yeah. Commercially, what this directly unlocked, after having incorporated the company for only 3 months and then releasing this model, was received really well. That was amazing.
Yeah.
That was a lot of fun. Did you know that when you started this? You had to get all the clusters together and all the data. You had to start from scratch.
We had to start from scratch, yes, but we knew what to do. Obviously, we had a lot of experience.
You’d just done SD3. Did you have a hypothesis about what you’d do differently?
We’d obviously gathered a lot of knowledge over all that time, even before Stability. That all culminated, and we also had a team that was all in one place. We were very focused, which had never been the case before, because people were working from very different parts of one company. Also, as Dustin said, Dustin wasn’t actually on our team before.
We had a super nice team together and spent a lot of time discussing how we’d design the learning process for this model. That happened at the offsite, and then we just kicked off. That resulted in this model.
Yeah. Was it discussions of what FLUX.1 would be, or was it broader, about where things go in a few years? How much of this vision was part of it?
I guess I can jump in. One of the reasons the company was started was that there were some frustrations at Stability. We saw the impact our models had made out in the world—not just in the open-source world, but also in companies that were started, run, and quite successfully using our models. We knew there was a lot of demand and potential in what our models could do.
Unfortunately, I think we also felt at Stability that we couldn’t fully capture the value of what the models could be. We believed in our team’s ability to do the research, build these models, and get them out into the world. This would be our shot to correct the inefficiencies or mistakes that had kept us from capturing the value of the models themselves.
That was a big part of why we thought we could move forward. We had a track record of doing this and making an impact, and we figured we could continue and build a successful company out of it. I think it’s gone pretty—
I think it was a good thesis.
Yeah, it’s gone pretty well. I think it was a good thesis. So—
As I’ve said before.
Yeah.
Well, what use cases were you seeing? A lot of people went straight to Hollywood. “That’s our goal.” Imad used to say, “We will make Game of Thrones Season 8 again.” I’m trying to remember the name of this company, but this might even have been before SDXL. For a week or two, it went viral: anyone could take a picture of themselves, and it—
Yeah, the older version of Stable Diffusion, right? The older version. It used the old DreamBooth-style fine-tuning, which was extremely inefficient.
But it blew up. Everyone was using it, and everyone was excited about it. The models could make so many things, and we saw a lot of potential in how people wanted to use them. We saw where the models were going. At the time, they couldn’t do typography. We knew typography would be important, and we eventually got there with SD3 and beyond. There wasn’t just one use case. We saw this as a general technology with a lot of potential that would continue to grow, and that potential is now expanding much more into the physical world.
This was already the potential we saw.
Yeah, absolutely.
BFL wasn’t founded to be an image-generation lab. The FLUX movement was so good that we capitalized on it, too, because it gave us a kickstart. But when we pitched the company to investors for the Series C, it was clear we wanted to work on video. We see that as—
Yeah, I think we shared this on a podcast back in 2023, when we were still at Stability. We were saying video modeling is actually the central means to understanding how our world works. As I said earlier, we humans first observe the world, then start interacting with it, and through that we learn so much. That was always the thesis.
Then we had to take this detour through image generation, which was very successful. You always face the issue of where to spend your time. Unfortunately, the day only has 24 hours. But FLUX.3 is really where we saw this going for a long time, and what we wanted to be doing for a long time. That’s why we’re so excited to have the model out there, able to create videos, images, and audio, and act as a robot brain in real manufacturing workflows.
There’s a lot of stuff still coming this year that we have to announce on the physical AI side. As Dustin said, our goal was always to create the most general technology. We see that happening now, which is why we’re so excited.
Yeah, I think you also took the right paths to stay focused. If I’m not mistaken, FLUX.1 launched with xAI. It was their image-generation model.
That’s right.
Good partnership. That’s a good way to solve the question of whether to build distribution yourselves. Even with the latest FLUX.3 Max, post-training is with fal. They do a lot of inference distillation. They’re good at that, right? It’s interesting.
But it seems like you’re expanding now. What’s the latest one with robotics?
Flux Reaction.
Yeah. Before we get there, there’s one more I want to cover.
Yeah, that’s okay.
Image editing in context.
Sure.
There’s also a really nice history on the website.
Yeah.
It’s weird that we’re now at the stage of the company where things just happen, and I’m like, “Oh, we have this.” I was kind of sad FLUX.1 wasn’t on there to view, since it was the original one.
Yeah, it’s fine.
Pro looked good. Even just the visuals here—maybe someone will clip the progression from original Stable Diffusion into XL and through all the papers.
You see the progression, right?
Yeah. But one thing I want to cover at some point: I no longer personally see the progression. You show me state-of-the-art image and video models, and a lot of Chinese labs are putting them out. It’s hard to evaluate. I know people care. I’ve talked to Summit at BFL about how people in the industry aggressively A/B test, and there’s a lot they care about in evaluations. But there’s also a level where it all looks very realistic to me.
The rest is a prompting skill issue.
I think there are a lot of little nitpicks. There were issues with hands, and there are still issues with text sometimes.
There are still hand issues?
No, there were hand issues. There are issues with perception and layout. I’m redesigning my living room, and it’s horrible.
8. The Case for Image Editing
Controllability is a huge thing. I think controllability is the thing we need to solve at large scale. There are so many ways you can control these models. Maybe that’s a good segue into FLUX Kontext, because that was a pain point we saw for a long time. People were training so many LoRAs just to get character consistency into models.
Yes. Shout-out to Replicate, with all the LoRAs back then.
Exactly. Then we had this amazing pitch from Dustin. It was at Northside, in France, I think. It was another cabin. A lot of things happen in cabins. I think we should go to the cabin.
You should do more.
Then we had this pitch from Dustin, who saw the potential: “I need to get this context. I need to be able to prompt not only with text, but also with images for image editing.”
Yeah. I originally—
That made pre-hype as well. I’ve heard the story that FLUX Kontext was you and one or two other people.
For a long time, it was basically my side project. Then I pulled one of our data guys into it a little and said—
“Come, we’re building this thing.” For me, as Andy was saying, it originally started as a pitch of mine at a cabin in France. I like to do these presentations every offsite, or every time we get together, where I try to look at the state of the field, where I think it’s going, and what we aren’t doing yet. What’s something these models could do that’s novel and useful, but that we might not be looking at right now, or that isn’t solved?
I did a presentation on what I called the “omni model”—the most overloaded word in the world at this point. There was a lot of research out there at the time on other editing models. If we remember back, there was the old Prompt-to-Prompt, and then there’s—why can’t I remember the name of this? The pitch was essentially that these models are clearly learning deep relationships about the world, and that’s going to keep coming up. It’s about these relationships. A Coke can is sitting on top of a table; this bottle is sitting here. You prompt the model, “I want a photo of a dog wearing a hat,” and it needs to know that the hat is on the dog. It needs to have this general understanding.
My theory at the time was that since the model already has all this spatial knowledge, if we build the right data and train an already pretrained model the right way, we should be able to evoke general image editing with very little training, capturing these prebuilt relationships: when things are there, when they’re not there, and how these spatial relationships inside an image work.
Then it became a three- to four-month project. I don’t remember the timeline. I was here in our San Francisco office with the one data guy I stole, building out early test training datasets. I remember the moment I saw that it was clearly working quite well. We made an effectively tiny dataset of very high-quality labels describing simple relationships in an image, and that was really it. It wasn’t a way you would prompt a modern edit-instruct model; it was just literally describing these relationships.
I trained the model for maybe six hours, maybe four hours. It was a very short training run. All of a sudden, I saw what felt like clear generality coming out of it. I think it was a photography dataset, just photos of stuff, but even from that I was seeing that I could put a logo on a shirt. I could do things like that. It wasn’t good, but the model was clearly learning to attend between these 2 concepts and use these pretrained relationships.
Then I took that to our next offsite, showed the results, and it became a more company-wide project for a couple of months. We really put effort in and brought people in. It was mostly a dataset project. At the end, there was a lot of very nice distillation work by Axel that sped it up. Then we were able to put this model out into the world, and I was quite proud of it.
That was the first model to really do character-consistent editing at high speed, right? That made it accessible to so many people. For most people, it’s very common to generate something, chat with it, and change it. Before this, image generation took quite a while, and you needed to recreate the image. You couldn’t just change the parent to be blue. You wouldn’t get the same character.
And you’d need to mask something you didn’t like and fill in the blank.
Yeah, I just wanted to get away from the slot machine, essentially. Everyone had to do the slot machine.
Yes. Prompt and prompt and prompt, and it was very clear that iterative editing was a much more effective way to do this.
My crappy ComfyUI was that: it just slopped out 10 different versions, and you could pick.
I still slop out versions. I’m changing things, living in my “generate 4, generate 6” routine, redoing something. Just make 10 iterations, and eventually one is good enough. My favorite is that you pray to RNG Jesus because it’s a random number.
To me, this concept of what a foundation model is is something like this: you know all these little workflows you do to do this and that, and they all use different tools? We’re going to replace that with one model.
And that was where Flux.1 Kontext came from. This was a very clear “aha” moment for me.
Yeah.
I never tried Prompt-to-Prompt, but to me the closest thing to Flux Kontext is Qwen Image. The Qwen folks are doing really good work. They had a Photoshop layer model.
Yeah, it was incredible. They’re doing pretty cool stuff over there.
It’s interesting how much of this just comes back to there being a lot of good data.
Yeah, it was targeted good data. I don’t know how to describe whether it was a good concept, but yeah.
One thing I’m surprised by: Japan had this whole law where they would protect you if you trained on copyrighted material in the country. Everyone was like, “Ooh, Japan.” It’s interesting—nothing happened.
I always wanted to visit that office. I was sad I never did.
On your omni point, it was an interesting time. Gemini is obviously the latest omni model. ImageBind, which I didn’t realize was from 2023, was from Meta. They threw in embeddings for heat maps and depth.
Heat maps.
Depth. That’s a big one. I think this work has progressed; there’s definitely newer work on it.
You can think of it as just another color, right? It’s RGBA, whatever that is.
Yeah, yeah.
The fun thing was that it was all in, all out, so you can go from text to heat map if you want. You can go from IMU to depth. It’s a big market for that.
Interestingly, there was no video out, which is the central thing we need to understand the world better. I think that’s always been at the core of what we’ve been doing.
Okay. Before we leave the data or IP discussion, is there a particular kind of data that everyone in the industry knows is valuable? If I have a data vendor listening, how do they sell to you? How do they approach you?
It depends heavily on what type of problem you want to solve. If I train a robot, I obviously need a lot of egocentric data. That’s not a secret, right? That’s just how this thing will be.
Egocentric, or—
There’s also a benefit for you guys, right? On the language model side, there are a lot of people doing RL environments. Is there a parallel where people should come to you with that kind of stuff?
I’m trying to get you vendors.
We have so many data vendors approaching us, actually.
Too many?
Yeah.
Okay. What’s good versus bad?
Again, it depends on which stage of training you’re at and what your effective goal is. There’s a trade-off between the scale of the data and its quality. For every training run, for large-scale pretraining, you have to find that sweet spot. Depending on the architecture you’re using, that might look very different. That’s one of the central pretraining experiments, or pre-scaling experiments, that you need to do. You need to do a ladder to understand at scale how much data you actually need, and to predict how the quality should look. Where is the sweet spot between these 2 dimensions of data quantity and data quality?
Any other major jumps we want to call out? Flux.1 Kontext was a big one.
Flux.2, Flux Tools, Action—we talked a bit about those. Any other things?
I mean, I think the 3 sizes are very important: the Dev models and Schnell.
Schnell?
Schnell means “fast” in German.
I think you probably covered that in a blog post. But I think having open source, closed source, and a range of performance makes sense.
Yeah, absolutely.
If there’s anything to track, the one thing that made me quite sad about Flux Kontext is that it was only single-image editing. It was clear there was a huge need for multi-image. People want to put in many different images and relate them—
References is a better word here.
Yeah, many different references. I think Grok is king right now, though I don’t know the current state of it. I know Midjourney does quite a bit with style books, where you can build up references.
They’ve always been very good at style reference, but this is more general. Flux Kontext would only take one image, so you could only iterate with that. If you had a bad photo of one person, that was all you had.
But you could have 3 different photos of the same person.
It was actually general enough. I remember it was quite funny: people were stitching a bunch of images into one image to emulate it. With Flux.2, we were finally able to spend the time to build out the data and do larger-scale training for handling, I think it was 10 reference images at the time. That gave you much more context to work with.
And, to a large extent, that’s also a data problem. You need the data to have all these references.
Is there a practical limit? Let’s say the industry standard is 10 or 18 references. That’s it, right? I don’t need 100.
Obviously, you can show crazy numbers for how many characters or images you can get into one image, or video. But then you need agent work to get me 100 references.
Which is the easy thing, right? It’s agent work. But then I have to check it. I have to review 100 images to see if that’s actually what I wanted.
It’s also a hard creativity problem. Where should I put these 100 images?
Composite them.
Without over-indexing on the number, anything over 10 references is hard for a human to follow. If we’re creating a video, I can put in 50 people. But in the most important movies, how many central characters do we have? Five or 6 maximum, unless you’re doing Avengers.
Exactly.
At some point, that approaches the limit of what we humans can process in a 2-minute clip. This also connects to the evals you mentioned earlier. Building evals for larger contextual models becomes a much bigger process. We ended up releasing an open eval set called ContextBench, built off the internal benchmark we use to evaluate context. As you scale up the number of references—
I haven’t seen this.
Please look up ContextBench. I’ll pull it up. It was a lot of work, but it was fun to put together.
“Make the cat very fat.”
I remember sitting in the back room of our San Francisco office with a few of the other guys. Tim Dawborn and I were laughing.
Do you just take some edibles and go, “Dude—”
We were sitting there late at night, literally just pulling images and typing in the stupidest things.
It works, right?
But you need to make one step to make it enjoyable, and you need to decide what the ground truth is.
This is just a benchmark, basically. It’s just inputs.
Right, but how do you evaluate the output?
That’s the interesting thing. You ask people. This is a public benchmark to give everyone some arena for improvement.
I see. So this is half of an eval.
This is the eval. You have 2 models you want to compare, and then you judge.
Arena style, yeah. So it’s not a validation set in that sense. It’s just an eval benchmark.
This is just me sitting in a room. We removed some of it. Genuinely, 80% of this is that. It’s also heavily influenced by our Italy offsite.
Why don’t you do Mad Libs, a combinatorial explosion that’s more structured than just random inspiration?
We definitely do more things like that now. We had to filter some things out of the public set because there were some very dumb things in the internal set.
I spend a lot of time looking at how people actually use our models. I’m someone who really likes to use our models. I think it’s very important that you test and play—really just play with your models. I came from Discord and open source communities, where you see people doing the stupidest things, but people are having fun with it. Besides covering actual use cases, including production use cases, I want to make sure the models are fun to use.
Good vibe eval—vibe bench.
I underappreciated this 3 years ago. Yes, it’s memes, and it’s a Midjourney subscription or whatever, but it actually becomes very real work.
Because they do it for fun, then they think, “Oh, I can do this at work.”
Exactly. When you automatically evaluate these models, there are a bunch of human evals everyone is doing in the field, but every evaluation has blind spots. You should combine as many independent evaluations as possible to get the best picture of how the model is performing.
That doesn’t remove the burden from you to sample the model yourself. It’s actually super fun to try to understand where the strengths and weaknesses are. Constantly sampling and checking the model is a core part of developing it. Is it where I’d expect it to be?
At some level, it’s what you guys care about too, right? These are your use cases. If you want something to get better, make a little benchmark and see how things perform.
Absolutely. I think it’s very important that the people building the models actually want to use them.
There’s a big distinction between that and a microphone brand trying to use it for product photos—a very niche use case they can min-max on.
It’s very general.
One thing I do question is Midjourney’s well-known loop: on Discord, everyone sees other people prompting. You get 4 options, you choose one, and that’s feedback for your next model. How come you didn’t do any of that?
We didn’t do it publicly at scale.
But isn’t it valuable?
It can be valuable, but only up to a certain extent. If you cross certain limits, the signal you get is pretty weak. In the early days, we saw the insane potential of that tuning on Midjourney’s models, like V5. But once your base model reaches a certain level of strength, you don’t get those super good examples where you can clearly see the model improving from this. There’s also a lot of noise.
There are a lot of other ways you can do this. I had this debate with someone: back then, it was very useful because prompt adherence wasn’t great. It’s pretty judgmental to decide which one follows the prompt better when neither of them does it well. You get a lot of preference tuning because there’s very high signal: people prefer this one.
Now, back to my earlier point: if I take the top 5 models and tell them to remove a bed, they’re all going to look the same to me. There isn’t that much signal in the preference, right? They’re all filling it in pretty well because they understand this task very well. So that side of the evaluation is kind of gone for me. It all looks fine.
I was going to jump in and say that was actually one of my favorite parts of moving into Kontext. Finally, it wasn’t just, “Did it actually do the task?” I really enjoyed getting to these very clear, concrete tasks.
9. Autoregressive or Diffusion
One more part of the text tree, and then we want to go into robotics in action. Autoregressive image generation seems like a big trend, with both Gemini and ChatGPT doing it. Is that as big an impact in your mind as it is in ours? It seemed like a big change because then you could edit conversationally. It’s the next obvious step from Kontext: instead of editing with an image and a text prompt, you’re working conversationally with a model.
I’d say that’s not a feature of autoregressive models alone. It’s based on the data you train on. You can get language capabilities into a diffusion model as well, which we know is working. In the long term, this is where we believe things will go, and where we’re working toward getting more modalities in, including language. So it’s not a question of the algorithm; it’s a question of what you focus on. I think both approaches are extremely exciting and have a lot of strengths and shortcomings.
I’d say autoregressive models are much more data-efficient. You can train them better on data because, for diffusion models, you have this trajectory from data to noise. Every training example effectively gives rise to infinitely many training examples, because you can pick any possible point to denoise from along the trajectory between data and noise. Obviously, the model would overfit if you trained only on that one example. Theoretically, one data point gives rise to infinitely many training examples. Autoregressive models, on the other hand, can train on all the tokens in parallel and get the maximum gradient from each training example.
That’s the training side, where I’d say autoregressive models are more data-efficient. At sampling time, though, autoregressive models have to generate one token after another because they’re iterative in the data direction. Diffusion models can distill 50 steps down to 4 and speed things up a lot. That isn’t possible in the same way with autoregressive models, because they have to generate token after token.
You make it sound like autoregressive generation is just worse, then.
Autoregressive generation has a limitation in that respect, but diffusion has a limitation in terms of data efficiency and training. I like both approaches. We’ve also been working on autoregressive models for a long time, and both are super exciting.
So, on data efficiency, autoregressive models win?
Yeah.
Okay. So, scale. One of the other big things you hear from people who’ve worked on both ChatGPT and Imagen is adding in-loop RL, so models can act agentically: generate something, reason about it, edit the image, and continue that workflow. They can do that too, and this is separate from pipelining diffusion generation, right? It’s all the same model doing it, trained at the data level, so you can scale that. Is there a counter to this in diffusion?
There’s no inherent limitation that would prevent a diffusion-based model from doing the exact same process and all of these things.
That’s really blowing my mind. I had the wrong idea. Nano Banana happened and I thought, “Holy shit, autoregressive is a thing.” But you’re saying it’s not. Am I wrong that Nano Banana is diffusion?
I think it is diffusion-based. I don’t have insider knowledge, so I might be wrong, but I’m pretty sure it has the language component to do autoregressive things. I think what we’d call it is a Transfusion model. It can essentially do both. There’s no inherent limitation saying you have to do it one way or another. These are explorations.
So it’s not settled consensus. I thought there was a wave of people shifting from diffusion to autoregression for Imagen 2.
There’s a lot of exciting research going on in that area right now. We’ve seen models work extremely well on that front. The same is happening with flow matching, which is also improving. I’d say the debate isn’t settled yet. There’s significant research on both fronts right now.
That’s exciting. The same is happening for robotics, too. We have VLAs versus VAPs, which is super interesting. The last thing is diffusion for text. There are benefits, of course, like speed. We see that clearly with text. People can play around with Diffusion Gemma, too. Will these models ever reach parity? Is it just a matter of investment in money and data?
I’d say a lot of it is data. If it doesn’t work, why doesn’t it work? My simple answer is that we reason left to right; we don’t reason by diffusing into vision. At the same time, I think people like to focus on the step-by-step nature of tokens in autoregressive models, but they can also parallelize. We can take in a whole book at the same time, too, which those models can do.
Yeah, you can fake it in. I totally get it. It’s a valid question: how come diffusion for text never took off? It’s slowly getting slightly better, but how?
I think it’s a matter of which approach came earlier and became the standard for language models. As you said, if it’s more data-efficient and the world has been doing a lot of scaling, that gives it an advantage. That doesn’t mean there aren’t pockets of use cases where you can get really good results at a certain size. Diffusion Gemma is a good example: it’s pretty good, meets a bar, and can run really fast. If you want that, great.
On the autoregressive language front, I think it really comes down to the fact that it was simple, scalable, and there early. It’s clearly been a very successful trajectory. A lot of time, money, and research have gone into making it work. There’s a lot of exciting research on the diffusion front now, and I’d separate the idea of diffusion language into 2 camps. A lot of people don’t realize this: there are discrete diffusion language models, which I almost don’t like to call diffusion models. Diffusion Gemma and the Mercury models are in this camp. They’re more like masking-style models, where they jump from a mask to a token, but they’re still in the discrete space.
Now there’s a big wave of research on continuous diffusion language models. Those are actually operating in a denoising space. I don’t want to call them flow-matching models, but they’re essentially operating that way.
So, more like in— This was used for audio, if I’m not mistaken.
Audio generation, continuous diffusion. Continuous diffusion. I mean, well, diffusion has also been used for—
It’s for everything, yeah: visual modeling, audio modeling. My last fun question there is: do we have a good way to do GRPO-style RL on pure diffusion text models?
Yeah, we’re doing this, actually. For FLUX.3, we did this, and it has had tremendous gains.
So, FLUX Action is part of the very recent launches and work you’re doing. How do you approach this? What’s the feedback and impact so far?
10. FLUX Action and Robotics
Basically, you pre-train on videos, images, audio, everything together, and then you have this very general model that understands the world. Then you can take a bit of action prediction data. Basically, you add predictions in the state space of this robot, which is a parameterization of the robot arm. Each joint is described by a translation and a rotation: 6 degrees of freedom, basically. Then you predict that together with a video.
Since the model can already reason about the video and what it needs to create to fulfill a certain task in video space, it’s simple to add this new dimension, this state space, and predict that as well. Given a chunk of earlier video, you condition on that and enter an instruction. The model creates the next chunk of actions to fulfill that action.
This works extremely well. We can do it in the real world, with real-world robots, and in a simulation. Computer use is also a mapping of this to the digital world, and we can make the model play games, right, and stuff like that.
So this is all enabled by this single FLUX.3 model. You also see that it’s extremely capable because we have this very general pre-training, which isn’t limited to classic robot data. It uses all kinds of data that gives the model an inherent understanding of the world, and you see that in its performance.
Being a diffusion model is also pretty speedy, as we see here, with these distilled versions, right? Would you call this a world model or a world action model? Do you have a term you like to use?
It’s a world action model. I don’t like to use the term “world model.” It’s so overloaded.
I’m in the same boat. It feels like a homonym at this point.
Let’s call this a visually intelligent model because it can do all these basic things, right?
I guess the distinction I see in these models is that you’re not just predicting video or the world; you’re also predicting action states, right?
Yeah. One is what’s happening in the world, what the state is, but also what the next action to predict is.
Exactly. So it’s sort of two outputs, right?
Yeah.
Is there a target frame rate that you aim for as an industry standard? Because you don’t have to predict that far out, is my idea.
Do you mean in terms of the real-time factor? It’s 10 FPS, I think, what the model currently runs at. It’s a very small model.
And you’re projecting out one frame, 5 frames, whatever, right?
I would have to look it up. I think it’s a small chunk of frames, somewhere between 9 and 21 frames, that it predicts. Then the model is conditioned on the previous generations again, and on what the camera perceives when it runs on the robot. Then you predict further. So it’s a bit of a chunk. It’s not running autoregressively in time, so to say, but it’s chunk prediction.
What I’m driving at is that this video prediction side is very much for predicting the results of what you do. So probably you need some kind of long-range planner outside that model to coordinate the actions.
I mean, this is why we were discussing the language side earlier.
Yeah. So as it runs right now, there’s no high-level planner involved or something?
Yeah. This is just a model running on a loop.
Some of the interesting things here: this is what we did with Dyna Robotics.
Yeah.
Dyna has one, Runway has one. I guess let’s talk about the fine-tuning side. If I want to take your general model and fine-tune it for a new robot with a new task, something out of distribution that it hasn’t seen—say, my own home robotics, where I can program in how to use a humanoid arm—what’s that process like? How long does it take? How important is fine-tunability?
Since this model is so general and the pre-training was so good, you need a bunch of hours. I think we showed one example where we compared it with Pi0.5, which was also open source. It took 8 times less data to train on it. That was for an industry application. We needed 4 hours of data to get to a success rate that was good enough for this basic application.
Is that something you want to help improve right now?
Yeah. We go back to the context thought. We shouldn’t be doing fine-tuning. What we’re targeting and working on right now, extremely carefully and with a lot of focus, is in-context learning for robotics.
You should be showing a robot an action one time, and then it should be able to do it. This would truly be superhuman intelligence. If you look at how long it takes us as children to learn to walk, or to use our thumb correctly to grab this can, for instance, it takes weeks or even months. Our vision is that an autonomous robot should be able to observe any action once and then be able to do it. That would be clearly superhuman visual intelligence, and that’s what we want to reach.
What’s the standard now? How many demonstrations do we need?
Around 4 hours of fine-tuning, at least.
Four hours, yeah.
Even 4 hours is pretty good. I’d say 2027 will be the year we crack this task: single-shot learning.
“Mid-training” is an interesting one here, too. Is this like going from manufacturing to, say, medical stuff, like filling pill capsules? Is that where—
The model is trained quite generally. Even before fine-tuning, it might be able to handle many of the tasks you throw at it. This version might be able to handle many tasks. Fine-tuning is really just nailing it.
In the robotics world, you want a certain level of accuracy you expect out of it.
Exactly. This is just standard plain large-scale mid-training. Then we move on to the fine-tuning stage, where we add the high-quality robotics data and get a further bump. But you see that even with mid-training, the model can already do these tasks to a somewhat good extent.
I guess the other interesting thing is that it’s 7B, right? Is that mostly because these things are deployed on device?
Exactly. You want a model that’s as small as possible because it needs to act on device if you’re adding this to a manufacturer, and it needs to run in real time.
And this is on the backbone of FLUX.3?
Yes.
Is the image and audio stuff all derived from this?
All of it. This is derived from one model, basically. We have this multimodal model. That’s really important to understand. It was trained on a huge corpus of images, videos, and audio, and from that we can derive everything we released this summer: FLUX.3 Video, FLUX.3 Video Edit, FLUX.3 Action, FLUX.3 Image, and FLUX.3 Image Client. We also derived an upscaler from it. We have a really nice 4K video upscaler, which is super fast, too. I can only recommend it. It’s really good.
This is all powered by this one model, and it shows how training these very general models compounds over time, because you can derive all these applications from them.
Is there a difference between a video upscaler and an image upscaler? Isn’t it—
For a video upscaler, we talked about temporal consistency. If you had an image scaler with no temporal understanding, you’d get this flickering again. Especially in the high frequencies, you don’t see it much when you compare two images side by side. But if you switch between them, you notice it. One of the basic tasks everyone who looks at a lot of images does is opening two images in two tabs.
You do notice, yeah.
And then switching left and right, you see every single difference. That would amount to flickering in temporal space if you didn’t have this temporal component in the video upscaler.
Got it. And that’s what makes the difference from an image upscaler. Any notes on real robot evals and how to scale this? I see “sponge in bowl” three times. At some level, if you look at our DynaPod, there are literally people testing robots one-off because you have to test at real scale. But come on, two out of three runs on “sponge in bowl”—just restart and do it again. You can be three out of three, right?
It’s a hard problem, and I think one thing that’s constantly being worked on is closing the sim-to-real gap. This will get better and better over time. But to be honest, evals are painful no matter where you do them.
They’re also painful when you look at content creation evals. Ask three different groups of voters what a good image looks like, and you’ll get three different answers. You get wars over it. We have wars over it internally. Evals are just hard for generative modeling.
Is the commercial side already developed? How developed is it?
We’ll have more of these coming soon. I can’t tell too much, but there will soon be the first BFL-powered robots live in production.
And it’s open, right? 2027 will be the year of robotics. Say more on predictions. Where do we go from this in 2027? How fast do things pick up?
Single-shot learning. In-context learning is a big thing we’ll see first. We’ll see the first commercial applications of these things in real manufacturing. That’s why it’s good that we have a German base, because there’s a ton of industry data and lots of potential for partnerships. People are really hungry for robotics, too. There are good synergies that we want to use to make 2027 the year of robotics. I’m very excited about it.
The research is also moving very quickly, so I suspect robotics will improve very quickly.
What about the fact that robotics has physical manufacturing constraints? You can scale up tech and models, but that’s pretty hard to affect. Predictions for 2027: where are we realistically on robotics? Can I buy something off the shelf that can single-shot learn?
I suspect that later in 2027, people will be able to get some quite interesting robotics in their house. Whether it’ll be able to do fine motor tasks and all that, I don’t know. This is entirely predictive; I have no baseline for it. But I suspect next year there might be some interesting, potentially useful robotics in your house, at least for a niche of tech people like us—not the average person.
And for manufacturing?
I think 2027 will be a year for manufacturing.
Because manufacturing is more restricted than the long tail of autonomous driving or consumer robotics, where ideally you’d want it to do anything you want.
There’s a very good chance we’ll see a lot of applications unlocked by robotics in 2027. That’s what we’re working toward, basically.
Awesome. Exciting. It seems like a very fun place. You guys have robot labs, an image and video model, and you’re building the future. You’re just missing the flying car.
You mentioned Germany a lot, but you also have a pretty big San Francisco office. I didn’t know there was a big European side.
There’s a substantial San Francisco presence. It’s called Black Forest Labs. We invite everyone to come over to our place at 410 Townsend Street. We’re not a European company; we’re a company with roots both in Europe and the US, and we’re acting globally.
That’s fun. We got you on because you’re visiting town, and we thought, “Okay, we should have you on.” How can people help you? Are you hiring?
I think just supporting us in marketing and talking about our models. Help us get our name out here in the US. In the community, we’re well known. In broader society, not that much. There are the big players, and that’s what we need to change in 2027: really getting our name out, because we have extremely strong technology. We have a long history of building the core pillars that everyone is using. It’s time to get our name out to a broader audience.
The tech community knows you, but beyond that, not so much. Tell your mom about FLUX.
I’m a marketing guy, and I help companies with marketing all the time. You guys are very different, because generative media in general has a mixed reputation with the broader population. Normally, I’d suggest a TikTok or Instagram marketing campaign, but you’d get killed because people hate AI so much. It’s interesting to find the niches where this isn’t regarded as AI. It’s just a normal thing you can do with tech. The moment you say it’s AI, the population will have pitchforks. Do you have any response, thoughts, or explanations about that?
I think it’s important to emphasize that we take this responsibly. Our models aren’t creating avalanches of slop. They live in real use cases and professional workflows, and they help make professional people’s lives much easier and increase productivity so much.
It’s Photoshop.
Exactly. It’s Photoshop. It’s crazy how much more productive you are. That’s really the goal on the content side. It’s not about putting people out of work at all. It’s about making everyone’s life so much easier.
I wouldn’t say that. If I worked on your team, this would be my go-to-market problem right now: how do we get people to go viral the way Wispr Flow does a little bit? People say, “I understand how to use this. This is not AI. It’s just Wispr Flow.” Or it’s Granola, same thing.
Generative media hasn’t had that crossover, except for Nano Banana, and whenever ChatGPT had the image generation Studio Ghibli moment. Those are the moments where people say, “This is a fun trend I want to hop on.”
There were moments that hit well. Your point about seeing yourself when you’re older—that had a huge moment. People would look at themselves in their 80s and send it to their family. There have been moments, but nothing super predictable. It’s very hard to predict one of these viral moments. Nano Banana was their internal name, and they kept it because people liked the name and it stuck.
One big use case we saw a ton of was image restoration, which we didn’t necessarily expect. I saw amazing examples of people taking old, degraded family photos and restoring, colorizing, and fixing them up, while keeping a very high level of similarity to the original. That was one of my favorite moments: people showing their families these restored photos and seeing their reactions. I really enjoyed that positive moment. But, like I said, so often we can’t predict what the thing is going to be. Even back in the Stable Diffusion days, it was always surprising what the trend was online. They just emerge.
You just never know. They emerge.
You cannot plan them. Once you have them, you need to exploit them, obviously. You should always be asking, “What’s the next big thing? What’s the next capability that models haven’t been able to do?” Then try to bake that into your model.
Absolutely, yeah. Making bets. Some of them will work, and some others will not. But it’s extremely important to always have this mindset: “Hey, what’s the next thing we should be working on that will bring a lot of joy or productivity increases to people?”
Yeah. Well, that is the domain of research. You guys are doing an amazing job, so thank you so much for sharing your story. I hope more people come to know you as a team, because they’ve seen a lot of the model buzz. But I feel like this is also the first time I’ve heard the full story of BFL.
Yeah. No, thank you so much for having us.
Thank you so much.
It’s an amazing presentation. It’s fun to finally have gotten to do this.
Yeah. I mean, it’s a year in the making. I bought a bunch of MicroDucks, and I’m looking forward to them running around.
Oh, do you have one? I want to get my hands on one. I feel like I missed the window when these were available.
You have connections. I’m sure I can—
Whoever out there can get me a MicroDuck, get me a MicroDuck!
I’ll get you a MicroDuck.
Okay, perfect.
I bought a bunch because I know this is going to be a hot gift for Christmas.
Fantastic.
But, yeah, robotics in 2027—that’s one year away. We’ll see where this all lands. I’m very optimistic about this approach, and I think you guys have explained it very well.
Fantastic. Yeah. Amazing. Awesome. Thank you.
Cool. Thanks so much.
All right. Thank you.