AI Engineering for Art - with comfyanonymous

I'm Alessio Fanelli, partner and CTO of Sable Partners, and I'm joined by my co-host, swyx, founder of Smol AI.

Hey, everyone. We are in the Chroma Studio again, but with our first-ever anonymous guest, comfyanonymous. Welcome.

Yeah, well, hello.

I feel like that's your full name. You just go by Comfy, right?

Yeah. A lot of people just call me Comfy, even when they know my real name.

Hey, hey, Comfy.

swyx is the same. Not a lot of people call you by your legal name.

You have a professional name that people know you by, and then you have a legal name.

Yeah, it's fine.

I think people who are in the know know that Comfy is the tool for image generation and now other multimodality stuff. When I first got started with Stable Diffusion, the star of the show was AUTOMATIC1111. I looked back in my notes from 2022, and Comfy was already getting started back then, but it was kind of the up-and-comer. Your main feature was a flowchart.

Can you rewind to that moment, that year, and talk about how you looked at the landscape and decided to start ComfyUI?

I discovered Stable Diffusion in October 2022, and I started playing around with it. Back then, I was using AUTOMATIC1111, which was what everyone was using.

When I started, I had no idea how diffusion models worked or how any of this worked. I hadn't written a line of PyTorch before that, so it was completely new.

What was your prior background as an engineer? Any experience with image processing, orchestration, distributed systems, or GPUs?

Just a software engineer. A boring software engineer. I wasn't doing anything interesting—CRUD web development.

Well, not web development, really. Just some basic automation stuff. No big companies or anything.

But you already had some interest in automation and probably a lot of Python?

Yeah, of course, Python. But I wasn't used to the node-graph interface before I started ComfyUI. I was just thinking, “What's the best way to represent the diffusion process in the user interface?” I thought a node graph was the most natural and best way I had found.

I started writing the code on January 1, 2023, and I released the first version on GitHub on January 16, 2023. That's how things got started.

Was it called ComfyUI right away?

Yeah, ComfyUI. The reason is that my name is Comfy. People thought my pictures were comfy, so I just named it ComfyUI.

Were you doing that mostly on your own when you started, or did you already have a subgroup of people?

I was on my own. It was just me experimenting with stuff.

I had gotten addicted to generating images, as we all did. Then I started experimenting with Hires.fix in AUTOMATIC1111. For those who don't know, Hires.fix was a way to generate higher-resolution images, since the diffusion models back then could only generate images at a low resolution.

You would generate a low-resolution image, upscale it, and then refine it again. That was the hack for generating high-resolution images. I really liked generating higher-resolution images, so I started experimenting with it and modified the code a bit.

I thought, “What happens if I use different samplers on the second pass? What happens if I use different settings or a different number of steps?” I edited the AUTOMATIC1111 code to try those things.

Back then, Hires.fix was very basic. I wanted to go further. I wanted to try using a different model for the second pass. The AUTOMATIC1111 code wasn't good enough for that, though. It would have been harder to implement in the AUTOMATIC1111 interface than to create my own interface, so that's when I decided to create my own.

Was there a particular segment of the community that you targeted as users—more intensive workflow artists, compared with the AUTOMATIC1111 crowd?

This was my way of experimenting with new things. With the Hires.fix feature I mentioned, the first thing you could easily do in ComfyUI was chain different models together.

One of the first times it got some popularity was when I started experimenting with applying different prompts to different areas of the image. I called it area conditioning and posted it on Reddit. It got a bunch of upvotes, so I think that's when people first learned about ComfyUI.

Was that mostly for fixing hands?

No. It was still difficult to do things like, “I want a mountain here, and I want a fox here.” Compositing the image that way was very easy.

When you run the diffusion process, you generate the entire image at every step. You do one pass for the whole image, one pass for one area with one prompt, another pass for another area with another prompt, and then average everything together at every step. That was area composition, which I called area conditioning.

A month later, a paper came out called MultiDiffusion, which was essentially the same thing.

Could you do area composition with different models, or do you need to use the same model because you're averaging everything together?

You could do it with different models. I had implemented it for different models, but the models have to share the same latent space. For example, you couldn't use an SDXL model and an SD 1.5 model, because they have different latent spaces. But you could use different SD 1.5 models.

There are some models that try to work in pixel space, right?

They're very slow. That's the reason Stable Diffusion became popular: the latent space is small. Pixel-space diffusion models are just too slow.

Have you ever tried to talk to Robin Rombach and that crew—the latent-diffusion people?

I used to work at Stability AI.

That's the part of the story I didn't know.

I got hired in June 2023. The reason I was hired is that they were doing SDXL at the time. SDXL had a base model and a refiner model, and they wanted to experiment with chaining them together. They saw ComfyUI and thought, “We can use this to do that. Let's hire that guy.”

They didn't pursue the same approach for SD3, though.

What do you mean—the SDXL approach?

The reason for that approach was that they had 2 models and wanted to publish both of them. They trained one on lower timesteps, which became the refiner model, and trained the first one normally.

During testing, they realized that if they chained the models together, the quality increased, so they decided to publish both. It worked. But I don't think many people use the refiner anymore, even though it is a full diffusion model. You can use it on its own, and it will generate images. People have mostly forgotten about it.

Can we talk about models a little bit? Stable Diffusion is obviously the best known, and I know FLUX has gotten a lot of traction. Are there any underrated models that people should use more? What's the state of the union?

The latest state of the art, at least for images, includes FLUX and SD 3.5. SD 3.5 has 2 models: a small one at 2.5 billion parameters and a larger one at 8 billion parameters. It's smaller than FLUX, and it's more creative in a way.

FLUX is probably the best overall. People should give SD 3.5 a try because it's different. I won't say it's better, but it's better for some specific use cases. If you want something more creative, SD 3.5 might be better. If you want something more consistent, FLUX is probably better.

Do you ever consider supporting the closed-source model APIs?

We support them through custom nodes. We actually have some official custom nodes from different providers.

I assume DALL·E would have one.

Yeah, but I'm not the person who handles that.

There's been a lot of community discussion about the transition from SD 1.5 to SD 2, and then from SD 2 to SD 3. People are still very loyal to the previous generations.

SD 1.5 still has a lot of users. It's the last base model.

SD 2 was mostly ignored because it wasn't a big enough improvement over the previous one.

So, SD 1.5, SD 3, FLUX, and SDXL. Is that the main group?

Stable Cascade was also a good model. The problem is that SD 3 was announced 1 week after Stable Cascade was released. It was a strange release.

What was it like inside Stability AI? The statute of limitations has expired, and management has moved on, so it's easier to talk about now.

That model was ready about 3 months earlier, but it got stuck in red-teaming. If the model had been released by the authors, it probably would have gotten very popular. It was a step up from SDXL, but its momentum was stolen by the SD 3 announcement.

People didn't develop much on top of it, so it was mostly ignored for some reason. It was a good model.

The naming also matters. It seemed like a branch off the main tree of development.

It was developed by different researchers. They were the Würstchen authors. I met them in Vienna. They worked at Stability for a while and left right after the Cascade release.

This is Dustin, right?

No, Dustin was SD3. SDXL was Pablo and Dominik. I think I'm pronouncing their names correctly.

Yeah, that's very good.

It seems like the community moves very quickly. When there's a new model, people just drop whichever one they're using and move wholesale to the new one. They don't really stay to explore the full capabilities of the previous model. If Stable Cascade was that good, people could have A/B-tested it more, but instead they said, “SD3 is out. Let's go.”

I find the opposite, actually. The community doesn't jump to a new model unless there's a significant improvement. If there's only an incremental improvement, which is what most of these models are going to have—especially if they stay at the same parameter count—you're not going to get a massive improvement unless there's something big that changes.

How are they evaluating those improvements? There's a whole chain of ComfyUI workflows. How does one part of the chain affect the whole process?

It depends on the specific workflow. Most workflows are compatible between different models, but you might have to completely change your prompt.

Maybe the question is really about evaluations. What does the ComfyUI community do for evals?

They don't really do formal evaluations. It's more like, “I think this image is nice.”

They just subscribe to fal.ai and see what fal.ai is doing?

They just generate images. I don't see anyone really doing scientific comparisons, at least on the ComfyUI side. ComfyUI users are more focused on generating images and seeing whether an image is nice.

The more scientific checking happens on the model side. There are also a lot of vibes involved, because it is an artistic medium. You can't create a very good model that doesn't generate nice images, because most of the images on the internet are ugly.

If you say, “I have the best model. It's super smart. I trained it on all the images on the internet,” the images aren't necessarily going to look good. They'll be very consistent, but they won't have the look people expect from a model.

Can we talk about LoRAs? We talk about models, and then the next step is probably LoRAs. I'm curious how LoRAs entered the toolset of the image community. The LoRA paper was published in 2021, and there were other methods, like textual inversion, that were popular in the early Stable Diffusion era.

Textual inversion is basically training a vector that you pass to the text encoder. You're training a new word.

You take the words in your prompt and convert them into tokens with the tokenizer. Those tokens are converted into vectors. Each token represents a different vector, and depending on your words, that's the list of vectors passed to the text encoder.

The text encoder is a stack of attention layers, so it's quite close to an LLM architecture. What you're doing is training a new vector. You have all these images and want to know which word represents them. You train that vector, and when you use it, it hopefully generates something similar to your images.

I would say it's surprisingly sample-efficient at picking up the concept you're trying to train it on.

People have mostly stopped doing that. When I was at Stability, we trained some textual inversions internally on T5-XXL, and they worked pretty well. For some reason, people don't use them.

They might work across models, too. I would have to test it, but if you train a textual inversion on T5-XXL, it might work with all the other models that use T5-XXL.

The textual inversions trained for SD 1.5 also work on SDXL, because SDXL has 2 text encoders, and one of them is the same as the CLIP-L encoder in SD 1.5. They don't work as strongly because they only apply to one of the text encoders.

The same thing happens with SD3. SD3 has 3 text encoders, so you can still use a textual inversion trained for SD 1.5 with SD3, but it's much weaker because it gets diluted across 3 text encoders.

Do people experiment much on just the CLIP side? There are models like SigLIP and BLIP. Do people experiment with replacing the text encoder?

You can't really replace it, because the model and the text encoder are trained together. What I've seen people experimenting with is Long CLIP. Someone fine-tuned the CLIP model to accept longer prompts.

That's a little bit like long-context fine-tuning.

Yeah. Regular CLIP is 77 tokens, and Long CLIP is 256 tokens.

The hack in Stable Diffusion 1.5 is that it still works if you use prompts longer than 77 tokens. You split the prompt into chunks of 77 tokens and pass each one through CLIP. Then you put everything together at the end.

It's not ideal, but it works. If someone gives it a massive prompt—the Bible, for example—it splits it into chunks of 77 and processes each one. The positioning of the words really matters, so the concepts at the end can still be present in the image.

And while we're on this topic, prompt weighting and negative prompting are all part of this layer of the stack, right?

The hack for prompt weighting works well on CLIP. In SD 1.5, prompt weighting works well because CLIP-L isn't a very deep model. There's a high correlation between the input token, the index of the input token vector, and the output token. The concepts are very closely related.

If you interpolate the vector, you have the CLIP output for the negative prompt and the CLIP output for your prompt, and then you interpolate between them depending on the prompt weight.

That's how ComfyUI does prompt weighting. You have the vector for your prompt and the vector for the empty prompt, and it interpolates between them based on the weight of the tokens.

This stops working as the text encoder gets deeper. On T5-XXL, it doesn't work at all.

Is that a problem for people?

You just use words to describe what you want, because it's a larger language model. Honestly, it might be fine. I haven't seen many complaints about it not working in FLUX. I guess people can get around it with language.

Coming back to LoRAs, the popular way to customize models is with LoRAs. I saw that you also support LoCon and LoHa, which I'd never heard of before.

A LoRA works by fine-tuning smaller weights instead of the entire model. Fine-tuning the entire model is heavy, so you can fine-tune smaller weights to make things faster and less demanding.

You train 2 low-rank matrices. When you multiply them together, they represent the difference between the trained weights and the base weights. Training those 2 smaller matrices requires much less computation.

They're also portable and easier to share because they're smaller.

For inference, you apply the LoRA directly to the model weights. There's only a small delay before sampling, when it applies the weights, and then it runs at the same speed as before.

All the LoRA types—LoHa, LoKr, and everything else—are different ways of representing that difference. You can think of it as compression, even though it's not really compression. It's just different ways of representing the difference in the weights.

The basic LoRA says, “Let's multiply these 2 matrices together.” The others are different algorithms for representing that same difference.

Let's talk about what ComfyUI actually is. Most people have heard of it, and some people may have seen screenshots, but fewer people have built very complex workflows.

When you started, AUTOMATIC1111 was the simple way to use these models. What choices did you make? The node workflow is there, but is there anything else that stands out as a unique take on image-generation workflows?

Everyone was trying to make an easy-to-use interface, so I thought, “Everyone is trying to make an easy-to-use interface. Let's make a hard-to-use interface.”

I didn't need to do what everyone else was doing. I wanted to make a powerful interface, even if it wasn't easy to use.

There's a node execution engine. Your README lists a really good set of features that you prioritized: re-executing only the parts of the workflow that changed, an asynchronous queue system, smart memory management, and so on. That seems like a lot of engineering.

There's a lot of engineering in the backend to make things work locally as well as possible. I was always focused on making things work locally, because that's how I was using it.

ComfyUI is more of a backend—at least it was before. Now the frontend is getting a lot more development.

Version 0.1 was only released in August this year?

That was before the version naming changed.

What was the big rewrite for version 0.1 and then version 1.0?

That was mostly on the frontend side. When I first wrote it, I thought, “How can I make a node interface? I can do web development, but I don't like doing it. What's the easiest way I can slap a node interface on this?”

I found a JavaScript library called LiteGraph. It already had the whole node interface, so I just plugged that into the backend. I didn't want to spend time developing the frontend.

If Streamlit or Gradio had offered something similar, would you have used them? They're Python-based.

I don't like Gradio. That's one of the reasons AUTOMATIC1111 was so bad. Gradio makes your interface logic and backend logic stick together.

It's supposed to be easy for Python developers. It makes it easy to slap a quick interface on your machine-learning project, and that's what it's made for. There's no problem using it for that.

But if you want to make real software that will last a long time and be easy to maintain, I would avoid it. The frontend and backend should be well separated with a defined API. That's how software is supposed to be made.

Would you say Streamlit has the same problem?

I haven't used Streamlit as much, but it seems to have a similar philosophy. For quick AI demos, it's perfect.

Going back to the core technology—asynchronous queues, selective re-execution, and smart memory management—was anything particularly difficult to figure out or something you're especially proud of?

The biggest pain in the ass is probably memory management.

Were you just paging models in and out?

Before, it would load the model completely, unload it, load the new model completely, and unload it. That works well when the models are small.

But if the models are large—for example, if someone has an RTX 4090 and the model is 10 gigabytes—loading and unloading can take a few seconds. You want to keep things in GPU memory as much as possible.

What ComfyUI does now is estimate how much memory a sampling operation will probably take. It removes enough of the models already loaded on the GPU to make room and then executes the operation.

There's a fine line, because you want to remove the least amount of models that are already loaded. On Windows, the NVIDIA driver creates another problem. By default, it automatically starts paging to system RAM when you overflow your GPU memory, although there's an option to disable that feature.

That makes everything extremely slow. When people complain that a model works but suddenly slows down a lot, that's probably what's happening.

You have to use as much memory as possible without using too much. Otherwise, things start slowing down or you run out of memory. You have to find the point where the Windows driver starts paging.

PyTorch is also annoying because its high-level APIs don't give you much fine-grained control over specific memory operations. You have to leave a lot of the memory freeing to Python and PyTorch.

As a maintainer, you're designing for a very wide surface area of compute. You even support CPUs.

That's just PyTorch. Supporting CPUs isn't difficult.

Is there a market-share estimate? Is it 70% NVIDIA, 30% AMD, and then miscellaneous devices like Apple Silicon?

For ComfyUI, I don't know the market share, but I think it's mostly NVIDIA.

AMD works horribly on Windows. On Linux, it works fine. It's slower than the price-equivalent NVIDIA GPU, but you can use it to generate images and everything works.

The problem is that most people who bought AMD GPUs probably use Windows, and they probably aren't going to switch to Linux. Until AMD ports ROCm to Windows properly—and until there's a good PyTorch ROCm build that works on Windows—they're going to have a hard time.

We have to get George Hotz on that.

He's trying to get Lisa Su to do it.

Let's talk a bit about the node design. Unlike the other text-to-image tools, you expose a very deep set of controls. There's a separate node for CLIP and a separate node for the sampler, with all these different pieces.

How much do people actually play with the settings? How do you guide people toward the parameters that have a major impact versus the ones that are less important but you still want to expose?

I try to expose everything. For the samplers, there are 4 different sampler nodes that go from easiest to most advanced.

The regular sampler node has just the basic settings. If you use the advanced sampler node, you can access the individual components and settings.

What are the most impactful parameters? Which ones really make a difference?

They all have their own impact. For example, with steps, you usually want them to be as low as possible. If you're optimizing your workflow, you lower the number of steps until the images start deteriorating too much.

The number of steps is how many times you run the diffusion process. If you want things to be fast, lower is better.

CFG is more like the contrast of the image. If the image looks too burnt out, you can lower the CFG. CFG controls how strongly the negative prompt is applied relative to the positive prompt.

When you sample a diffusion model, it's basically a positive prediction minus a negative prediction. CFG is the multiplier.

What are good resources for understanding what these parameters do? Most people start with AUTOMATIC1111 and then move over. They see settings like steps, CFG, sampler name, scheduler, and denoise, but they may not know what they mean.

You should try them out yourself. You don't necessarily need to know exactly how they work to understand what they do.

For example, if CFG is 1.0, the negative prompt isn't applied. It also means sampling is twice as fast. But other than that, you should see what the settings do to the images yourself. You'll get a more intuitive understanding that way.

Are there any other nodes or features you want to shout out? IP-Adapter and the AnimateDiff tools seem to be among the most popular.

Not specific nodes, but I like when people build things that use ComfyUI as the backend. There's a plugin for Krita that uses ComfyUI as its backend, so you can use all the models that work in ComfyUI in Krita. I've tried it once, but I know a lot of people use it and probably do some nice things with it.

What's the craziest node that people have built? What's the most complicated or unusual thing you've seen?

Some people have made video games in ComfyUI. Last year, someone made a Wolfenstein-style game in ComfyUI. One of the inputs allowed you to generate a texture, and then it changed the texture in the game.

You could plug that into a workflow. If you look around, there are a lot of crazy things people do.

There's also a node registry that people can use to download nodes.

There has always been ComfyUI Manager, but we're trying to make things more official with the node registry.

Before the node registry, how did a custom node get into ComfyUI Manager? The person running it searched GitHub every day for new custom nodes and added them manually to the custom-node manager.

We're trying to make that require less effort for him.

I was looking at the available nodes. There's a YouTube download node. This is almost a data pipeline more than an image-generation tool at this point. You can get data in, apply filters to it, and generate data out.

You can do a lot of different things. I made it easy to create custom nodes, and I think that helped the ecosystem a lot. It's very easy to make a node.

Sometimes it's too easy. Then we have the problem that many custom-node packs share similar nodes. That's something we're trying to solve by bringing some of that functionality into core.

People can also do video generation now.

The first video model was Stable Video Diffusion, which came out last year.

I don't consider it a true video model. It generates video, but it still has a 2D latent space. They took SD 2, added temporal attention to it, and trained it on videos. It's similar to AnimateDiff.

A true video model, like Mochi, has 3D latents, so you can move through space. Mochi also has a temporal VAE that compresses in the temporal direction. AnimateDiff and Stable Video Diffusion only compress spatially, not temporally.

That's why I call Mochi a true video model. There are a few of them, but Mochi is the one I've implemented in ComfyUI because it seems to be the best one so far.

Another open model I've seen is CogVideoX.

CogVideoX seems decent too. There are a few others that were released around the same time, but I can't remember their names. The rest are closed source, like Kling.

OmniGen also released around the same time, which seemed interesting.

I think SD 3.5 and Mochi were released on the same day, so everything else was completely drowned out. A lot of people picked that day to release their models for some reason.

What's the relationship between ComfyUI and comfy.org? You are Comfy, and then there's comfy.org. I know we do a lot of news and research, and those people have a more open-source-oriented project going on. How do you work together?

I should explain the rest of the story.

The first version of ComfyUI was released to the public on January 16, 2023. At the end of January or beginning of February, I made the Reddit post about area composition. A YouTuber, Olivio Sarikas, made a video about ComfyUI in March 2023. I think that was its first real burst of attention.

I continued developing it, and more people started using it. Unfortunately, that meant my time to experiment started going down, because I had to add all these features and stuff.

Then I got hired by Stability AI in June 2023. They hired me because they wanted to work on SDXL.

When SDXL was released, Stability released the code first but not the model checkpoint. They gave early access to people who signed up, and they only allowed people with edu email addresses to access SDXL 0.9.

Of course, it leaked. If you do that, it's going to leak.

The only way people could easily use it was ComfyUI. People started using it, and I fixed the issues they were having. Then the big SDXL 1.0 release happened.

ComfyUI was the only way many people could run SDXL on their computers, because AUTOMATIC1111 had a quick implementation that was so inefficient and bad that it just wouldn't work for most people. People with regular GPUs couldn't run it effectively. They didn't have much choice, so they used ComfyUI.

That was the growth hack. I have an RTX 4070, so think of me.

Right now, we're hiring. On the core itself, it's mostly me, but all the focus has been on the frontend because that's the thing that had been neglected for a long time.

We'll soon have more people to help with the backend. Once we have the version 1 release—the packaged version with a nice interface that's easy to install on Windows and hopefully Mac—there will be a lot to do on both the backend and the frontend.

What's the timing? I'm on the waitlist.

Soon. I don't want to promise a release date, because we do have a real target date, but I'm not sure whether it's public.

We're going to continue making ComfyUI the best way to run Stable Diffusion models locally—at least on the open-source side. It will be the best way to run our models locally, but we'll also have a few ways to make money from it, such as cloud inference and things for enterprises.

How do you feel about the other Comfy startups? I think it's great that they're using your name.

It's better for them to use ComfyUI than something else. We don't want to stop people from using ComfyUI, because it helps the ecosystem.

Even if they don't contribute directly, the fact that they're using ComfyUI means more people are likely to join the ecosystem.

Would you ever do text generation?

You can already do text generation with custom nodes. It's something I've wanted to add to core eventually, but it's not a very high priority.

A lot of people use text models for prompt enhancement and similar things. It's useful, but my focus has always been on vision models. If an open text-diffusion model comes out, I'll probably implement it, since it fits with the whole system.

David Holz is investing a lot in text diffusion.

If a good open model comes out, I'll probably implement it.