Teaching AI to See: A Technical Deep-Dive on Vision Language Models with Will Hardman of Veratai

Hello, happy New Year and welcome back to The Cognitive Revolution. Today I’m excited to share an in-depth technical survey covering just about everything you need to know about vision-language models and, by extension, how multimodality in AI systems currently tends to work. My guest Will Hardman is founder of AI advisory firm VerAI, and he’s produced an exceptionally detailed overview of how VLMs have evolved from early Vision Transformers to CLIP’s pioneering alignment work to today’s state-of-the-art architectures like InternVL and Llama 3 Vision. We’ll examine key architectural decisions, high-resolution images and documents, and evaluation frameworks like MMMU and BLINK. We’ll also explore Flamingo’s Perceiver Resampler, InternVL’s dynamic high-resolution strategy, LLaVA’s synthetic data generation, and Qwen-VL’s multistage pretraining. Will has spent an estimated 40 hours preparing for this episode, and his detailed outline is available in the show notes. Will Hardman, AI adviser at VerAI and AI scout on all things vision-language models. Welcome to The Cognitive Revolution.

Thanks, Nathan. Great to be here.

I’m excited about this. We’ve talked about this for a few months now, and you have put a real Herculean labor into a very deep dive into all of the techniques, data sets, different strategies, and variations involved in making vision-language models work.

I think this is going to be a really interesting crash course and overview on all of that. It’s something that I know I want and need, and I think a lot of people will really benefit from getting the fast-forward version of all the research you’ve done. Thanks for putting all the legwork in up front to make this happen.

Basically, what I want to do today is give you the floor and let you take us through everything that you’ve found to be important in vision-language models. I’ll certainly have my questions along the way, but I’m excited for this.

This would have been a lot easier to compile if the field had stayed still for 5 minutes, the leaderboards hadn’t jiggled around every day, and new papers hadn’t come out every week, making me think, “We should probably include this.” But we’re at a checkpoint in time, so it’s worth saying that we’re recording on December 20, 2024. We’re still not at the end of OpenAI’s 12 Days of Christmas, so something may change tomorrow. A new model may be released tomorrow.

This is a point-in-time view of vision-language models. I suppose it’s a deep dive if you’re coming from the perspective of someone who’s interested in AI but not super familiar with vision-language models. But if we’re talking about vision-language models specifically, it’s definitely not a deep, deep dive into the research, because the field is huge and there’s so much going on. Some of it is very complicated, and there’s a lot we could cover.

What I thought we could do is stick to a few things. First, let’s look at some of the most important architectures and trends in research. We’ll illustrate these through some of the most notable models from the last couple of years—models that anyone working or building in this space is quite likely to encounter.

Then we’ll talk a little about the key data sets and benchmarks. We’ll explore one benchmark in particular in more depth because it’s really interesting. Finally, we’ll talk about recent attempts at what we call true multimodality.

A vision-language model is really reading audio—sorry, images—and text inputs and then reasoning about them. True multimodality would involve generating images as well. We’ll come to that at the end. Then we’ll finish by taking an “as of today” snapshot of what’s best in class across some of the key benchmarks, where to get those models, and what you can do with them.

That sounds good. I’m already taking away that you’re not classifying me as a truly multimodal entity, insofar as I can’t produce image outputs. Talk about the bar rising quickly. I think I’m already outclassed by what you’re calling the true multimodal models.

You mean you can’t draw?

Not very well. Not well enough that I would see an API anytime soon, that’s for sure.

In that case, I’m like you. I can just about doodle. But to start off, like all good research overviews, we should have a motivation section. Why do we care?

Obviously, there are lots of interesting use cases for VLMs. It was interesting recently when you had the team from Google talking about the new Gemini APIs. One of the things they said was that loads of people are building with large language models, but relatively few are building with vision-language models right now. They think that’s going to be a growth area next year.

There are loads of use cases. The obvious ones include medical assistance—being able to look at image modalities as well as a patient’s history and then say things about the patient that might be useful to the clinician.

Other use cases include content filtering, knowing what is in an image and text—for example, if you were looking at a social media platform and trying to screen out images or content—and indexing large quantities of archival material or product catalogs where there are both visual and textual components that you want to understand.

I’ve also seen applications in insurance where people have photos of cars, along with a description of what is supposed to have happened to the car. The question is whether you can actually see the damage in the image and whether it reflects what the person is reporting.

There are various use cases, but I think there are 2 other reasons we might be interested in vision-language models. One is that in building VLMs, you’re learning to integrate 2 modalities. They start off very separate, and somehow you’re going to reason over both of them. If you can find the right recipes for doing this, then in the future you can think about integrating audio, touch, light, radar, and other modalities.

If you think about robotics in the future, just consider the number of different sensory modalities a robot needs to cook a meal. It has to handle everything and see everything. You can think about VLMs as the first step toward learning how to do this, so that we can integrate many more modalities in the future.

The second reason is more philosophical. Is multimodal understanding important on the path toward AGI? It’s not entirely clear that it is, but some people argue that it is. One reason to research these things is to see whether integrating information from different modalities produces another kind of transformational leap in a system’s ability to understand the world and reason about it.

I would say “in inverted commas,” similarly to the way we do. We know they don’t do things the same way we do.

There you go. The argument against multimodal understanding being important would be that frontier language models show lots of evidence of high-level abstraction, world models, and sophisticated reasoning. There’s no obvious ceiling in performance as of today. Maybe grounded, multimodal understanding of the world isn’t that important for achieving AGI.

But I’d be very surprised if we ended up there. Imagine yourself unable to see. It would certainly be a major hurdle to have to get over. My guess is that we’ll never really answer the philosophical question of whether we could have built an AGI that isn’t multimodal.

There has been a lot of progress, and if nothing else, multimodality seems like the path of least resistance. Multimodality is clearly going to work. The details remain to be unpacked, but maybe the philosophical crowd will continue to say that we might have been able to do it without multimodality, or that it would have been impossible. In the end, though, this is going to be the norm, and these things are probably going to walk among us sooner rather than later.

There’s always “sooner rather than later” in this world.

Before we dive into the first vision-language model we’ll cover, there are 2 important prefaces. First, we should talk about Vision Transformers for a moment. Then we should talk about the CLIP model from OpenAI. Both of these are going to crop up again and again, so let’s refresh our memories about what they are before diving into VLMs themselves.

I’m going to assume we’re all familiar with the Transformer architecture for language models, especially the decoder architecture. The canonical paper here is called “An Image Is Worth 16x16 Words,” which came from Google about 4 years ago, in 2020.

Previously, most vision models had been based on convolutional neural networks. They were basically stacking convolutional filters to extract increasingly global features from images. The Google team asked whether they could use the Transformer recipe to build something that understands images.

The recipe is quite straightforward. You take an image and divide it into nonoverlapping patches. You then linearize the patches, using a linear embedding that converts them all into tokens. Now you have a sequence of visual tokens.

Through a learned embedding, you feed these patches one by one into a Transformer encoder and use full attention across them. Every little image patch can pay attention to every other image patch in the image. This is similar to how a model like BERT is trained.

You prepend a classification token to the sequence, and the training objective is to classify the image into 1 of a large number of categories. You take the classification vector at the end and use it to determine whether you got the classification right.

It’s a very simple recipe. The key finding was that if you make these Vision Transformers big enough, the Transformer architecture beats the convolutional neural networks of the day. That makes it a useful building block.

There are a few things to take away from the design of the Vision Transformer, also called a ViT. I’ll probably use “ViT” throughout. First, note that the image resolution is fixed by design. In the original Vision Transformer, it was 224x224 pixels, so everything had to be that size when it was fed in. That gives us a fixed number of patches.

For the original training, they inserted the classification token. When we get to vision-language models, normal practice is to take the entire sequence of hidden states from the Transformer and use that as the encoded image. We don’t just take the classification vector; we take everything.

That means you can get quite a lot of vision tokens out of the model. If you start with a 224x224 image and your patches are 16x16, then the back-of-the-envelope math says you’ll get 196 visual tokens at the end, which can be quite a lot.

The third thing is a naming convention. You’ll hear Vision Transformers described as ViT-H/16, for example. The “ViT” tells you it’s a Vision Transformer. The “H” stands for “huge,” which means roughly 600 million parameters, and the 16 tells us the patch size. We’re patching the image into 16x16 pieces. If I refer to ViT-G/16 later, that’s a giant model, even bigger than huge.

I want to make sure I understand the contrast between the image attention pattern and the language-model attention pattern we’re most familiar with, which is a look-back-only pattern.

In language, attention generally unfolds token by token. In an image context, the attention is all-to-all. There’s no sense of ordering. The image is a snapshot of a scene in time, and everything is on par with everything else as it’s processed.

The other thing I want to dig into is tokenization. In language, we have tokenizers that try to figure out the optimal way to break text up. Those typically aren’t part of end-to-end training. There’s this separate, bolted-on system that people have tried to get rid of for a long time.

You can go to OpenAI’s tokenizer, paste in text, and it will chunk it into pieces and color-code them. It has a vocabulary of up to roughly 100,000 different pieces of text that text is broken into before it’s translated into numerical form and processed.

The translation from a token to a vector representation is fixed at runtime. There are as many possible input vectors as there are tokens. People often refer to this as one-hot encoding.

How is that different for images? There isn’t a fixed vocabulary size of possible visual tokens, right?

That’s correct. We’re going to use the term “tokens” quite loosely. As you correctly said, text tokens can be mapped back to text through a codebook. You literally have roughly 100,000 codes, and you look one up to get the byte pair or whatever it is at the end.

The same isn’t true of visual tokens. They exist on a continuum. To go from a little patch—which is really a matrix with a few dimensions and some channels—you simply pass that through a matrix that generates the vector you want to put into the Transformer. That transformation is learnable.

The important thing is that the tokens come out on a continuum. They don’t need to be quantized into a codebook at this point. There’s nothing in the Transformer architecture that says tokens have to be quantized. You can still run the attention mechanism even if the tokens exist on a continuum.

Because we’re training the Vision Transformer with a classification objective, we don’t have to decode anything at the end. I’ll save my next question for when we get a little deeper into the journey.

The last thing worth reflecting on is how small the images are that are being processed. I’ve done a little bit of this, although not recently, because these days we have foundation models where I can throw basically anything into them.

With earlier generations of models, you had to preprocess the image. It was your responsibility as a user to take the image and resize it into the required dimensions that the model could handle. Everything was smaller back then, compute resources were more limited, and the results weren’t so spectacular.

Even until fairly recently, 224x224 was common. With OpenAI’s low-resolution mode, it’s remarkable how much performance can come out of these very small images. They can be dramatically shrunk and distorted because the original image might not even be square, but you just make it square anyway.

It’s amazing how well that works.

Without wanting to spoil the big reveal, the leading visual-language models aren’t compressing everything to 224x224 and using that as the image input. There are much more sophisticated things going on, and we’ll see how they work in a bit.

That was the Vision Transformer. The other thing we should introduce is the CLIP model from OpenAI. The paper came out in 2021, and it’s called “Learning Transferable Visual Models From Natural Language Supervision.” CLIP stands for Contrastive Language–Image Pre-training. It’s a canonical model in the field and a nice introduction to how image and text encodings are aligned.

You start with a vision encoder, which could be a Vision Transformer, and a text encoder, which could be an encoder-only Transformer. You have a large data set of images with captions scraped from the web.

The process is to jointly train both encoders so they generate embedding vectors for text and images. If you take an image and its caption, the 2 vectors should have high cosine similarity. If you take an image and a random caption, they should have low similarity.

You pass the image through the Vision Transformer and the text through the text Transformer. You then apply a linear projection to make them the same dimensionality and use a contrastive loss function.

Suppose you have a batch with N image-caption pairs. You know that within that batch you have N true pairs and N²−N bad pairs. You set up the loss function to penalize dissimilarity between the true pairs and similarity between the nonpairs.

That’s a contrastive loss function. It brings supposedly equivalent things close together and pushes nonequivalent representations apart.

Once you’ve trained CLIP, you can use it for things like image search. You can also take the trained Vision Transformer out and use it downstream. You’ve already trained it to embed things similarly to a language model, so putting it into a vision-language model should be a small step away from aligning it with whatever language model you eventually use.

I remember using CLIP in the early days of Waymark, when we were creating videos for small businesses. We had reached the point where we could get fine-tuned GPT-3 to write a somewhat decent script. Then we had the challenge of taking that narrative and choosing images from a collection we had found on the web.

At the time, most of this was done on finite, pre-established data sets. You had ImageNet, a canonical set of images with a fixed set of classifications, and the game was to develop an architecture that performed better on that standard data set.

That didn’t do much for us in an application like Waymark. An image of a particular business might not be represented by any of the classes in ImageNet’s set of 1,000 classes.

As far as I know, CLIP was the first moment when people said, “Let’s stop competing on these standard data sets. What people really want is to understand anything they might be looking at.” The web-scale data was available, and enough of the images had captions.

There was a tremendous amount of noise in the original data set. People caption images in all sorts of ways: with jokes, straightforward descriptions, or lines from poems. We found that CLIP was pretty good if you asked, “Which image from this set most closely matches this query?”

You would pass the text query through the text side, pass all the images through the vision side, represent each as a vector, calculate the cosine similarity, and sort by that. But you saw all sorts of artifacts.

For example, if you were making a video for a pizza restaurant and used “pizza” as the text query, an image with the word “pizza” in it could rise to the top, even if it wasn’t the best visual representation of pizza. That was a reflection of the extremely noisy web-scale data. At the time, there wasn’t a great way to clean it, so the approach was to throw as much data into the model as possible and hope that some signal emerged.

It did. But CLIP’s aesthetic understanding was basically nonexistent. We wanted to distinguish between a beautiful picture of something and an unattractive picture of it, because small businesses have a wide range of image quality. Sometimes you get user-generated content posted to Facebook, and sometimes you get professional photography. The difference matters a lot.

A business doesn’t want to use an ugly image in its marketing, but there was no real aesthetic signal in CLIP. It captured content, not quality.

It’s crazy that wasn’t all that long ago—3 or 4 years. Now we’re going to jump to 2 years ago, and everything else we talk about will be from the last 2 years.

One of the stories that unfolds as we cover a few of the VLMs is an increasing obsession with filtering data for quality, both in training and in subsequent fine-tuning stages. That’s an effort to get rid of the noisy-data problem, which seems to hurt VLMs in particular.

The next model we’ll discuss is one that I’ve heard described as the GPT-3 moment for vision-language models: DeepMind’s Flamingo. It contains a number of interesting innovations, so it’s worth covering in more depth. It’s also the first example we’ll see of how VLMs are constructed.

The basic pattern we’ll see across all of these models is that you encode the 2 modalities—text and images—separately. You use a text tokenizer and an image encoder, normally a Vision Transformer. You then select a language model, which is called the backbone. The backbone is what reasons over both the text and the images.

That leaves the question of how to connect the 2 modalities. For Flamingo, DeepMind looked at both a Vision Transformer and a convolutional neural network, although everything I’ve seen since has used a Vision Transformer. They used a Chinchilla language model as the backbone.

To connect the 2 things, they froze the language model and introduced cross-attention layers between, I think, every 4th Transformer block in the Chinchilla language model. The cross-attention layers look up information from the Vision Transformer.

There are immediately a couple of challenges. If the input contains more than 1 image, encoding those images produces a variable number of image tokens. With cross-attention, the keys and values are coming from the image rather than the text encoding, and the dimensionality of the cross-attention mechanism is fixed. So we need a fixed number of visual tokens, regardless of whether there are 2, 3, or 4 images, or images of different sizes.

The second challenge is that we can get a lot of visual tokens. If we want to train efficiently, can we reduce the number of visual tokens we actually attend to?

Cross-attention layers don’t need to be particularly deep. The DeepMind team solved this with what I think is a very smart mechanism called the Perceiver Resampler.

The Perceiver Resampler is a separate model that looks at the visual tokens and samples the most important visual information encoded by the Vision Transformer. But if the Vision Transformer produces a very long sequence of visual tokens, we don’t want the Perceiver Resampler to compute an all-to-all attention matrix over that entire sequence, because it could be enormous.

In a normal attention mechanism, you have queries, keys, and values, and they all come from the context you’ve read. The computational explosion comes from multiplying the queries by the keys, which creates an all-to-all attention matrix whose size is the sequence length squared.

The Perceiver approach says that instead of generating queries based on the context we’ve just read, we can use a small number of fixed queries. By “fixed,” I mean learnable: they’re latent vectors learned during training.

For Flamingo, they selected 64 query vectors. In the Perceiver Resampler, you look at all of the visual tokens that have come in, but the query has a size of 64. The query-key matrix is now sequence length times 64, which is much smaller.

At the end of the attention calculation, you have something that is 64 times the hidden dimension. If the hidden dimension is 768, for example, you get something very small. The Perceiver Resampler is essentially this module, applied in a number of stages.

The important thing is that we now know the size of the visual tokens at the end. It will always be 64x768, for example. That means we can define a cross-attention layer that always reads visual tokens of that size.

This happens after the initial Vision Transformer layers, right? You still have all-to-all attention happening on the vision side.

Yes. We still have all-to-all attention in the Vision Transformer. It’s a 2-stage process: the image is processed by the Vision Transformer, and then the Perceiver Resampler standardizes the output.

I remember the Chinchilla diagram. Even more than that, I remember the bowl of yarn soup that I carried around and showed people at parties in the spring and summer of 2022. I would say, “Look at this. This exists now.”

You have 2 separate components, similar to CLIP. Images are processed through 1 main model, and text is processed through another. These are often completely frozen when you try to figure out how to fuse them.

There are 2 parts to the fusion here. One is cross-attention, and the second is making life easier for the cross-attention mechanism and the main language model by standardizing the sequence length for the image.

Toward the end of the image processing, after full attention has been applied, you have an adapter that says, “Regardless of how big the image was or how many tokens it produced, the output from this step will always be 64 tokens.” Then the cross-attention and everything downstream can stay simpler.

I think this is a striking example of how malleable all these latent spaces are. You see one space bridged to another, reformed, or reshaped. Even the Python libraries that let you reshape matrices can give you a little intuition for this. Matrix reshaping doesn’t seem semantic, but when you scale it up to these systems, a similar thing starts to happen.

You have all-to-all attention, but what you really need is a fixed-length sequence. So you train something that always outputs a fixed-length sequence and hope it works. Not everything works, but almost everything of this sort seems to work. That’s a striking reality.

I also remember BLIP-2, which really brought this home for me because of how few new parameters had to be trained. Do you have those numbers?

A team from Hugging Face did a comparison, which we’ll discuss in more detail later. They said that in a 7-billion-parameter language-model class, adding cross-attention layers means that about 25% of the parameters need to be trained. If you use a simple projection instead, it’s about 10%.

So cross-attention introduces more parameters, but it’s still much smaller than retraining the entire language model. One benefit is that you can freeze the Vision Transformer, freeze the language model, and train only the cross-attention parameters and the Perceiver Resampler.

I’m amazed that the Perceiver Resampler works. It feels like tipping an image into a blender, pressing a button, and somehow retaining the important features when training is finished.

It feels like you’re accessing parts of the text embedding space that text itself can never reach. The language model was trained on a vocabulary of perhaps 100,000 tokens, but an image projection into the text embedding space isn’t bound by those tokens.

The space of possible inputs is much larger than the space the vocabulary routinely accesses. The image projection can go directly into that space, and the language model still handles it normally. It’s amazing that this works.

I don’t like to overdo the analogy, but I’m trying to imagine an equivalent challenge for a human. We’re natively multimodal, but putting something into word space that isn’t words feels a little like putting something directly into thought space.

These were leading indicators for me of how much was going to be possible. If a language model could do that without breaking, then we had to expect a lot more “Frankensteining” across modalities.

To train Flamingo, you switch on the next-token-prediction objective and train the newly initialized layers in the language model. When an image appears in the text prompt, the model is prompted to look up the outputs from the Perceiver Resampler.

These aren’t tokens that receive a score during decoding. Instead, the image acts as a signal that says, “I need to look up something in the Perceiver Resampler.” That gives the language model enough context to determine what the next text token should be.

In terms of the training data, the team used images paired with alt text, as in CLIP. But that wasn’t enough. One of Flamingo’s contributions was recognizing the importance of interleaved data.

Interleaved data is scraped from web documents, such as HTML pages, where images and text appear together. When scraping the website, you can use the document object model to determine roughly what order the images and text are supposed to appear in.

You can then pass that interleaved data through the model. When you reach an image, you look it up and continue generating text. The Flamingo team found that interleaved data was extremely important for maintaining downstream performance metrics. That was one of the major findings of the paper.

They evaluated the model on visual question answering, OCR, and captioning. Flamingo worked quite well in a few-shot setting. Training was computationally efficient because they didn’t need to modify the language model beyond the newly initialized parameters.

The output was competitive with much more focused, task-specific models. It could perform several tasks while remaining competitive with models trained specifically for each one. That’s why it was such a foundational model for VLMs.

Flamingo was never released publicly, was it? I don’t remember ever having a chance to use it.

I don’t think it was. That was before Google was dancing in the public arena. Most of what we’ll cover today was released publicly.

One of the things that makes large language models so good is instruction tuning. That has been one of the big unlocks. The Flamingo recipe didn’t have a clearly defined instruction-tuning step, partly because it was difficult to create an instruction-tuning data set. Those data sets didn’t really exist at the time.

The next model we’ll look at is LLaVA, which stands for Large Language and Vision Assistant. This is a 2023 model, so we’ve jumped another year. The original LLaVA came from a team involving people from Microsoft and academic institutions. It was the first in a long series of LLaVA models based on the same basic recipe.

The big innovation was instruction tuning and how they built the instruction-tuning data set. They started with the observation that generative vision-language models of the time could follow only a limited range of user instructions. They could caption images and answer basic questions, but they couldn’t perform the wide range of tasks that an instruction-tuned language model could perform.

The data sets used for training included interleaved text and images from web documents, as well as large captioning data sets. Relatively few examples contained task-oriented labels for images. That was the missing component.

The LLaVA architecture also differs from Flamingo. Flamingo is a cross-attention model because it introduces new cross-attention layers. LLaVA chose a simpler approach, derived from the Salesforce team’s BLIP-2 model. We can call it an autoregressive architecture rather than a cross-attention architecture.

The idea is that you take vision tokens processed by a Vision Transformer—preferably one from CLIP, because it has already been aligned to a language model—and train a simple projection matrix. You inject those projected vision tokens directly into the decoder stream.

There’s no cross-attention mechanism. You have a simple projection matrix, and then you mix the image tokens into the sequence. In the original LLaVA model, they prepended all the visual tokens to the text tokens going into the language model backbone. The training sequence would be a set of vision tokens followed by text, with each token attending to everything to its left.

That’s the autoregressive architecture. It seems simpler than cross-attention, but it has some downsides. If you generate a long sequence of visual tokens, you have to unroll all of them in the decoder, so they all become part of the decoder’s attention mechanism.

Also, if you use a simple projection matrix, you have far fewer parameters than you would have with a large cross-attention mechanism. You have to learn all the alignment using those parameters. If that isn’t enough and you want to improve the model, you have to unfreeze the language-model backbone and modify its attention mechanism.

As we know, fine-tuning the language model and its attention mechanism can lead to catastrophic forgetting of tasks the language model was originally fine-tuned for. That’s one downside of using the autoregressive architecture with self-attention.

The BLIP-2 authors were on one of my first podcast episodes. I remember the moment when I understood that they were somehow converting images into the text embedding space and then treating the images as if they were text.

From that point onward, the rest of the model didn’t know it had received anything other than text. It was frozen and could only handle text, but someone had figured out how to represent images in the text embedding space.

That was one of the leading indicators for me of how much was going to be possible. I saw the same architecture working across modalities, and then I saw bridges between those spaces. If you can take 2 frozen things and connect them in a couple of different ways, and they all seem to work, then we’re heading toward a world where everything is integrated.

The LLaVA team used a very smart method to create the instruction-tuning data. They started with images from the COCO data set, which Microsoft produced around 2014. COCO contains roughly 200,000 images, descriptions, and bounding boxes describing what appears in different areas of each image.

The data includes text such as “in this region is this object,” along with the bounding box for the object. It was originally intended to teach visual grounding.

The LLaVA team used a strong language model—GPT-4, I believe—and carefully designed few-shot prompting templates. They asked GPT-4 to generate a conversation between a questioner and a vision assistant, framed as though the assistant could see the image, even though GPT-4 itself couldn’t see the image.

Suppose the image description says that a group of people is standing around a vehicle, with luggage on the floor. The bounding boxes identify a suitcase, a bicycle, and a person. GPT-4 can generate a question such as, “What type of vehicle is in the image?” That’s easy because the description tells it what kind of vehicle is present.

It can also generate more detailed questions, such as, “What is the thing to the left of the car?” The bounding boxes tell us that the thing to the left is a bicycle. It can generate reasoning questions as well, such as, “What challenges do the people around the car face?” If people are loading luggage into the car, that’s something a human might infer from the image.

The smart part was generating these conversational dialogues using GPT-4. The team got both the questions and the model answers, because GPT-4 knew what the answer should be from the descriptions and bounding boxes.

When they built LLaVA, they froze the Vision Transformer and updated the language model and projection matrix. They pretrained on 600,000 image-caption pairs, using a simple form of instruction tuning. Then they used the more sophisticated instruction-tuning data set for fine-tuning, including multiround conversations, questions about specific regions of images, and reasoning questions.

They generated roughly 150,000 examples using this process. When they evaluated the model, it outperformed other VLMs of the time on complex image-reasoning tasks and was slightly better on conversational tasks. The work showed that investing in the instruction-tuning data set could produce a much smarter language model capable of a broader range of downstream tasks.

The same general recipe has been followed in subsequent LLaVA models, although the models have become much more sophisticated. The latest one, LLaVA-OneVision, from ByteDance, ranks very well on the MMMU leaderboard. The recipe is still strong a year and a half later.

This gives you a sense of why the leading developers think they can get around naturally occurring data walls. These results were achieved with largely synthetic data, at least when it comes to the final step of making the model behave as a useful image assistant.

In the summer of 2022, I was fine-tuning the text-only Davinci models. OpenAI never released the fine-tuning capability for one of the later versions, and GPT-4 was being developed, although I didn’t know that at the time.

We had early access to a model and were using it to process images. One challenge was taking a video and answering questions such as, “What text is on the screen?” and “How long is that text on the screen?”

You could take a still frame and use OCR, but then a 30-second video might contain 900 frames. What frequency should you use to take stills? How do you OCR all of them and then have the language model process the results? It was a complicated problem.

I could have hired human annotators to create a data set and trained on it, but I found myself using a recipe that still seems powerful today. I would do 10 examples myself, fine-tune on those, have the language model do the next 100, correct the errors, and fine-tune again.

That let me bootstrap a capability that humans weren’t particularly good at. It was unfamiliar data and not the sort of thing we’re evolved to handle. You could do it if you forced yourself to sit down and work through it, but it would have been difficult to hire people to do it with consistent quality.

That was a transformative moment for me. I could bootstrap my way into a fundamentally new capability much faster and more affordably than by hiring human annotators. If I could do that in a couple of days, then many other people were going to do the same thing across a wide range of tasks.

Using AI models to generate synthetic data for the next generation is now common in vision-language models. For example, you can generate synthetic OCR data to train models to read corrupted images. I’ve also seen people generate LaTeX documents. You generate the LaTeX and know the text you started with, so you have a paired example.

Growing data sets is difficult, particularly in vision-language models, because you can’t deal with just 1 modality. You need correspondence between the visual and language components, and that correspondence needs to be good.

The challenge is to create much larger, high-quality pretraining data sets where you know that the visual information and language are properly aligned. Many teams spend their time thinking about how to create high-quality data at scale.

The LLaVA recipe is a creative way to build a sophisticated instruction-tuning data set. The later LLaVA models have followed that approach, and it has a dramatic effect on the model’s ability to act as an assistant and perform tasks that earlier VLMs could not do.

This is the point where the models start to work well enough that an application developer can stop wrangling all the idiosyncrasies. In the next several models, we start to hit the point where things are simply beginning to work.

The next benchmark we should discuss is MMMU, which stands for Massive Multi-discipline Multimodal Understanding. It’s a mouthful, but the easiest way to think about it is as the multimodal version of MMLU.

It’s probably the most interesting and relevant benchmark for understanding how smart a VLM is and how much reasoning it can do. It’s explicitly designed to measure 3 skills: perception, or what the model can see; knowledge, or what it knows about what the image is showing; and reasoning, or what it can infer from the image.

The benchmark was compiled by students from different disciplines and subjects. They drew questions from online sources, textbooks, and lecture materials. Each question was required to involve expert-level understanding of the relevant domain.

They built roughly 11,000 questions across about 30 subjects, including history, medicine, electronics, market research, and music. Some questions require mathematical reasoning.

For example, a question might show harmonic intervals in musical notation and ask which of 4 musical scores is constructed incorrectly. Every question has 4 answer options, so the model selects A, B, C, or D.

When MMMU was released in November 2023, GPT-4V was the top-scoring model, with about 55%. Random guessing would produce 25%. The top open-source model was one of the LLaVA models, at about 34%.

Since then, o1 has moved to the top of the leaderboard, with roughly 78%. It’s 8 points clear of the runner-up. The benchmark hasn’t been completely crushed, but a lot has happened in 18 months.

It’s worth noting that the leaderboard may change quickly. The current top score is from o1-preview, and we don’t yet have scores for every newer model, including Gemini 2.0 and the latest Claude models.

When the MMMU team released the benchmark, they also asked text-only GPT-4 to answer the questions. They extracted text from the images using OCR or had a LLaVA model caption the images, then gave the text-only model the caption and the question.

GPT-4 scored about 34%, which is above random guessing and highlights the importance of reasoning. The remaining improvement up to the current scores reflects the model’s ability to interpret visual tokens and reason over them.

From now on, we’ll use MMMU as a reference point for the models we discuss.

The next topic is pretraining. The model I want to use to examine pretraining recipes is the Qwen-VL series from Alibaba. There are 2 models in the series: Qwen-VL and Qwen2-VL, from 2023 and 2024.

They use a self-attention, autoregressive architecture, with Qwen’s language model as the backbone and a Vision Transformer as the encoder. To connect the 2, they use a single cross-attention layer rather than adding many cross-attention layers to the language model.

The cross-attention layer has learnable queries. Its role is to compress an arbitrary number of visual tokens down to a smaller number before injecting them into the language-model decoder. In an autoregressive architecture, all those tokens would otherwise have to be unrolled into the decoder, so reducing their number makes training easier.

The Qwen-VL training recipe is divided into 3 stages. Instead of performing general pretraining in a single step, they use 2 pretraining stages followed by supervised fine-tuning.

In the first pretraining phase, they use image-caption data sets and interleaved data. They train the Vision Transformer and connector module but freeze the language model. Images are resized to 224x224, so the model can process many images efficiently.

In the second phase, they unfreeze the entire language model. Images can now be larger, at 448x448. Because the Vision Transformer is fixed, each image is split into 4 tiles. This produces many more visual tokens.

They also introduce multitask data, including synthetic OCR examples, visual-grounding data with bounding boxes and textual references, and visual question-answering and document question-answering data sets.

Some of these data sets contain tens or hundreds of thousands of images. They add them to the pretraining data so that the model sees a much broader range of tasks. They also include text-only data to preserve the language model’s text performance while modifying its attention mechanism.

The pattern is to start with a large, lower-quality data set, then move to a smaller, higher-quality data set, and finally use a much smaller supervised fine-tuning data set containing more carefully constructed prompts and image augmentations.

This recipe produced a strong VLM. Qwen2-VL sits just behind the leading models from OpenAI, Anthropic, and Google on MMMU, and above almost every other open model. The smaller models are open source, while the 72-billion-parameter version is available through an API and may eventually be released.

Let me summarize the training narrative. You start with a language model, then use high-volume, mixed-quality multimodal data for the first vision-language pretraining step.

The purpose is to bring the visual and language latent spaces onto the same page. You don’t initially want to update the language model, because you’re reasonably happy with it and you don’t want to risk losing capabilities. You mainly need to change the part that maps images into the language latent space.

Once that works, you open up training for the full model. Now everything can be trained end to end, with more diverse data. You continue to mix in standard text-only data so that the model doesn’t over-index on multimodal tasks.

Then you have the final supervised fine-tuning phase, where the data is much smaller and more expensive to generate.

That’s exactly the narrative. Multimodal data is scarcer, so you go up the quality scale as you reduce the size of the data set and unfreeze more parameters.

The final instruction-tuning data is highly proprietary and expensive. Without a frontier model such as GPT-4 to generate the data, it would be difficult to produce the volume and quality required.

As models begin to reason more explicitly, the situation is evolving. Google’s thinking models and some Chinese models expose reasoning traces, but the final chapter hasn’t been written. The data powering the last stage of training has to be extremely high quality, so it’s valuable and is rarely released.

I’m always struck that Chinese companies are not far behind. I’m interested in how you think about leaders versus fast followers. The West seems overconfident about how much of a lead it has relative to Chinese researchers.

Even in the papers we’ve discussed, there are many Chinese names on papers coming out of Western institutions. Chinese companies are also more open. If we’re talking about leaders in open models, the Chinese companies are clearly right there.

Overall, including proprietary models, OpenAI and Google often seem to get there first, but they aren’t open. Do you see the same division?

The next model is InternVL, from the OpenGVLab at Shanghai University. It’s probably the leading open-source model. The story behind the InternVL series is about scaling these models and training them efficiently.

The top open-source VLMs are hovering around the 70- to 80-billion-parameter range. We don’t know how large the top proprietary models are, but they’re probably larger.

The latest InternVL model was trained on roughly 120 billion tokens of mixed data. By comparison, Qwen2-VL was trained on about 1.5 trillion tokens. The OpenGVLab team has since released a data set containing roughly 1.5 trillion tokens, so they may be planning to scale quickly.

They already have the top open-source model, and it’s competitive with proprietary systems. This is a university team without the enormous funding available to frontier labs.

When you say “top vision-language model,” are you referring to MMMU?

Yes. I’m using MMMU as my benchmark of benchmarks. We should discuss other benchmarks later, but MMMU is the default benchmark for multimodal reasoning, much as MMLU is for language models.

The available multimodal data sets have grown dramatically. At the start of 2024, the largest publicly available image-caption data set was LAION, with roughly 5.8 billion captioned images filtered for quality.

In the last few months, much larger interleaved data sets have appeared. MINT-1T, released by a multi-contributor team with Salesforce involvement, contains roughly 1 trillion tokens and includes HTML documents, PDFs, and research papers.

OpenGVLab has also released OmniCorpus, an interleaved data set containing 2.2 billion documents sourced from Common Crawl, around 8 billion images, and 1.6 trillion text tokens.

These are much larger than anything previously available. For open-source researchers, the last few months have dramatically increased the size of the available pretraining data.

Scaling is a hell of a drug.

The original InternVL model focused on scaling the Vision Transformer. Everything before it had used relatively small Vision Transformers that were pretrained separately from the language model they would eventually connect to.

They trained a fresh Vision Transformer with 6 billion parameters, which is large for a vision model. ViT-H has around 600 million parameters, and the largest standard Vision Transformers are around 1.8 billion. This one was several times larger.

They performed contrastive pretraining using a frozen 7-billion-parameter Llama model as the decoder. They used image-text pairs, passed images through the Vision Transformer and text through the language model, pooled the resulting states, and used a contrastive objective to bring true pairs together and push false pairs apart.

This produces a high-quality, well-aligned Vision Transformer. The Llama model used for contrastive pretraining can then be discarded, and the Vision Transformer can be connected to a different language model. It works well because the Vision Transformer has been aligned with something much more similar to the language model it will eventually connect to.

The InternVL 1.5 model focused on image resolution. Images don’t naturally come in 224x224 or 448x448 squares, so the question is how to extract more information from higher-resolution images.

They developed a strategy called dynamic high resolution. An image with an arbitrary aspect ratio is segmented into fixed-size 448x448 tiles. The number of tiles depends on the aspect ratio and resolution of the image.

They choose among tiling configurations such as 4x4, 4x2, or 1x2, trying to match the configuration to the image’s natural resolution. Each tile is encoded separately, and a thumbnail of the entire image is concatenated to the end of the sequence.

This produces a large sequence of visual tokens, so they use a pixel-shuffle strategy to compress it. If a tensor has width, height, and depth, the number of tokens is determined by the width and height. Pixel shuffle changes the shape so that more information is stored in the depth dimension and fewer tokens are produced.

This allows images to be processed at their natural resolution by generating more tokens when necessary. In a self-attention or decoder-only architecture, the main cost is the extra computation required to unroll those tokens. You don’t need a Perceiver Resampler to force every input to the same size.

The latest InternVL model, InternVL 2.5, uses progressive scaling. They built several model classes: a 7-billion-parameter version, an intermediate one, and a 78-billion-parameter version.

They first align the Vision Transformer with the smaller language model. Once training reaches a plateau, they replace the smaller language model with the next larger one and continue training. Then they replace that with the largest language model and continue.

The result is more efficient than starting with the largest language model and Vision Transformer and aligning both from scratch. The smaller model converges early, and the Vision Transformer learns much of what it needs to learn before being connected to a larger model.

They report that InternVL 2.5 used about 120 billion tokens during pretraining, while Qwen2-VL required approximately 1.4 trillion tokens to reach a similar level of loss. The recipe is dramatically more efficient.

InternVL 2.5’s 78-billion-parameter version sits just behind o1 on MMMU. It beats GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro on that benchmark. It also performs very well on visual question answering and OCR benchmarks.

The data-scale result is interesting because it isn’t just saving compute by starting with a smaller model. It’s also using roughly one-twelfth as much data.

That reminds me of an episode we did on TinyStories, or tiny language models, where the models had tens of millions of parameters and were trained on short, child-oriented stories. The researchers looked at what the models learned and in what order.

The models first learned structural elements such as parts of speech, then began to distinguish nouns and other categories, and eventually developed what I remember as micro-reasoning skills. For example, if Sally doesn’t like soup and Jimmy offers her something, the model eventually learns that the answer probably shouldn’t be soup.

The smaller model was learning faster and was more sample-efficient. Do you have an intuition for why progressive scaling works?

Larger models have more capacity to learn complicated forms of reasoning, but they also have many more free parameters. With many free parameters, gradient descent takes longer to find useful basins that represent good capabilities.

If you start with a smaller language model, you align the Vision Transformer to a model with a limited level of complexity. Because there are fewer free parameters, you find a good solution sooner.

Once you’ve found that solution, you replace the language model with a larger one. There are more free parameters, but you’re starting from a reasonable place in the search space. You don’t have to search as widely.

That’s my intuition for why progressive scaling works.

There was a recent claim from one of the Chinese companies that it trained a roughly frontier-class model using single-digit percentages of the compute used by leading Western developers. Progressive scaling could be one way that happens.

The savings compound because you’re using fewer parameters and far fewer data points. If Qwen2-VL used 1.4 trillion tokens and InternVL’s progressive approach used 120 billion, that’s a very striking difference.

We’ve now seen several strategies: dynamic high resolution, scaling up the Vision Transformer, and progressive scaling of the language-model backbone.

Most of the models we’ve discussed since Flamingo use the autoregressive, self-attention architecture. That raises the question of whether the cross-attention architecture is dead. The answer is no.

When Meta released Llama 3.2 Vision, it used a cross-attention model. It used a ViT-H/14 Vision Transformer and introduced new cross-attention layers. Meta also modified the Vision Transformer in ways that aren’t fully detailed in the technical report.

They used large pretraining data sets, machine-generated OCR, improved safety filtering, deduplication, and quality filtering. They also used synthetic augmentation, including captions, tabular data, and LaTeX documents.

When training the Llama 3 Vision model, they added new cross-attention layers and froze the rest of the Llama 3 model. They trained only the cross-attention layers and the Vision Transformer during both pretraining and supervised fine-tuning. They also used direct preference optimization, or DPO, at the end.

Freezing the rest of Llama 3 preserves its language capabilities. If you use the autoregressive architecture, you have a projection matrix to align the visual tokens with the language model. Once that is trained, further improvement requires unfreezing the language model and risking degradation.

With Llama 3.2 Vision, Meta instead introduced enough new cross-attention parameters to learn the visual alignment while preserving the original language model. The 90-billion-parameter version is the second-place open-source model on MMMU, so both architectural recipes can produce competitive systems.

I can imagine this being partly a reflection of Meta’s organizational structure. At Meta, it felt like there were many different teams moving in different directions. That’s different from the perception of smaller, more focused teams at OpenAI or Anthropic.

Maybe not everybody wants or needs vision, so creating a modular architecture makes sense for the open-source community. I also wonder whether this is a legacy of having separate teams for different modalities before the architectures began to unify.

Is there any performance or practical difference between the architectures, or does either work as long as you do a good job?

There are some differences. Introducing cross-attention blocks adds more free parameters than using a simple projection matrix. With a large model such as Llama 3, there are enough newly introduced parameters to learn the visual-language alignment without modifying the original language model.

The decision may have been motivated by preserving Llama 3’s language-model performance. If you use the autoregressive architecture, once you’ve gotten the best alignment you can from the projection matrix, you have to unfreeze the language model to improve further. That creates a risk of degrading its capabilities.

There is some evidence that cross-attention models are weaker on OCR and other tasks requiring fine-grained understanding of small details. The Perceiver Resampler or another compression mechanism can shuffle visual tokens, which may affect OCR.

Some authors have also speculated that multimodal reasoning is better in the decoder-only, self-attention architecture. Putting the visual and text tokens into the same sequence may make it easier for the attention mechanism to reason over them.

Those are modest findings, but they may explain why most teams have moved toward the decoder-only architecture.

We’ve covered several important models and trends: the importance of interleaved data at scale; data augmentation; staging pretraining; increasing the resolution through tiling and thumbnails; progressive scaling; and the importance of task diversity in instruction tuning.

The main architectural alternatives are the self-attention or autoregressive approach, where visual tokens are injected into the decoder stream, and the cross-attention approach, where new cross-attention layers look up encoded visual tokens.

A team from Hugging Face tried to compare the 2 architectures systematically. They built a series of models called Idefics and used them to explore what makes VLMs work well.

They kept the decoder and Vision Transformer fixed while comparing the connection strategies. When they froze the language model and trained only the newly initialized parameters, the cross-attention architecture performed better. That’s not surprising, because it has more parameters to work with than a simple projection.

When they updated the language-model backbone, the autoregressive architecture performed much better. Full updates of the attention mechanism created training instabilities, but low-rank adapters worked. With those adapters, they could update the language model, and the autoregressive architecture performed better.

Increasing the size of either the Vision Transformer or language model improved performance. But for a fixed parameter count, increasing the language-model component produced more benefit than increasing the Vision Transformer.

A Perceiver Resampler speeds up training in both architectures but doesn’t necessarily improve final performance. It’s necessary in the cross-attention architecture to fix the size of the visual tokens, but it can also reduce the number of tokens that must be unrolled in the autoregressive architecture.

The Hugging Face team also removed interleaved image-text documents in an ablation and found a dramatic drop in performance. Interleaved data particularly benefits few-shot learning.

NVIDIA performed a similar comparison in a model called NVLM. They used Qwen’s language model as the backbone and the InternVL Vision Transformer as the vision encoder. They trained a decoder-only version and a cross-attention version.

The decoder-only version performed best on multimodal understanding, reasoning over images, and OCR. The cross-attention version was more efficient to train because it didn’t have to unroll the entire sequence of image tokens in the decoder.

They also created a hybrid architecture. High-resolution image tokens were presented through cross-attention, while the thumbnail was injected into the decoder stream. The decoder could reason over the overall image directly and use cross-attention to look up high-resolution tiles when needed.

This hybrid model was more efficient to train. The decoder-only version still performed better on OCR, but the gap was smaller. On chart understanding, the hybrid model was competitive. On the validation split of MMMU, the hybrid model actually beat both the decoder-only and cross-attention versions, although only by about a percentage point.

Is the hybrid model strictly better in any sense? If I’m OpenAI or DeepMind and I’m trying to build the best possible model, is there any result suggesting that the hybrid approach is the best, or is it mainly appealing because of its efficiency?

It’s too early to tell. The decoder-only NVLM model was better on chart understanding and OCR, but the hybrid model beat both alternatives on the MMMU validation split.

There aren’t many other hybrid models, so it’s unclear whether this is a generally superior architecture. But the result is interesting. On the current MMMU leaderboard, InternVL 2.5 is the top open-source model, Llama 3.2 Vision is just below it, and the NVLM hybrid architecture sits below those.

I’m always struck by how simple the Transformer is. My brain is a tangled mess of feedback loops, but these systems get as far as they do without any of that.

I have a strong prior that the simple Transformer will continue to be the best because it has been best for a while. But another part of me thinks that a more complicated architecture should eventually work better. Surely there are architectures that can exploit more structure, even if they’re harder to train.

The challenge is finding those architectures and making them computationally efficient. The current architectures have enormous advantages because they’re simple, scalable, and performant.

One interesting result from the NVIDIA paper concerns text-only performance. They evaluated several open-source VLMs on language benchmarks such as MMLU, math, and HumanEval. They found that the VLMs often performed worse on text-only benchmarks than their original language-model backbones.

That kind of degradation is expected if you fine-tune on a new task without preserving data for the old tasks. The LLaVA and InternVL models had some degradation. Llama 3 Vision did not, because Meta froze the language-model backbone.

NVIDIA created a high-quality text-only supervised fine-tuning data set and included it in the training mixture. The resulting NVLM models improved on all the language-only benchmarks relative to their original backbones.

The improvement was especially clear on mathematics. Their multimodal fine-tuning data contained many mathematical questions in image form, including geometry. They believe that introducing mathematical questions as images improved the model’s general mathematical reasoning, including on text-only questions.

Let me make sure I have that right. Text-only performance usually degrades if you fine-tune on image data without maintaining text-only data in the mix. NVIDIA maintained text-only data and saw an improvement on text-only math and coding benchmarks.

Exactly. The introduction of multimodal data can help, as long as you preserve the original capabilities with appropriate text-only data. It suggests that multimodal understanding is beneficial, even though it doesn’t prove that multimodality is necessary for AGI.

It makes sense. If you had never seen a drawing of a triangle and only encountered math problems as text, it would be strange to expect you to handle diagrammatic geometry as well as someone who had seen those diagrams.

I wonder how far this generalizes. We’re already adding video and audio. Could we train natural-language models on biological sequence data, such as DNA or amino-acid sequences?

I’ve been interested in models trained on sequence data because they can learn higher-order concepts from raw data that aren’t mediated by human language. When concepts emerge from protein or DNA sequences, it’s easier to see that something genuinely new may have been learned.

The global maximum seems like a model trained on everything: text, images, audio, weather data, biological sequences, and so on. It would have a robust, all-to-all understanding of the world.

The way we’ve described VLMs is as systems that learn to exploit relationships between modalities. There may be more efficiency to gain from better ways of transferring information between modalities.

A term we haven’t introduced is early fusion versus late fusion. This describes where in the information-processing architecture the modalities come together.

Everything we’ve discussed so far is late fusion. We have separate encoders—a Vision Transformer and a language transformer—and then align the 2 representations. Very early fusion would mean having a single component that encodes text and vision together from the beginning.

My guess is that the brain uses both early and late fusion, with multiple integration points for different modalities. AI may eventually use multiple fusion points as well. The NVLM hybrid model already has 2: a cross-modal cross-attention layer and an image thumbnail inserted into the decoder stream.

When people talk about early and late fusion, is that mainly a statement about the architecture, or does it also refer to the pretraining process?

The hybrid model still has separately pretrained modules, which are then connected at different points. Joint pretraining seems like another dimension. You could imagine a 2x2 matrix: separate versus joint pretraining, and early versus late architectural fusion.

There isn’t a settled definition. I’d think of early and late fusion as a continuum. Everything we’ve looked at so far is relatively late fusion, including CLIP, because the modalities are encoded separately and then aligned.

Very early fusion would mean having a single representation for text and vision right from the start. We’ll look at an example of that later.

That makes sense.

There are a few other important benchmarks. The first is VQA, or Visual Question Answering. It contains roughly 50,000 images, many from COCO, with open-ended questions generated by Amazon Mechanical Turk workers.

The questions are intended to require understanding of the visual content, language, and common sense. There are roughly a million questions overall, with multiple questions and answer options associated with each image.

For example, an image might show a woman with bananas hanging from her upper lip like a mustache. One question is, “What color are her eyes?” The model might lock onto the yellow in the middle of the image and answer “yellow,” even though that’s not correct.

Another question is, “What is the mustache made of?” That requires knowing where a mustache is on the face, recognizing its shape, and identifying which objects are serving that role in the image.

VQA is different from MMMU. MMMU is primarily about reasoning over images, while VQA is about understanding common objects, their relationships, and what is happening in an image.

There’s also DocVQA, which contains roughly 50,000 questions over about 12,000 images extracted from industry documents. These include scanned PDFs, charts, graphs, tables, invoices, business infographics, and handwritten notes.

The task is to isolate and report precise spans of text from the images that answer a question. For example, the question might be, “What is the number on this invoice?” followed by a scanned invoice.

DocVQA is important because processing scanned documents is a common use case for VLMs. If that’s what you care about, DocVQA is a benchmark worth examining.

The Hugging Face team bundled 50 fine-tuning data sets together in a collection called The Cauldron. It’s available on their platform and is probably the easiest way to acquire a good fine-tuning data set.

If I were starting a task that required a VLM and struggling with performance, I would look at The Cauldron for a similar task. I’d examine the image augmentations and prompt structure and ask whether I could augment my own images or restructure my prompts in the same way.

Whether you’re doing this at inference time or building your own supervised fine-tuning data set, The Cauldron is an informative place to start.

That’s a good tip.

Let’s linger on the BLINK benchmark because it’s fascinating.

BLINK was produced by an academic and Allen Institute for AI team earlier this year. It contains just under 4,000 multiple-choice questions across 14 common perceptual tasks.

The authors say that humans can solve these tasks in a blink, but they should be difficult for VLMs. Human performance is generally in the mid-90% range, while random guessing gives you just over a third because of the number of answer options.

The motivation is that many questions in MMMU are really reasoning questions. The authors describe them as dense-captioning tasks: can the model extract a rich description of what it sees in the image?

If you replaced the image with a rich description, a language model could answer many MMMU questions. The MMMU team itself showed that a text-only model could do substantially better than random guessing when given a caption.

The BLINK authors wanted to put more emphasis on classic visual perception. The large jump from GPT-4V to GPT-4o may be partly attributable to reasoning, but BLINK tries to measure perceptual skills more directly.

The best-solved task in BLINK is art style. At the time of recording, GPT-4o was the best model I could find, at around 83%, compared with human performance of roughly 95%.

The task might show a sketch as a prompt and 2 possible paintings, asking which one matches the visual style. A human can answer immediately.

Visual similarity is another relatively well-solved task. You might see a reference photograph and 2 other images and have to identify which is most similar. Humans score around 97%, and GPT-4 Turbo scored around 80%.

The worst-solved tasks are more interesting. One is an IQ-test category. You see a sequence of shapes and have to select the option that completes the pattern. Humans solve these about 80% of the time, while GPT-4 Turbo scored only about 32.7%, barely above the 25% random baseline.

This reminds me of ARC. I’m not sure whether that’s a lazy analogy, but I seem to solve these IQ tests in much the same way I solve ARC challenges.

I wonder whether I’m using a perceptual prior to guide a program search. That might explain why I can solve them in a blink while a language model struggles.

That’s a reasonable speculation. The point of BLINK is that humans solve these questions very quickly. There’s something about the perceptual features we naturally extract that lets us answer them without exhaustive search.

Humans have priors that constrain the search space. We can zoom in on the likely transformation or rotation quickly. A model may not understand the perceptual features in the same way and may need to perform something closer to an exhaustive search.

A reasoning model with enough compute might brute-force many of these problems. But that probably isn’t the most efficient way to solve them.

There’s another difference from ARC. In ARC, the input can be represented directly as arrays of values. In BLINK, the model has to process an image and extract the relevant structure first.

I’ve also noticed that even the best models can be weak at simply describing ARC images. I took screenshots from the ARC website and asked the model to describe them without solving the problem. It struggled with basic things such as counting squares and identifying the dimensions of the grid.

That suggests the issue is not just reasoning but the visual representation itself.

That’s right. The BLINK team pointed out that much more work has gone into language models than into the architecture of Vision Transformers.

One feature of convolutional neural networks is that they include inductive priors. Those priors are missing from Vision Transformers. At smaller model and data-set scales, convolutional networks can actually perform better. As scale increases, Vision Transformers overtake them.

Another possibility is that contrastive learning is not the optimal training objective for a Vision Transformer. Language models use a generative pretraining objective, which is extremely effective. Perhaps there is an equivalent generative recipe for vision.

That question brings us to another interesting paper.

A final poorly solved BLINK task is relative reflectance. The model sees an image with 2 marked points and must answer which point has the darker surface color or whether they’re about the same.

The example in the paper is a hotel bedroom. The headboard is cream-colored, and the wall behind it is white, but light from a window makes the pixels on the headboard appear brighter than the pixels on the wall.

A human compensates for the lighting and knows that the cream-colored headboard is darker than the white wall. Humans score around 95%, while the best VLM at the time, a LLaVA model, scored just under 40%.

That shows that models may not perform the same perceptual correction that humans do.

One other interesting observation is that GPT-4o has improved on some BLINK tasks but regressed on others. For example, GPT-4V scored around 60% on counting, while GPT-4o scored around 49%.

It isn’t clear why. It could be an artifact of distillation, fine-tuning, or some other training change.

These systems are strange. Every time a new model comes out, people discover some capability that has regressed. The major benchmark scores may stay stable, but the long tail of tasks is unpredictable.

Counting seems especially interesting because it feels simple. Why is it so difficult for a VLM to count objects?

There’s a paper called “The Effectiveness Assessment of Recent Large Vision-Language Models” that found several open-source models, including LLaVA 1.5, outperforming GPT-4V on counting tasks.

A DeepMind team also looked at the problem when trying to get diffusion models to generate the correct number of object instances. They searched commonly used image-caption data sets and found that captions specifying object counts are very scarce.

Even when captions contain numbers, there may not be enough examples to learn how to bind the number to the appropriate visual feature. The problem may simply be that counting is underrepresented in the pretraining data.

That suggests augmented, task-specific data could help. The recipe we’ve seen repeatedly is to create a data set specifically for the missing capability and add it to the training mixture.

Anthropic’s documentation also shows that careful prompting can improve counting. You can ask the model to decompose the image, enumerate the objects it sees, and reason through the count.

Reasoning can compensate for perceptual deficiencies, but the deficiencies are still there.

The Apple team explored whether contrastive learning was part of the problem. They asked whether multimodal pretraining of a Vision Transformer could use a generative objective instead.

They started with a vanilla Vision Transformer and a Transformer decoder, both trained from scratch on image-caption pairs. The captions were a mixture of web-scraped alt text and synthetic captions.

The data consisted of image-patch tokens followed by text tokens. Training used prefix attention. They randomly masked some of the visual tokens, and the decoder had to generate the remaining visual tokens followed by the text tokens.

The loss was calculated only over the nonprefixed tokens. For text, they used the standard cross-entropy loss. For image tokens, they used mean squared error between the generated visual token and the true token.

The result was a generative pretraining recipe for the Vision Transformer. They jointly trained the Vision Transformer and the decoder from scratch.

Afterward, they discarded the decoder and connected the new Vision Transformer to a Llama 3 model through a simple multilayer perceptron. They then trained the resulting VLM using the LLaVA supervised fine-tuning mixture.

This creates a clean ablation. You can compare the Vision Transformer trained with generative pretraining to one of the same size trained with contrastive learning.

The generatively pretrained model improved on every VLM benchmark they tested, with particularly large gains in captioning and visual question answering.

That suggests that the contrastive objective may be limiting the Vision Transformer. Switching to a generative pretraining objective—the same general recipe that works so well for language—also works well for vision.

Can we linger on the difference between contrastive and generative training?

With contrastive training, you encode an image and its caption. You have 2 vectors, and within a batch you identify the true image-caption pairs. You want the cosine similarity of each true pair to be high and the similarity of all nonmatching pairs to be low.

That objective shapes the embedding space so that images and their corresponding captions are close together.

With the generative objective, you prepend a sequence of visual tokens and mask some of them. The decoder generates the missing visual tokens followed by the text tokens. The Vision Transformer and decoder are trained together.

The model’s success at reconstructing the visual tokens is measured with mean squared error. Its success at producing text is measured with cross-entropy.

Because the visual tokens come first and the text follows, the model learns to attend from the text to the image. More of the learning pressure is placed on reconstructing the visual information, and the resulting Vision Transformer appears to be stronger.

The original Vision Transformer was trained as a classifier, right?

Yes. The original Vision Transformer was trained on ImageNet with a classification objective. It prepended a classification token and used a linear projection to predict which of a large number of categories the image belonged to.

All the VLMs we’ve discussed since then have generally used contrastively trained Vision Transformers, as in CLIP.

Would you call the Apple approach early fusion or late fusion? The paper shows a separate vision component and a separate decoder, but they’re pretrained together.

It’s a bit of both. Because there’s still a separate Vision Transformer that is eventually connected to a language model, you could call it late fusion. But the alignment occurs much earlier because the 2 components are trained together.

The terminology isn’t completely standardized.

The next frontier is multimodal generation.

Most of our discussion has focused on understanding images and, to some extent, video. We haven’t talked much about video, but many of these models can process video because a sequence of frames isn’t fundamentally different from a sequence of pictures.

There are video-specific benchmarks, such as Video-MME, that measure how well models handle sequences of frames. Whether you consider video a separate modality is partly a matter of terminology.

What we haven’t discussed is generating images. The simplest way for a VLM to generate an image is to have it generate a prompt and hand that prompt to a diffusion model. Gemini and GPT-4o initially used that approach with Imagen and DALL-E.

The original GPT-4o announcement suggested that GPT-4o is a true multimodal model capable of directly generating image outputs. The “o” stands for “omni,” meaning it can generate other modalities as well. That capability has not been released, reportedly because of safety and infrastructure concerns.

There has been work from Meta and FAIR on true multimodal models that can both input and generate images and text. The sequence includes CM3, a causal multimodal model of the internet; CM3Leon; and a separate model called Chameleon.

The model I want to focus on is Meta’s Transfusion. It may be the most promising recipe for multimodal generation and understanding.

Transfusion pretrains a single Transformer on an even mixture of image and text data, but it uses a different training objective for each modality.

Earlier models in the series quantized image tokens before feeding them into the Transformer. Text tokens are naturally quantized because they come from a codebook. Visual tokens, by contrast, usually exist on a continuum.

One way to quantize image tokens is through vector quantization. You pass an image through the encoder of a variational autoencoder to obtain latent vectors. You then learn a codebook, similar to the centroids in k-means clustering.

Each image token is quantized to the closest vector in the codebook. You decode the quantized representation through the variational autoencoder and compare the reconstructed image with the original. The codebook is learned until the reconstruction is good.

That gives you a codebook that can be used to quantize image tokens. Earlier multimodal models used this approach.

Transfusion discards image-token quantization. It passes the image through the encoder portion of a variational autoencoder, producing latent patches, and then through either a multilayer perceptron or a U-Net downsampling block. These latent vectors are inserted into the Transformer as continuous tokens.

Text tokens are handled in the usual way. The model is trained using next-token prediction, but the losses differ. For text regions, it uses a linear layer and cross-entropy loss. When it detects that it is generating image tokens, it passes them through the corresponding U-Net upsampling path and variational-autoencoder decoder and uses a diffusion loss.

The attention masking is important. Text uses causal attention: when decoding a text token, the model can attend to everything to its left. Image generation uses bidirectional attention within the image, so every patch can attend to every other patch in that image.

An image can attend to preceding text and previous images, but not to future text or future images. It’s a much more complicated setup and decoding regime.

Transfusion is essentially a latent diffusion model split in half, with a Transformer inserted in the middle. The text side is handled in the standard way, while the image side uses the diffusion architecture.

The results are impressive. The model produces images of similar quality to earlier approaches after roughly one-third as many FLOPs. On text-to-text tasks, it matches the training loss of earlier models after about half as many FLOPs.

It appears to be a more efficient recipe, although it is much more complicated to set up.

The same general approach was used by DeepSeek, and the paper suggests that the recipe could extend to other modalities, including audio. We’ll likely see much more exploration of hybrid architectures that both understand and generate multiple modalities.

I’m not sure why you would ever quantize the image tokens in the first place. If you do that, you can train a single decoder-only Transformer to generate both images and text, using a larger or separate codebook.

It seems simpler to use a common cross-entropy objective. Transfusion requires the autoencoder, the U-Net downsampling and upsampling, and the diffusion objective. It sounds much more computationally and architecturally complicated.

That’s fair. The earlier models did produce surprisingly good images despite using a relatively small codebook. In Chameleon, the codebook contained 8,192 tokens, and a 512x512 image was represented by 1,024 discrete tokens.

That corresponds to 16x16 patches. It’s surprising that 8,192 codebook entries can represent the possible variations in such patches, but the image outputs are remarkably natural.

Transfusion reported that it outperformed DALL-E 2 and Stable Diffusion XL on its image-generation evaluations, while its 7-billion-parameter model reached roughly LLaMA 1 performance on text-only tasks.

The image editing is particularly impressive. There was an earlier model called InstructPix2Pix that allowed you to provide an image and an instruction to edit it. I tried it on an ultrasound image of my son and asked it to make it look like a newborn baby instead of an ultrasound. It did something recognizable, but the quality and control were limited.

Transfusion looks much more useful for precise local edits. You can say, “Change the graffiti on the side of the truck into calligraphy,” and it preserves the scene while making the requested change. You can remove objects, replace objects, or change colors without turning the image into something unrelated.

That’s exactly the kind of capability we need for Waymark. Small businesses want realistic marketing images. They don’t want a generated image that looks nothing like the business when the customer arrives.

Purely generative models have had difficulty with control and grounding. Image prompting often adds noise or takes the image in a different direction. What we want is a precise local edit that preserves the integrity of the original.

Character and scene consistency have also been difficult. Transfusion suggests that a lot of these capabilities may emerge from a more unified multimodal architecture.

Let’s finish by looking at what the frontier labs offer and what is currently winning on the benchmarks. I compiled the results from several sources because they aren’t all in one place.

I looked at Grok 2 Beta, Claude 3.5 Sonnet New, Gemini 1.5 Pro, GPT-4o, and several open-source models. I couldn’t find complete results for every newer model.

For MMMU, o1 is clearly ahead of the others at roughly 78%. Claude 3.5 Sonnet New and InternVL 2.5 are around 70%. Gemini 1.5 Pro is a few points below them, while Grok 2 Beta is around 66%.

The Chinese open-source models are extremely competitive. InternVL 2.5 is roughly level with Claude 3.5 Sonnet New and ahead of GPT-4o on this benchmark. LLaVA-OneVision comes from ByteDance, and Qwen comes from Alibaba.

The parameter counts of the leading open-source models are around 70 to 80 billion, and they’ll probably become larger. The progress in training recipes and data sets explains much of the rapid improvement. New models tend to get better because researchers learn how to train them more effectively.

On DocVQA, most models score above 90%. Qwen2-VL is the best result I could find, at around 96.5%. It performs better than Grok 2, Gemini 1.5, and GPT-4o on that benchmark.

On BLINK, GPT-4o is around 63.2%, while InternVL 2.5 is slightly higher, at roughly 63.8%. Gemini 1.5 is around 61%, and Claude 3.5 Sonnet New is around 56.5%.

For smaller models, Gemini 2.0 Flash scores around 70.7% on MMMU, putting it well above the other models in its class. Grok 2 Mini is around 63.2%.

On DocVQA, the 8-billion-parameter InternVL 2.5 model scores around 95.1%, the same as its larger version. On BLINK, GPT-4o Mini scores around 51.9%, while InternVL 2.5’s smaller model scores around 54.8%.

The surprise in the smaller-model class is Microsoft’s Phi-3.5 Vision, a 4-billion-parameter model that scores around 58.3% on BLINK. It doesn’t perform as well on MMMU, but it does extremely well on BLINK.

The Phi technical report describes a pretraining data set of roughly half a trillion tokens, which is large for a 4-billion-parameter model. Its supervised fine-tuning data set contains about 33 billion tokens, including a substantial amount of in-house Microsoft data.

That’s an unusually large supervised fine-tuning data set, especially for a small model. Phi-3.5 Vision also used DPO. Only a few other models, including Llama 3 Vision, explicitly report a DPO stage.

I’ve seen people comment that Phi-3.5 Vision is very good at visual understanding, so it’s an interesting model to keep an eye on.

Looking ahead, I expect to see many more true multimodal models following the Transfusion recipe. Open-source VLMs will probably continue to grow in parameter count, especially now that progressive scaling of the language-model backbone has proven effective.

We’ll also see more experimentation with Vision Transformer pretraining and perhaps eventually replacement of the Vision Transformer itself. There will be continued development of fine-tuning data sets containing programmatic or human image augmentations.

We haven’t seen much systematic exploration of alignment and post-training for VLMs. DPO is beginning to appear, so I expect that area to receive more attention.

That’s a relatively short tour through the last 2 years of vision-language models.

The depth of research you put into making this possible is outstanding. I learned a lot from it, and I come away with a much better understanding of the different options, strengths, weaknesses, and even a few prompting techniques.

How does this relate to what you typically do? In the last couple of minutes, tell us about your normal work and the sort of stuff you do commercially.

A lot of this is relevant to some of the work I’m doing. VerAI is the small consultancy that my colleagues and I set up a couple of years ago. We do AI strategy work and prototyping and proof-of-concept work, with a structured way of running experiments cheaply and finding what’s easy and difficult. We also work in medicine and open-source intelligence, where much of the data we interpret is multimodal, so we need to understand how images correspond to claims and surrounding context.

It’s a useful reminder that the ability to answer a practical question quickly often comes from having spent a long time studying the surrounding space. Once you’ve done that work, you can pattern-match a new problem to the relevant architecture, data set, or training strategy.

Exactly. You need a prepared mind. I keep a searchable archive of papers, newsletters, Substacks, and other material, and when a new problem appears I try to match it to something I’ve already studied.

You need to be steeped in the field to recognize where a technique, model, or training method might apply.

I’m writing a fair bit at the moment about AI strategy, and actually I’ll be writing about various other things, and maybe about vision-language models as well.

Cool. Well, I’ll be sure to connect with you there and encourage the audience to do the same. This has been a fantastic walkthrough of vision-language models. I know a lot of work has gone into it, but if you want to tackle another topic like this, I would love to do it. I’ll say thank you for this one, and officially, Will Hardman from VerAI, thank you for being part of The Cognitive Revolution.

Thanks for having me, Nathan.