Teaching AI to See: A Technical Deep-Dive on Vision Language Models with Will Hardman of Veratai

Is multimodal understanding in an AI important on the path towards AGI? It's not entirely clear that it is, but some people argue that it is. So one reason that one might want to research these things is to see if by integrating information from different modalities, you obtain another kind of transformational leap in the ability of a system to understand the world and to reason about it. I would say, in inverted commas, similarly to the way we do. For open source researchers, like the last few months have really seen the arrival of these huge interleaved datasets, which has kind of really jumped the pre-training dataset size that's available. I'm kind of amazed that the Perceivable Resampler works because it feels to me just like tipping the image into a blender, pressing on, and then somehow when it's finished training, the important features are retained. I'm still there for you.

Hello, happy new year, and welcome back to The Cognitive Revolution.

Today, I'm excited to share an in-depth technical survey covering just about everything you need to know about vision language models, and by extension, how multimodality in AI systems currently tends to work in general. My guest, Will Hardman, is founder of AI advisory firm Veritai, and he's produced an exceptionally detailed overview of how VLMs have evolved. From early vision transformers to CLIP's pioneering alignment work to today's state-of-the-art architectures like InternVL and Llama 3V. We'll examine key architectural decisions like the choice and trade-offs between cross-attention and self-attention approaches, techniques for handling high-resolution images and documents, and how evaluation frameworks like MMMU and BLINK are revealing both the remarkable progress and the remaining limitations in these systems. Along the way, we dig deep into the technical innovations that have driven progress, from Flamingo's Perceiver Resampler, which reduces the number of visual tokens to a fixed dimensionality for efficient cross-attention, to InternVL's dynamic high-resolution strategy that segments images into 448x448 tiles while still maintaining global context. We also explore how different teams have approached instruction tuning, from LLaVA's synthetic data generation to the multi-stage pre-training approach pioneered by the Chinese research team behind QuenVL. Our hope is that this episode gives anyone who isn't already deep in the VLM literature a much better understanding of both how these models work and also how to apply them effectively in the context of application development. Will spent an estimated 40 hours preparing for this episode, and his detailed outline, which is available in the show notes, is probably the most comprehensive reference we've ever shared on this feed. While I have not worked personally with Will outside of the creation of this podcast, the technical depth and attention to detail that he demonstrated in what for him is an extracurricular project was truly outstanding. So if you're looking for AI advisory services and you want someone who truly understands the technology in depth on its own terms, I would definitely encourage you to check out Will and the team at Veritai. Looking ahead, I would love to do more of these in-depth technical surveys, but I really need partners to make them great. There are so many crucial areas that deserve this kind of treatment, and I just don't have time to go as far in-depth as I'd need to, to do them on my own. A few topic areas that are of particular interest to me right now include, first, recent advances in distributed training. These could democratize access to frontier model development, but also pose fundamental challenges to compute-based governance schemes. Next, what should we make of the recent progress from the Chinese AI ecosystem? Are they catching up by training on Western model outputs, or are they developing truly novel capabilities of their own? There's not a strong consensus here, but there's arguably no question more important for US policymakers as we enter 2025. I'm also really interested in biological inspirations for neural network architectures or any comparative analysis of human and artificial neural network characteristics. The episode that we did with AE Studio stands out as one of my favorites of 2024, and I would love to have a more comprehensive understanding of what we collectively know about this space. I'm similarly interested in the state of the art when it comes to using language models as judge or otherwise evaluating model performance on tasks where there's no single right answer. This is a problem that we face daily at Waymark and which seems likely to have important implications for how well reinforcement learning approaches will work and scale in hard-to-evaluate domains. Finally, for now, I would love to catch up on the latest advances in vector databases and RAG architectures. I've honestly been somewhat disillusioned with embedding-based RAG strategies recently, and I've been recommending Flash Everything as the default relevance filtering strategy for a while now. But I do wonder, what might I be missing? In any case, the success of our previous survey-style episodes, including our AI Revolution in Biology episodes with Amelie Schreiber and our Data, Data Everywhere: Enough for AGI episode with Nick Gannon, suggests that people find these detailed overviews to be a helpful way to catch up on important AI subfields. So if you have or are keen to develop deep expertise in an area that you think our audience would benefit from understanding better, please do reach out. I'm open-minded about possible topics and very interested to hear what you might propose. You can contact us as always via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. And of course, I'm more than happy to give you the chance to plug your product or services as part of your appearance on the show. Now, I hope you enjoy my conversation with Will Hardman, AI advisor at Veritai, about all aspects of vision language models.

Will Hardman, AI advisor at Veritai and AI scout on all things vision language models. Welcome to The Cognitive Revolution.

Thanks, Nathan. Great to be here.

Yeah, I'm excited about this. We've talked about this for a few months now, and you have put a real Herculean labor into a very deep dive into all of the techniques, datasets, different strategies, variations on making vision language models work. I think this is going to be a really interesting crash course and overview on all that. And I think it's something that I know that I want and need. And I think a lot of people will really benefit from getting the sort of fast-forwarded version of all the research you've done. So Thanks for putting all the legwork in upfront to make this happen. And basically what I want to do today is just kind of give you the floor, let you take us through everything that you have found to be important in vision language models. And I'll certainly have my questions along the way as we go. But yeah, I'm excited for this.

Cool. Yeah. So this would have been obviously a lot easier to compile if the field had stayed still for 5 minutes and the leaderboards hadn't jiggled around every day. And new papers haven't come out every week, making me think, hmm, we should probably include this. But we're kind of at a checkpoint in time. So it's worth saying we're recording on December 20th, 2024. We're still not at the end of the OpenAI 12 Days of Christmas. So something may change tomorrow. Paper may be released tomorrow. So this is a kind of a point in time view of vision language models. And I guess it's a deep dive. If you are coming from the perspective of someone who's interested in AI, but not super familiar with vision language models. But if we're talking about vision language models specifically, it's definitely not a deep, deep dive into the research because it's huge and there's so much going on and some of it's so complicated and there's so much we could cover. So what I thought we could do is just stick to like a few things. So firstly, let's look at some of the most important architectures and some of the trends in research, and we'll illustrate these through some of the most notable models from the the last couple of years. And these will be models that anyone who's working or building this space is quite likely to encounter. And then we'll talk a little bit, I mean, touch briefly on the key datasets and some of the benchmarks. And one benchmark in particular, we'll explore in a bit more depth because it's really interesting. And then I guess we'll talk a bit about recent attempts at what we call true multimodality. So a vision-language model is really reading images and text inputs and then reasoning about them. But true multimodality would be generating images as well. So we'll come on to that at the end. And then we'll finish up, I think, just by taking a, as of today, snapshot. What's best in class across some of the key benchmarks? And where do I go and get it? And what can I do with it? Does that sound good?

Sounds good. Yeah, I'm already taking away that you're not classifying me as a truly multimodal entity in as much as I can't produce image outputs. So the talk about the bar rising quickly, I think I'm already outclassed by what you're calling the true multimodal models.

You mean you can't draw?

Not very well. Not, not well enough that I would, I would not see an API anytime soon. That's for sure. Yeah.

So in that case, I'm like you. I mean, I can just about doodle. That's about it. Because let's just start off like all good. Research overviews with a motivation section, like why do we care? Because obviously there's lots of interesting use cases for VLMs. It was really interesting recently when you had the team from Google who were talking about the new Gemini APIs. One of the things they said was loads of people are building with large language models. Relatively few are building right now with visual language models. And that's, they think, going to be a growth area next year. So that's, that's kind of cool. There's loads of kind of use cases. The obvious ones are like medical assistance, you know, being able to look at image modalities as well as patient history, and then say things about the patient that might be useful to the clinician. But I mean, other use cases, you know, content filtering, for example, and knowing what is in an image and text. For example, if you were looking on a social media platform and you're trying to screen out images or content of concern, indexing large quantities of archival material, or like product catalogs or something like that, where you've got both visual components and text components. You want to better understand, you know, what is this product, given I can see it and given I've got some information about it. But I've also seen applications, for example, in insurance, where people are having photos of cars, for example, there's a description of what's supposed to have happened to the car, might be some damage. And the question is, can I actually see the damage in the image? Does it kind of reflect what the person said on the report? Yes or no? There's various use cases. But I think beyond that, there's kind of 2 other reasons we might be interested. In vision language models. One of them is that in building VLMs, what you're learning to do is integrate 2 modalities. They start off very separate, and somehow you're going to reason over both of them. And then if you can find the recipes for doing this right, then into the future, you can think about, okay, can I integrate audio, touch data, LiDAR data, other modalities? And if you think about, like, for example, robotics of the future, Just think about the number of different sensory modalities that you'd need to have a robot cook a meal. It's got to handle everything, see everything. You can think about VLMs as being the first step to learning how to do this so that we can integrate lots more for the future. So, that's kind of a longer-term thing. And the second thing is a bit more of a philosophical question, is multimodal understanding in an AI important? On the path towards AGI? It's not entirely clear that it is, but some people argue that it is. So one reason that one might want to research these things is to see if by integrating information from different modalities, you obtain another kind of transformational leap in the ability of a system to understand the world and to reason about it. I would say in inverted commas, similarly to the way we do.

Yeah.

We know they didn't do things the same way we do. There you go. And I guess there's arguments against, you know, is it important would be to say, well, look, frontier language models show lots of evidence of high-level abstraction, world models, sophisticated reasoning. There's no obvious ceiling in performance as of today. Maybe a grounded multimodal understanding of the world is not that important for achieving AGI. But we'll kind of explore this. Maybe there are some little bits of evidence we'll come across today, which might point us towards one or the other view here. Yeah.

I mean, I would be very surprised if we end up, I mean, it just seems like, imagine yourself unable to see, right? It's like, it would certainly be a major hurdle to have to get over. And my guess is that, and maybe we'll, you know, shine some light, so to speak, on this question as we go, but My guess is we'll never really answer that philosophical question of like, could we have built an AGI that isn't multimodal? Because, you know, to state the most obvious spoiler in the history of the world, like there has been a lot of progress and it seems like if nothing else, it will be the path of least resistance. Like multimodal is clearly going to work. The details are to be unpacked, but it seems. Like maybe, you know, the sort of philosophical crowd will continue to say, well, we might have been able to do it without multimodality or it would have been impossible. But, you know, it seems like in the end, this is going to be the norm and these things are going to walk among us. Probably if I had to guess, you know, sooner rather than later.

There's always sooner rather than later, isn't there, in this world? So I suppose before we dive into the first vision language model that we'll cover, there's probably 2 Important little prefaces we ought to do. One, we ought to talk about vision transformers for a moment. And then we ought to talk about the CLIP model from OpenAI. And the reason is that both of these are going to crop up again and again. So let's just kind of refresh our memories as to what they are. And then we'll kind of dive into the VLMs themselves. So the vision transformer itself. So I'm going to assume we're all familiar with a language model transformer. especially the decoder architecture. The canonical paper here is called An Image Is Worth 16 by 16 Words, which is from Google about 4 years ago now, 2020, I think. And previous to this, like most vision models have been based on convolutional neural networks. So they've basically been stacking convolutional filters to extract more and more kind of global features from the images. And the question that the Google team asked was, well, could we use the transformer recipe to build something that understands images. And so the recipe is, let's say, quite straightforward. You take an image and you divide it into non-overlapping patches like this. Okay? You then linearize the patches and you have a linear embedding, which basically converts them all into tokens. Okay? So now we have just a sequence of visual tokens through a learned embedding. And then we feed these patches one by one into a transformer encoder, and we use full attention across it. So every little image patch can pay attention to every other image patch in the image. Okay? And this is very similar, kind of, in thinking to how a model like BERT is trained, because what we do is they stick a classification token, they prepend it to the sequence, and the training objective is, can you classify the image that you've seen? into one of a large number of categories. You take the classification vector out the end, and that's what you use to figure out if you've got the right classification. So very, very simple recipe going on there. And the key finding was that if you make these things big enough, these vision transformers, the transformer architecture does beat the convolutional neural networks of the day. And so that makes it a very kind of useful building block. There's just a couple of things we ought to take away from the design of the Vision Transformer, which is also called a ViT. So I'll probably use the word ViT throughout. The first is that, note, the image resolution is going to be fixed by design. So in the original Vision Transformer, it was 224 pixels square. Okay, so everything has to be that size when we feed it in, and then we get a fixed number of patches coming out. For the original training, They would stick, like I said, this classification token in. But when we come to talk about the vision language models later, normal practice is to take the entire sequence of hidden states from the transformer out and use that as your encoded image.

Okay?

So we don't just take the classification vector, we take everything. And that means you could get quite a lot of vision tokens out of such a model. So if you started with 224 times 224 as your image size, And your patches were 16 times 16, a bit of back-of-the-envelope math tells you you're going to get 196 visual tokens out at the end, which can be quite a lot. Okay? So that's the other thing to note. And I guess the third thing is, and this is just a convenience, when we talk about vision transformers, you'll hear them described like ViT-H16, for example. And this just tells you something about the kind of dimensionality of the vision transformer. So the ViT's telling you it's a vision transformer. The H in that stands for huge. And we just have to know that huge stands for about 600 million parameters. And the 16 tells us the patch size. So that's what we're patching our images up into. So if I use that later on in here, you'll know what I mean. If I say ViT-G16, for example, it's a giant one, which is even bigger than huge. There we go.

Hey, we'll continue our interview in a moment after a word from our sponsors. So a couple little follow-up points there just to make sure I understand this correctly. One, just to contrast the image attention pattern versus the language model attention pattern, at least the one that we're most familiar with, which is like a lookback-only pattern, right? In language, the attention in The image context is generally all to all, right? So there's no, there's not like a sense of ordering or the sort of, you know, obviously language unfolds token by token. So that's one kind of fundamental difference is just that this is more of a, which obviously reflects the modality itself, right? The image is a snap of a scene in time, and that is all on par with each other in the way that it's being processed, all to all. The other thing that I wanted to dig in on just a little bit more is how the tokenization happens in language. We have these tokenizers which sort of try to figure out what's the sort of optimal way to— and it's interesting that those are typically not part of the end-to-end training, right? You sort of have this separate sort of bolted-on system that people have kind of hated and have tried to get rid of for a long time. And maybe there's some signs that that could be about to happen. Some interesting, I was just reading last night a paper from Meta that's about a more dynamic way to batch text as opposed to with these fixed tokens that are predefined. But, you know, if anybody hasn't seen this, you can go to the OpenAI tokenizer and just paste in your text and it will immediately chunk it into bits of text and color code them. And you can see. what all the tokens are. So that's a vocabulary of, I think now up to like 100,000 different little bits of text that text is broken down into before it is translated into numeric form and then processed. And that translation of this token to this vector representation is something that at runtime is fixed. And there's basically as many Yeah. People often refer to this as one-hot encoding, where there's as many possible input vectors as there are tokens.

Yeah.

How— that's a little bit different now in this case, right? For images, my understanding, if I'm understanding correctly, is there's not like a fixed vocabulary size of possible tokens, right?

That's correct. So We're going to use the term throughout most of the day, we're going to use the term tokens quite loosely, because as he correctly says, text tokens can be mapped back to text through a codebook. You've literally got 80,000 codes, you look it up, and you get your byte pair or whatever it is at the end. The same is not the case with visual tokens. They exist on a continuum, as you pointed out. So, to go from your little patch, which is really just a matrix, you've got A few dimensions, some channels in there. You're simply gonna pass that through a matrix, which is going to generate you the vector that you want to stick into the transformer. And it's learnable. That transformation is learnable. But the important thing is that your tokens are gonna come out on a continuum. Okay? And they don't need to be quantized at this point. There's nothing in the transformer architecture that says tokens have to be quantized to a codebook. You can still run the attention mechanism, even if your tokens exist on a continuum like this.

Okay, cool.

Note that because we're training, like you said, the visual, the vision transformer with a classification objective, we don't actually have to decode anything at the end. So it doesn't matter.

I'll then I'll save my next question that I have for as we get a little deeper into the journey here. I think the last thing that's worth just reflecting on for a second is just how small the images are that are being Process. So I've done a little bit of this, not recently, because these days we have these foundation models where I can just throw basically anything into it. I think there are some hard limits that I've run into occasionally if my image is like north of 10 megabytes or whatever, but typically just throw it in there. They handle it. You as a developer don't really have to worry about it. With earlier generations of models, you did have to do this sort of preprocessing where you would take your image. It was your responsibility as a user of the model that somebody open sourced for your convenience to take your image and basically morph it, or I guess resize it is probably the right term, into the required size that the model could handle. And it's presumably just because everything was smaller back then and compute resources were more limited and the results weren't so spectacular in general.

Mm-hmm.

224 by 224, even until fairly recently, and even with the OpenAI kind of small mode, whatever their sort of low-res mode is, it is remarkable how much performance can come out of these very small images, even when they're dramatically shrunk. And they're often significantly distorted because your original image might not have even been a square, but you're just kind of like, whatever, I'm just going to make it a square. I don't care if things are smooshed. I don't care whatever. happens, that's what we're going to work with. And it's amazing to me how well that actually works. And I think these days that's getting liberalized for sure, because it's not all low res on like the OpenAI API, but it is remarkable how far that can go.

Yeah. Without wanting to spoil the big reveal, of course, they're not compressing everything to 224 by 224 and using that as the image inputs. There's some much more sophisticated and smarter things going on. At least with the leading vision language models. So we'll see how they do it in a bit.

Okay, cool. Well, let's carry on. That's a great start.

Cool. So that was the Vision Transformer. And I think the other thing is that we should introduce right now is the CLIP model from OpenAI, because that's again, going to be really fundamental. And the paper here is from back in 2021, and they called it Learning Transferable Visual Models for Natural Language Supervision. And CLIP itself stands for Contrastive Language-Image Pre-training. So it's a canonical model in the field, and it's a really nice introductory one to study for how we align image and text encodings. Okay. So the idea is you start with a vision encoder, which could be a vision transformer, and a text encoder, which could be, you know, encoder-only transformer. And you've got a large dataset of images with their captions that have been scraped from the web. Okay? So the process is to jointly train both encoders so that they're generating embedding vectors, both for the text and for the images, such that if you take an image and its caption, the 2 vectors are going to have very high cosine similarity for an image and its caption. But if I take 2 random captions and an image, they will have low similarity. And the way this is done is you simply pass the image through the vision transformer, the text through the text transformer. You then add just a linear projection to get them kind of the same conforming dimensionality. And then use this contrastive loss function. And then what that does, it says, suppose I've got a batch with like N image-caption pairs in it. I know within that, that I've got N true pairs. And I've got n squared minus m bad pairs. Okay? So you set the loss function up to say, penalize any dissimilarity between my true pairs, and likewise penalize any similarity between the non-pairs within the batch. And that's a contrastive loss function. It's basically bringing things that are supposed to be the same closer together, but pushing the vector representations apart that things which are not the same. So that was how the CLIP model worked. And one nice thing about this is that once you've trained it, I mean, obviously you can use CLIP itself for things like image search, but you can also just take the trained vision transformer out of it and use it downstream. And that's really nice because what you've done is you've already, in some senses, kind of trained it to embed things similarly to a language model. So therefore, if we were going to put that into a vision language model, it should only be a small step away from being aligned to whatever we're going to use in that model. So there's the thinking there. And that is why, as we kind of go through to the rest of today, we're going to see that they very often, the researchers start with a CLIP. It gets them kind of two-thirds of the way there. So, well, I wasn't going to do a deep dive into the CLIP model, but just to say that, you know, It is a vision transformer. That's how it's trained using this contrastive loss objective. And we can now use it downstream.

Yeah, I remember using this one in the early days of Waymark small business video creation. We were at the point where we could get the fine-tuned GPT-3 to write a somewhat decent script. You know, certainly things have improved a lot since then, but then you had the challenge of, okay, now that we have this. narrative that we want to put together for this random small business. And we've also found all these images off the web.

Mm-hmm.

What should we choose out of this bag of images? Right. And at the time, you know, you had sort of prior to CLIP, you had like classic, you know, most of this stuff was done on finite sort of pre-established datasets, right? I mean, then one of the big advances of CLIP was That they moved from your like ImageNet type thing where you have a certain canonical set of images with a certain fixed set of classifications. And the game used to be about how, you know, comparing architectures basically, right? Can I come up with an architecture that on this standard dataset does a better job than everybody who came before me in some way, shape, or form there, you know, before I can claim to have state of the art and I'm great. That didn't do much at all for us with a application like Waymark because it was, you know, maybe could have made it work, but it would have been very noisy because it would have been like, well, the image that I have of this, you know, business, whatever it may be, potentially is not even well represented at all by any of the classes in an ImageNet set of classes. I think it's 1,000 classes that they have in ImageNet. And there were smaller ones before that that had fewer classes. So this was the first moment, as far as I know, where they went, let's forget competing on these standard datasets. What people really want is to understand anything that they might be looking at. And the web-scale data is out there, and enough of these images are captioned. And by the way, when you really got into CLIP, tons of noise in the caption data. Typically, if you just ask yourself, like, how do people caption images? You know, it's lots of different ways, you know, sometimes with jokes, sometimes with just a straightforward description of what's in it, sometimes with a line from a poem, you know, I mean, it's a tremendous amount of noise in that original dataset. We found in using it that it was pretty good if you said, okay, I want an image of this. Which of the images from this set of possible images most closely matches that query, we could get a pretty effective sort. But we'd see all sorts of artifacts and weirdnesses. Like sometimes if the text that we were, you know, let's say we were, for example, doing a video for a pizza restaurant. There's a pizza restaurant in Detroit that I've always kind of used as one of my go-tos. If you put the word pizza in, as your text query and run that through the text side and then put all the images through the vision side. And, you know, each one now is represented by the vector. And as you said, like, if they're similar, you should have a high cosine similarity. So basically literally multiply these vectors together, take the sum and sort by that. That was the procedure. But sometimes what you would find is if the word pizza in text was in the image, Then that would pop to the top. And so you'd have all these sort of weird things to deal with. I think largely as a reflection of the fact that this was just extremely noisy web-scale data that at the time they didn't really even necessarily have a great way to clean, right? Because it's like all this technology is kind of being created out of nothing. These days you would say, well, why don't you just filter those images that have a caption image content mismatch, or that seemed nonsensical. And I think the answer at the time was basically, we have no way to do that. So we just have to try to throw as much data into this as we can and hope that some signal comes out. And it did. We did find also that, I promise I won't do this every single step, but I have memories of CLIP.

It's good.

Aesthetics were basically totally unrepresented.

Hmm.

And I guess this would be because a few of the image captions found online are like a beautiful picture of X or, you know, an unattractive picture of X. But we wanted that because small businesses have very wide ranging quality of pictures. Sometimes you get user generated content stuff posted to their Facebook. Sometimes it's professional. The difference matters a ton, right? They do not want some of these ugly images that might be associated with them online. in their marketing, but how could you tell? There was no real aesthetic signal in Clip. It was all content, but not quality. So it's crazy. That wasn't all that long ago.

No, we're still 3 years, 4 years ago. And now we're going to jump to 2 years ago. And then everything else we talk about is going to be in the last 2 years. And you're right that one of the stories that's going to unfold is we kind of COVID a few of the VLMs today. Is this increasing obsession with filtering data for quality, for both the pre-training and for subsequent fine-tuning stages. And that is to kind of get rid of this problem of noisy data, which really does seem to hurt VLMs in particular. Cool.

Well, I think that's enough memory lane down the clip alley for me. So let's keep going.

Hey, we'll continue our interview in a moment after a word from our sponsors. Okay. So we're going to jump forwards then to 2022 and a model that I've heard described as the GPT-3 moment for vision language models. And that is DeepMind's Flamingo model. A number of really interesting innovations in this. And so it's worth covering this in a bit more depth. And also it's the first example we'll see of actually how a VLM is constructed. Okay. So the basic pattern, and we're going to see this through all of the models we cover, is that you're going to encode the 2 modalities, that is text and images, separately. So you can use a text tokenizer, and you're going to use an image encoder, normally a Vision Transformer. We're then going to select a language model, which is called the backbone. And the backbone is going to be the thing that does all of the reasoning over both the text and the images. And that just leaves us with a question of how we connect the two. So, for example, in the Flamingo model from DeepMind, they chose, they actually looked at a vision transformer and a convolutional neural network. Everything I've looked at since has been just using a vision transformer. And then they used a Chinchilla language model. Okay. So to connect the two things together, they decided to freeze the language model and then introduce cross-attention layers sandwiched between, I think, every 4th transformer block. In the Chinchilla language model. Okay? So the idea of the cross-attention layers is that they're going to look up information that we got from the vision transformer. And there's immediately, upon thinking about it, you might see a couple of challenges that need to be resolved for this to work. So the first is, if we've got an input, say, that's got more than one image in it, then obviously when we've encoded those, we're going to have a variable number of image tokens generated. And if we're using a cross-attention mechanism, so it's similar to self-attention, other than the keys and the values are not coming from the text we're decoding, they've got to come from the image, then that's a problem because obviously the dimensionality of the cross-encoder mechanism is fixed. So we need a fixed number of visual tokens generated. So what if we've got 2, 3, 4 images, or we've got different sizes of images? And then second question is we can sometimes get a lot of visual tokens. So if we want to train really efficiently, is there a way to like reduce the number of visual tokens that we're actually going to attend to that will just make training the model a lot easier? Okay. Cross-attention layers don't need to be quite too deep. So the way that the DeepMind team solved this is, I think, very smart. And they used something called a Perceval resampler, which might actually have been their innovation, I think. So Perceiva Resampler is like a separate model that's going to be introduced. So not our cross-attention layers, we're introducing a separate fresh model here. It's going to look at the visual tokens. And it's designed to select out or sample, resample the most important visual information that the vision transformer has encoded. But you want to do this in such a way that if you have a very, very, very long sequence of vision tokens that have come from one or more images, What you don't want to do within your Perceiver Resampler, which has its own attention mechanism, what you don't want to do is compute the kind of all-to-all sequence-time-sequence attention matrix, because that could be very big. Okay. So the innovation here is that in a normal attention mechanism, you've got your queries, keys, and values, right? And they all come from the context that you're decoding. All right. So the idea here is that the explosion comes from the fact that the queries need to be multiplied by the keys at some point. And that creates this all-to-all attention matrix, which is size of the sequence length squared. And it is, of course, the bane of transformers. So what they did in the Perceiva model is they said, well, what if we jettison this need to come up with queries based on the context that we've just read? What if we instead have a small number of fixed queries? And when I say fixed, I mean learnable. So they're just latent vectors that can be learned at training time. And they selected, for Flamingo, they selected this to be 64 query vectors, which can be learned at training time. So in the Perceiver Resampler model, you're going to look at all of the visual tokens that have come in. Your query is going to be a size 64. So the actual query key matrix that you calculate is now sequence length times 64, which is way, way, way smaller. Okay? The beauty of that is once you finish the attention calculation in the Perceivable Resampler, you've got something that's 64 times whatever your hidden dimension is. So it could be 768, for example. So again, you get something very small out of it. And the Perceivable Resampler module is essentially this, and it does it in a number of stages. But it's essentially just using these learnable keys. And the nice thing is about the end, we know the size that the visual tokens are going to be at the end, 64 times 768, for example, which means that we can now define a cross-attention layer, which is always going to read visual tokens of that size that have come out of the vision transformer. Is that making sense?

Yeah. And this happens after the initial Vision transformer layers, right? So we've got still the all-to-all attention happening on the vision side of the, it's kind of 2. I can remember the chinchilla diagram and even more so I can remember the bowl of yarn soup that I went around showing to people at parties in spring of 2022 and into summer when I was like, Look at this. This exists now. So you've got basically, again, very similar to CLIP, and this will be a pretty common theme, although there are some exceptions too, but images being processed through one kind of main model, text being processed through another main model. These things are often totally frozen at the time that we now want to figure out a way to fuse them. And the way to fuse them is 2 parts here. One is a cross attention, but then the second is figuring out how to essentially make life easier for the cross attention and the kind of main language model that's going to carry things on by finding a way to standardize the sequence length for the image. So toward the end of the image processing with its full attention, Then there's this sort of adapter that says, okay, regardless of how big the image was or how many tokens, whatever, it's always going to be output from this step at 64 tokens. And therefore the cross attention can always be the same size and everything downstream of that can get simpler. I think this is a really interesting thing. I mean, I probably didn't have the sophistication at the time to appreciate it, but Yeah. it's a good indicator of just how malleable all of these latent spaces are and how I sometimes say everything is isomorphic to everything else, which I think probably doesn't actually mean anything. But the intuition that I'm trying to express there is just like, I've seen so many of these things at this point where one space is bridged over to another space or reformed, or you almost see this even just with the libraries that are used. I'm thinking of all the Python libraries that allow you to reshape matrices.

Yeah.

It's almost like those matrix reshapings are— you don't think of those as semantic, but when you scale them up to this sort of thing, a very similar thing starts to happen where you're like, okay, I've got all these, all the all, but what I really need is just a sequence and it needs to be fixed length. And Okay, I'm just going to train a thing that no matter what, it's going to output a fixed length sequence and hope it works. And I mean, not everything works. I'm told by the researchers, you know, there are many failures along the way, but it sure seems like almost everything of this sort kind of works. And I think that's a really striking reality. I remember seeing this one and there were, there was BliMP-2, which was another one that really brought it home for me just because of how few new parameters had to be trained. I don't know if you have those numbers like right off the top here, but it's a strong contrast between the amount of parameters and pre-training on both the text and image side versus how small these connectors can be in many cases.

Yeah. So one of the teams from Hugging Face did a comparison and we'll talk about it in a bit more detail later. They said in a 7 billion parameter language model class. If you add the cross-attention layers in, that's about 25% of the parameters now need training. Whereas they said, if you're going to do it the other way, which is just a simple projection, that's about 10%. So you do introduce a lot more parameters using this kind of cross-attention mechanism that we're talking about, but it's still, it's small compared to retraining the whole language model. And that's actually one of its benefits. is that you can literally freeze the vision transformer if you want. You can freeze the language model and just train the cross-attention parameters and the Perceivable Resampler. And like you, I'm kind of amazed that the Perceivable Resampler works because it feels to me just like tipping the image into a blender, pressing on, and then somehow when it's finished training, the important features are retained and still there for you. It feels to me like they would have all been mixed up, but— I guess it goes to show intuition, at least for me, doesn't really work when it comes to these very sophisticated mechanisms. Yeah. Okay, cool. So let's just say a little bit more then about— so we've got the Perceivable Sampler, we've got the language model, and then to train them, you simply need to switch on your next token prediction training objective and start training the new nullationalized layers in the language model. And that's not too difficult to do. And what's happening is you're replacing where your images would have been in the text prompts that you're giving to the language model during its training. That's prompting it to look up the outputs from the PerceivResampler. So these are not tokens that get scored, for example, when you're decoding, but it's prompting it saying, when I see this token, it's okay, I need to go and look up something in the PerceivResampler. And that's going to give me enough context to know what the next token should be, because it's where the image would've been in my input text. And in terms of the training datasets, like to see how this actually works in practice, they obviously used images paired with their alt texts, which is the same that's used in CLIP, but that's not enough. And one of the contributions that the Flamingo team made was to realize the real importance of what's called interleaved data. So interleaved data is basically data scraped from the web, like HTML documents, where you've got images and you've got text in the document. And they know by looking at the document object model, when they scrape the website, they know kind of roughly what order these things are supposed to appear in. That interleaved dataset that you can then pass through your decoder, and then you can say, when you get to an image, look up the image. Now I'm going to keep on producing the text. They found that it was really, really important for actually maintaining all of the downstream performance metrics. So interleaved data turns out to be super, super important for vision language models. That was one of the major findings of the Flamingo paper. So in terms of how they evaluated this thing once they'd done the training, it was visual question answering and OCR and captioning. Those are the key tasks they looked at. And they discovered, firstly, it works quite well in a few-shot setting. So the training was very compute efficient because you don't need to touch anything in the language model other than the newly initialized parameters. And yet the output was still very competitive with very much more focused task-specific models. So they had a model that could do several things and it could be competitive with all the task-specific ones. And that's why it was a really kind of, I think, foundational model in visual language models. It's because you had this kind of several tasks you could perform and you had this one training recipe strongly based on the transformer training recipe. And the basic pattern of using cross-attention layers has been used by other teams since, but it's not the only way of doing it. And we'll come on to the other way in just a second.

They never released Flamingo to the public, did they? I don't remember ever having a chance to Actually use it? I just remember—

I don't remember. I don't think they did, unlike what we'll come onto in a second and most of what we'll cover today.

Yeah, that was before Google was dancing, I guess.

So that's the basic training recipe we just covered for a cross-attention model. One of the things that makes large language models so good at what they do is the instruction tuning. And that's proved to be one of the big unlocks. And in the Flamingo recipe there, it's not kind of clear, you know, there was no specific instruction tuning step. And the difficulty there was it's just really difficult to come up with an instruction tuning dataset, right? There weren't such things around. There are now. And that kind of this, how to do this is the contribution of the next model that we're going to look at, which is the LLaVA model, which stands for the Large Language and Vision Assistant. And this is a 2023 vintage model. So we've already jumped on, like, another year. So the original LLaVA model comes from a mixed team, from some people from Microsoft and then from academic institutions. And it's the first in kind of a long series of LLaVA models, all of which have kind of been based on this same recipe. The big innovation here is instruction tuning and how they built the instruction tuning dataset. They start with a key observation that the generative vision language models that existed at the time they built this could only follow a relatively limited range of user instructions. So they could do some captioning for you. They can answer some basic questions about images, but they couldn't do anything like the range of tasks that a language model could do once it had been instruction-tuned. And one of the reasons they put this down to was just looking at the big datasets. That's, that we used to train these models. You had the interleaved text and images from crawled web documents. And you also had these big captioning datasets. We'll talk about a couple of them in a second, but relatively few of them were actually containing like lots of task-oriented labels for the images. And that was just missing. So the question is, how can you build such a thing that would work for a vision language model? And I should probably talk a little bit about the basic architecture of LLaVA, because it actually looks different to the Flamingo model. We've talked about that Flamingo is described as being a cross-encoder model, because they've introduced these new cross-encoder layers. The LLaVA team chose actually a simpler approach, and this is pioneered, I think, by the Salesforce team that built BLIP. So we can call this the autoregressive architecture rather than the cross-attention architecture. And the idea is that you're going to take your vision tokens, which have been processed by a vision transformer, even better if it's from a CLIP vision transformer, because that's already been aligned in the past to, to a language model. And then you're going to train a simple projection matrix and inject the tokens directly into the decoder stream. Okay. So no cross-attention needed here. You have a simple projection matrix. And then you're going to mix them in. Although in the original LLaVA architecture, they kind of prepended all of the text tokens that were going into the language model backbone with all of the vision tokens. So your kind of training sequences would be a bunch of vision tokens, text, and you'd then be attending to everything to the left. Okay. So that's the autoregressive architecture. It seems a lot simpler than the cross-attention one. But it's got maybe one or two downsides to it. The first is that if we generate a long sequence of vision tokens from the vision transformer, we've literally now got to unroll all of them in the decoder, right? So they're all going to be part of the attention mechanism in the decoder, right? So that's kind of one disadvantage, perhaps. The second is that if we've got this projection matrix, as was used in— and I think they used a simple like a linear layer in the LLaVA model, that gives you many fewer parameters than introducing a big cross-attention mechanism. So you've now got to learn all the alignment using those parameters. And if you want to do anything further, if that's not enough for you, you've got to unfreeze the language model backbone, and you've got to start mucking around with the attention mechanism. So one of the downsides there, as we know, is if we start fine-tuning, you know, the language model and the attention mechanism, is it's very easy to kind of suffer catastrophic forgetting on some of the tasks that the language model was fine-tuned for in the first place. This is one of the downsides of using this autoregressive architecture that just uses self-attention, but that's what they chose to do with LLaVA.

Yeah, you mentioned BLIP. We did one of my first episodes actually of the podcast was with, with the authors of Blip 2. And I remember this sort of, you know, moment of understanding where I was like, wow, so they are somehow converting these images into text embedding space and treating the images as if they were text from that point on. The rest of the model doesn't even know. Yeah. That it has been fed anything other than text because, you know, it's, it's frozen, right? So, I mean, it's been trained to handle text. It can only handle text, but now somebody's figured out how to represent these images as text in the text embedding space. And one of the really interesting things about that was how you are accessing parts of text embedding space. that text itself can never get to, right? It's similar a little bit to what we talked about earlier, where you have the one-hot token-level text encoding, but these image projections into text embedding space are not bound by that. At least they weren't in the BLIP one. I don't think they would've been in this LLaVA one either. And so you're just accessing, you realise that the space of possible inputs is actually hugely bigger than the space that the actual 100,000 token vocabulary routinely accesses. And that again, just kind of magically seems to work. It's like one of these divine benevolence things where you're telling me we trained an entire language model on nothing but this 100,000 tokens. And now it's going to be just, we're going to bypass, you know, that layer and just directly go into embedding space with whatever is learned by this projection process. And the language model is still going to handle that normally. And the answer is yes. It's kind of amazing.

It's quite an amazing result. I mean, as a simple projection matrix, taking, takes your vision tokens. Lets you mix them in. I mean, and in other models, arbitrarily mix them in with the text tokens, and it doesn't destroy the performance of the transformer. It doesn't seem to need complete retraining of the attention mechanism. Like, that simple projection matrix is enough to get it to work with the visual tokens, which I think is quite an amazing finding.

Yeah, no doubt.

I mean, I don't like to do analogies all that much, but I'm just, I'm trying to sort of imagine like, what is there any, you know, sort of analogous challenge that we could pose to a human? I mean, we're obviously natively multimodal, but to think about putting something into word space that is not words, it's like kind of going directly into thought space, I guess. And yeah, that was, There were a few of these moments for me. I think one of the reasons that I got so obsessed as early as I did, not having had an academic background in the subject, was seeing how the same-ish architecture was working across modalities and then starting to see these bridges where it was like, boy, if you can just take 2 frozen things and kind of mash them up in a couple of different ways and they all kind of seem to work, Then that means we are headed for a world where all of this is going to be integrated and it's going to be quite something to see when that happens. So I, these were for me, like very leading indicator moments of just how much was almost certainly going to be possible. It was like, if, if the language model could do that and not break, then I think, you know, we've got to expect that there's going to be just a lot more. Frankensteining to come, much of which, you know, might be weird in all kinds of ways, but a lot of it's going to end up being viable.

Yeah, absolutely. So we said we'd talk a little bit about how they did the instruction tuning with the LLaVA model. And so this is, I guess, what the main contribution of the paper was. We've talked about what the autoregressive architecture looks like. So the way they did this is actually really, really smart. So, they started with images from a dataset called COCO, which was produced by Microsoft, I believe, back in 2014. And the idea behind that, there's about 200,000 images in the COCO dataset. You've got images, and then you've got descriptions with bounding boxes in them describing what's in different areas in the image. And so that actually appears in the text. It says, in this area, region, And then this thing, region is this thing. And the idea is that it's to teach visual grounding for vision models. So what the LLaVA team did, which is very smart, is they used a strong model. I think they used GPT-4, not a vision model, just a language model, and a very careful and clever set of Q-shot prompting templates. And they asked the GPT-4, can you generate a conversation between questioner and a vision assistant framed as though the assistant can see the image, although GPT-4 cannot see the image. Okay? So for example, given a description of an image, which might say, here's a bunch of people standing around a vehicle, there's luggage on the floor, and then some bounding boxes. Here's a piece of luggage, here's a bicycle sitting at the side, here's a person, and so in the bounding boxes. So Read that, GPT-4, and now come up with a question for me, okay? As though you could see the image. So the question might be, what type of vehicle is in the image? Okay, and that's kind of easy for GPT-4 to do because it, the image, the caption will say what kind of vehicle is in. But then you can think of more intelligent questions. For example, what is the thing to the left of the car? And we know from the bounding box that there's a thing, you know, the thing to the left of the car is a bicycle. So, not even needing to see the image, you can come up with that question just by simply computing where the bounding box is. Or you could ask a more in-depth reasoning question like, what challenges do the people around the car face? So, people are loading luggage into the car. So, if you just saw the image, you would have to infer all of this stuff from the image. So, the really smart thing here is getting these kind of conversational dialogues built up. By a strong model like GPT-4. And then you've actually got all of this data you can use for instruction tuning. So you can now ask the questions, and you've also got the model answers, because GPT-4 knows what the model answer should be. Remember, it came up with a question and the answer using all the information it could get from the descriptions and from the bounding boxes. Okay? And that's the genius here, is that when they built the LLaMA model, they froze the vision transformer, and then they updated the language model they were using, and obviously the projection matrix, which is going to align the two. And they only did the pre-training on 600,000 pairs from a captioned images dataset. And they used a very simple form of their instruction question answering. So it was something like, here's a question that GPT-4's come up with about the image based on the caption. Then you show the image, And then what you want a language model to do is finish with the answer, which would be the answer to the question that it came up with. And then for the fine-tuning, that's when they used this really complicated, more sophisticated instruction tuning dataset they came up with, which includes multi-turn conversations, questions about the images, asking what's in particular regions of the images, asking these reasoning questions again. And they were able to generate 150,000 of these things using this process. And so they used that as the instruction tuning dataset. And when they evaluated it at the end, they found that it was outperforming all of the other vision language models at the time, complex reasoning tasks over images. And they found it was also slightly better at conversational tasks than anything else. So what they showed was that Putting all this effort into being really smart about generating augmentations for the instruction tuning dataset yields a much smarter language model at the end that could complete a much wider variety of downstream tasks. So just because you generated them all. And so I think that was a really smart innovation. And in the subsequent LLaVA models, the same kind of recipe for generating instruction tuning data has been followed. Although the models have got a lot more sophisticated. And the latest one, which is, I think, from a team at ByteDance, is called LLaVA 1 Vision. And that's sitting really, really well on the MMMU leaderboard, which we'll talk about in a moment. So it shows the recipe is still like strong a year and a half later. And the class of models is really competitive.

Yeah, it gives you a sense too for why the leading developers seem to think they'll be able to get around whatever data walls may be naturally occurring because these were, you know, these results were achieved with largely synthetic data, at least when it comes to the final margin of actually getting the thing to behave and like be a useful, you know, image assistant. So, yeah. This next one isn't so interesting. I don't know. Well, maybe we'll skip it, but At the time, in summer 2022, I was fine-tuning the text DaVinci 002 series of models, and they never actually released the fine-tuning there. I think it wasn't very efficient, and they had other things they wanted to do with their resources. And GPT-4 was cooking, although I didn't know that at the time. But we had this kind of preview access to that thing, and I was using it to process images in a similar way. One challenge I remember we had was we wanted to be able to take a video and look at, or try to answer the question, how much, well, first of all, what text is on the screen, but also how long is that text on the screen? And for an arbitrary video, you could take a still out of it and use OCR, and that's what we did, but then it was like, okay, well, now I've got 900 frames in a 30-second video. What is the right frequency to take these stills and then OCR all of them and then come through and have the language model process? That data was so gnarly, but that was one of the first things where I had this experience of feeling like, geez, I could go try to get human annotators to do this. and create some dataset and train on that. But actually what I'm finding myself drawn to do, and I think this is like actually still a pretty winning recipe these days for many things, is 10. I was like, I'll do 10 myself.

Yeah.

Then we'll fine tune on that. We'll have the language model do the next 100. And then, you know, hopefully most of those will be right. The ones that are wrong will correct, then we'll fine tune again. And in that way, kind of bootstrapping into something that could do this like very gnarly task that humans were not very good at, you know, very unfamiliar data, very sort of not, not the kind of thing we're evolved to really handle. You could do it if you really stapled your pants to the chair, but it was tough and it would have been really tough to hire out with any sort of quality. It was even tough for me to sit there and do the 10 or to inspect the next 100.

Yeah.

But that was definitely a moment where I was like, this is going to be transformative because now we're hitting the point where with the tools available, I can actually bootstrap my way into a pretty fundamentally new capability that, and I can do it in a way that's like way faster and way more affordable than having to go out and like actually hire human annotators. And that was just like, hmm, if I'm figuring this out, I bet a lot of people are figuring this out. And there's going to be a lot of new, in my case, it was narrow. I wasn't trying to create a general purpose thing.

Yeah.

But even then it was like, it's so, you know, if I can create a new capability in a few days of work as like a hybrid programmer, annotator, bootstrap my way into this sort of thing, If I can do that in just a couple days, like that's how long it would take me to like set up, probably would've even taken longer to set up like decent infrastructure to collect the data and figure out where, what platform are we going to use to have the annotations done and who do we trust to do this and how do we like validate that they're actually doing it versus, you know, just filling in whatever and not caring. And it was like, boy, I can bootstrap my way into that with just a day or two's worth of work. In many cases, once I kind of got the hang of it, it was another one of these moments where it was like, yep, this is going to happen across a very, very wide range of tasks.

Yeah. And I think this, this, this idea of using the AI models to generate your synthetic data for training the next generation is we see it all over the place with the vision language models. So there are examples where generating synthetic OCR data. For example, in order to train them to be able to read corrupted images and know what the text is. It's a kind of a classic one. I've seen generating LaTeX documents as well. So you generate the LaTeX document, and so you know the text you started with, synthetically generated the LaTeX, and now you've got a pair. And so this, you know, trying to grow the datasets, and this is always very difficult, especially in the realm of vision language models, because you can't just deal with a single modality of data. You know, there's got to be a correspondence. Between the vision components and the language components, and the correspondence has got to be good. Like, as you said back at the beginning, when we were talking about CLIP, it's like when there was a lot of noise in the data, it was only— there's only so far you can go with the quality of the model at the end. So one of the challenges and something that lots of the teams spend their time doing is thinking about how can I create much larger, really high-quality pre-training datasets? Where I kind of know I've got a good correspondence between the visual things that I want the model to learn and the language. So I think that, you know, the LLaVA recipe, and by the way, you can get the instruction tuning dataset from Hugging Face. You can see exactly how it's built there. I think it's one of the, a really smart way, a really creative way to think about how you build out a really sophisticated dataset. And then it has this dramatic effect when you actually use it for instruction tuning. On the capability of the model to act as an assistant, you know, it can do all of these other things that previous vision language models couldn't do.

Well, we're nowhere near the end. So, uh, I didn't expect this to be such a, you know, nostalgia fest. Uh, it'll, that'll be less because I think as we get closer to the present day, honestly, just the amount of time I've spent wrangling these like weird idiosyncrasies has dropped. Pretty dramatically as the foundation models have gotten better for an application developer like me. I've been very content to leave these sorts of weird things in the past. And we're, you know, in the next couple of models that you're going to get to through this history, we start to hit the point where it's like, it's just starting to work. So let's keep going. And I'll probably be doing less of this, I remember when stuff as we get to the present day.

It's all good. It comes with age, Nathan.

Time flies, that's for sure.

So, I just mentioned, like, at the end, I said that the latest LLaVA model is called LLaVA 1 Vision. And I said it ranks very creditably on this MMMU benchmark. So, we should probably say a word or two about what that is. So, MMMU, it stands for Massive Multidiscipline Multimodal Understanding. which is something of a mouthful. And I guess the easiest way to think about it is it's like the multimodal version of NMLU. So I think it's probably the most interesting and relevant vision language model benchmark for just understanding how smart your VLM is and how much reasoning it can do. So it's designed to measure explicitly 3 skills. Perception, what can it see in the image? Knowledge, what does it know about what the image is showing? And reasoning, so what can it infer from the image? And the way it was compiled is by a bunch of students, of course, from various disciplines and subjects at university, drawing questions from online sources, from textbooks, and even from lecture materials. And the idea is that each question that they find has to require an expert-level understanding from the domain in question. You need to have that to answer the question. So, they built up about 11,000 of these questions, and they ranged over about 30 different subjects. So, this includes things like history, medicine, electronics, market research, music. And some of them also require mathematical reasoning to solve. For example, some of the problems involve Fourier transforms. Each question says something like, Here is an image showing harmonic intervals in a musical score. Which one of these— and you've got like 4 musical scores— and they say, which one of these is constructed incorrectly? And then it's going to give 4 options. So it always comes in fours in MLU. And so the language model has just got to select, is it A, B, C, or D that's incorrect? So it's given the multi-choice questions. So quite diverse, a lot required of the models to answer them. And when they released the benchmark in November 2023, the top-scoring model was GPT-4V, and that got about 55% on the benchmark. So, like, random guessing would get you 25% on this. The top open-source model was one of the LLaVA models, like we just discussed, the second model in the LLaVA series, and that got 34%. One interesting development since then, I guess, is that O1, as you might not be surprised to know, is now topping the leaderboard. So, in just over a year, the leader score has jumped to 78% at the moment. So, O1 is sitting a full 8 points clear of the runner-up on this. You can see that benchmark is, I wouldn't say crushed yet, but a lot has been done on it in the last kind of 18 months.

By the time we get done with this recording, we might find a new leader at the top.

That is the risk because it's actually O1 Preview that is ranked at the top there. We haven't got, you know, who knows what's been done to O1 Preview since it was preview and now it's O1. But yeah, it could be beaten. Also, we don't have scores for Gemini 2.0, and we don't have scores for The new Claude model, I think. Actually, you might have to revise this because I think I might have them further down, actually.

Yeah, I'm looking at their leaderboard page right now. I don't know how often they are updating it. They do have 0.1. They do have a 3.5 Sonnet. Not clear if this is 3.5 Sonnet original or 3.5 Sonnet new.

Oh, this is the original 3.5 Sonnet. Yeah. I actually, actually think I've, I've managed to find all the scores for those models, actually. So I can tell you what they are later on. But anyway, yeah, O1 is standing top. So what's interesting about that is when the benchmark was first released, the team who produced it actually looked at asking text-only GPT-4 to answer all of the questions. What they would do is they'd extract, you know, any text they needed from the images by OCR, or they get the LLaVA model to caption. The images. And they basically give a text-only model, you know, the captioning that was extracted from the image and the question, and ask how it did. And they report a score from GPT-4 of 34% on the benchmark. And that, I guess, just highlights the fact it's above 25%, highlights the important role that reasoning will play in answering the questions. And so reasoning is really important. And the rest of the over 34% and up to whatever the models get, you can put down to their smart interpretation of the visual tokens and their ability to reason over them. So just a quick introduction to the MMU leaderboard. We'll come back to it because for now we're going to score every model on it, see how it does. Should we crack on?

Let's go.

Okay. So I guess the next thing to talk about is pre-training with vision language models and what's been learned about that. So what do people do these days? And the model we'll use for examining the pre-training recipes is the QuenVL series, which is from the team at Alibaba. So this series of models has 2 of them, QuenVL and Quen2VL, 2023 and 2024, respectively. So they're self-attention models. They follow the self-attention autoregressive architecture, and they're using Quen's language model, QuenLM, I think it's called, as the backbone. And they're using a vision transformer as the encoder, okay? To connect the two, they're using a single cross-attention layer. So they're not adding many cross-attention layers to the language model. They're using a single kind of standalone cross-attention layer, which, similar to the Perceivable Resampler, has got learnable queries. And so what they're solving for there is, if you've got an arbitrary number of visual tokens, can we compress it down to a smaller number? And then can we inject them into the language model decoder? This is really about, in this situation, this is really about using that mechanism just to reduce the number of vision tokens that are going into the decoder. Do you remember we said in the autoregressive architecture, you've got to unroll all of those tokens? They become part of the attention that the decoder's got to compute. So if we minimize them, that's better. So that's what they use for connecting the two. If we talk about the training now for the QuenVL language model, the innovation here is that they're actually going to break the pre-training into 3 stages. So rather than just doing general pre-training, what they learned and what everyone's done subsequently is that actually you want to break your pre-training into 2 stages and then do your supervised fine-tuning. So in the first phase of pre-training, For the VLM, they're taking their image caption datasets and they're taking their interleaved data. And so large, large quantities of data. They are training the vision transformer and they're training the connector module, but they're freezing the language model. Okay. And in the pre-training, they're having all the images resized to 224 by 224. So they're using the kind of natural resolution with the vision transformer. So, I presume this is designed so you've got, you know, not that much detail in the images, but you're able to process a lot of them. And the vision transformer is, of course, small compared to the language model. So it's okay to un-freeze that. And the cross-attention module, again, small compared to the language model. It's okay to unfreeze that. So that's the first pre-training they do. So the innovation is to add in a multitask pre-training step here. And the idea is that for the second phase, you're going to unfreeze the whole language model. Okay. You're now going to allow the images to be a larger size as well. So in the first training step, they were 224 by 224. Now they're going to be sized at 448 by 448. And the vision transformer is fixed, of course. So this, what this means is we're going to split each of the images into 4 tiles like this. Okay. So we've got many more visual tokens coming in to the second phase of pre-training. And by multitask, they're also going to be adding in synthetic OCR data. And then they're going to create a visual grounding dataset as well. And so the idea here is to have a dataset where they've constructed lots of bounding boxes and references for objects in the image. And they will try and do this at scale. With a pipeline that's going to build out this kind of second pre-training dataset at scale with bounding boxes in it, with references in the text, what the bounding boxes are pointing at. And then they go add in again, visual question answering datasets. By the time they started training the QuEN-VL model, there were a number of kind of visual question answering or document question answering benchmarks and fine-tuning datasets. That have been released. And some of these are of considerable size, 10,000, 50,000, 100,000 images in each case. So they actually added these to the pre-training dataset. So now they're representing much greater task diversity in the pre-training mix, slimming down the size of it somewhat. They're also adding in text-only data. And the idea here is because they're going to unfreeze the language model, they're going to be mucking around with the attention mechanism.

Now.

And so they need to keep that text-only data in there to kind of preserve text-only performance. So the multitask pre-training, it's bigger than fine-tuning. It's smaller dataset than the original pre-training dataset. And we'll see this theme again that because the multimodal dataset is scarcer, right? Pre-training like this, first on your low-quality data, then on your high-quality data. And then finally, taking a much smaller, like, supervised fine-tuning dataset, which involves a lot more kind of manual augmentation of the images and, like, using a self-instruct process to kind of come up with decent prompts, is the way to do it. And that actually, for the QuEN-VL team, produced a really, really strong vision language model at the end. QuEN-2, which is the latest one in the series, is sitting just behind the leading models from OpenAI, Anthropic, and Google. On the MMMU, but it's sitting above pretty much all other language models. The smaller models are all open sourced. The 72B class, which is their leader model, is available behind an API, but they do say in a paper that they plan to release it at some point. Okay. But I thought that's, it's an interesting model because they kind of really broke down this pre-training process into 2 parts. They mixed in much more kind of diverse data into the second multitask pre-training and had a really dramatic effect on the capabilities of the language model at the end.

So let me just try to summarize the narrative of the training of a model like this. You start with a language model, then you throw your sort of high volume, but low quality or mixed quality data edit. And that is the vision language pre-training step, which comes after the original language pre-training step. And the main purpose of that is to sort of get these things on the same page, right? Like, we at least need to sort of bring the latent spaces together. Once that's working, then you say, okay, now these things are, and I guess, why wouldn't you want to change the language model at first? Because basically, I guess you have a language model that you're at least reasonably happy with. And if you're doing backpropagation through the language model, you're like, well, I don't necessarily know what's going to change there. And I don't want to be changing things and potentially losing capability. When I kind of know what I need to change. What I need to change is the part that is mapping the images into the language latent space. So do that first, get to a decent place, then open up training of the full model. Now it's like, okay, you guys are generally working well together. Now all the sort of constraints are off. Now we're going to backpropagate through the entire thing. And the, I think the interesting detail, a couple interesting details there, but one was definitely the need to continue to mix in standard text-only stuff because again, you don't want to be over-indexing on this one particular task type that won't necessarily, I mean, what's nice about these things is they're super general, right? So you don't have to give them an image. You still want them to be able to work as normal. So you got to mix that.

Mm-hmm.

Text-only data in as you do this phase, but this is where they really sort of cohere or anneal or whatever is the right word into a single system that's all been end-to-end trained together. And then naturally the last step is the dataset's not given. And that's certainly been a very common theme outside of vision too. It's just that, you know, these sort of preference data Or instruct data on which a lot of the frontier models are trained is highly proprietary technology that is expensive to generate. From a recent episode that I did with Nathan Lambert, who studies this post-training stuff deeply, one of the major takeaways from that conversation was him saying, in the absence of a frontier model like a GPT-4, that we use to generate the data that we then turn around and use for instruction tuning, we would have no way to get the volume of that quality of data that we would need to do this. I've probably referred back to that on like half of the episodes since, because it's, I think, a really interesting data point for where things are going in terms of like, as they start to clamp down on reasoning. Of course, We've now also got Google's thinking model that is sharing its reasoning. So, and Chinese ones too, that are sharing their reasoning. So the final chapter by no means is written, but it's really interesting to see how that instruct, that final phase of training, the data that powers that has to be so high quality and is so expensive to get and therefore so valuable that it's typically not released. Meta, for all their openness, is also not releasing that kind of data. And so people are currently left to like try to generate it with GPT-4 if they want to do something in the, or whatever in the wild. But what is the future of that? Is that still going to be a viable option? Maybe with Gemini, you know, showing their reasoning traces, but, you know, maybe not. We'll see kind of how the dynamics evolve. But yeah, I think that's good. And it is remarkable how I always marvel too at like Chinese companies are not far behind. You know, I think, I don't know if you have a perspective on this, but in VLMs, in fact.

Yeah.

Yeah. Well, I don't know if now's the time where you want to maybe do this a little bit later, but I'm interested to hear your thoughts on leaders versus fast follows versus, you know, certainly I think we in the West, you know, a couple of very broad terms, the broad we and the broad West. seem to be, in my view, kind of overconfident about just how much of a lead we, quote unquote, have relative to Chinese researchers. Even on some of these earlier papers that we've been discussing, there's an awful lot of Chinese names on papers that are coming out of Western institutions. So that's a whole other dynamic here. But yeah, we can either put a pin in and come back to this, but I'm interested in sort of how you handicap this field and sort of how you think, like, how we should even determine, you know, I mean, the Chinese companies are definitely more open, but it seems like typically I would have said your OpenAIs and Googles seem to get there a little bit first, but they're not open. So if you were to say they're the leaders in open models, I would say, yeah, that seems pretty apt.

Yeah.

Leader overall, including the proprietary ones. I don't know if you would, you know, say the same or see it differently, but I'm interested in how you think about that head to head.

So we can answer that now. And then I'll offer a couple of comments to your observation about high quality data in pre-training. So the next model we're going to look at is the InternVL model. And again, this comes from an organization called the OpenGV Lab, which is based in Shanghai University. Okay. And it's probably the leading open source model. And the story behind the InternVL series is, it's been one about how do you scale these things up? How do you make them bigger and bigger and train them efficiently? And they're looking at efficient training and they're also looking at larger models. And one thing I think is quite interesting is that all of the top open source VLMs are kind of hovering around the 70 to 80 billion parameter mark in terms of size. The top proprietary models, we don't know, right? We don't know how big they are, but bigger than that, right? Maybe an order of magnitude bigger than that. The latest model that the OpenGV lab released, they looked at how efficiently they could train. And they said they ended up training on 120 billion tokens, I think. of mixed data. And they said this was very efficient because as compared to the QuEN-VL model, which we just looked at, the latest QuEN-2VL, this had trained on 1.5 trillion tokens. Shortly after they released the model, they released a dataset which housed 1.5 trillion tokens in it. So it's one of the largest out there. So you can see already that they've gone to a 72 billion Parameter model class. They trained on 120 billion tokens to get there. The dataset they just released is 10 times that size. So they can see that, you know, my guess would be they're planning to scale up very quickly and get to a much larger model since they already have the top open source model and it's pretty competitive with everything else out there. I'm really interested to see what they come up with. And this, of course, is a, you know, It's a team from a university, which is super impressive, without the enormous funding behind it that some of the frontier labs have.

Now, when you say top vision language model, is that an MMMU?

Yeah. So sorry, I'm being lazy. I'm talking about just MMMU, which is my benchmark of benchmarks. We should probably talk a bit later about a couple of the other benchmarks, which are really interesting, but MMMU is kind of the big one. In the same way that MMLU might be the default benchmark you refer to when looking at language models.

Yeah. Yeah. Okay, cool. My kind of rough understanding and not necessarily rule of thumb, but I remember Llama 2 was trained on 10 trillion tokens and I've always kind of rounded to say It seems like the multimodality comes at another 10% cost, like it's another 1 trillion-ish tokens typically. Do you think that's a good intuition? I mean, this intern VL one seems to be notably significantly less than that. It's like 2 orders of magnitude difference between the original full pre-training and the image portion of training. But what would you say is kind of the normal? Is this in fact an outlier in terms of how small the dataset is?

I think there's not a normal because what's happening is the datasets are getting much bigger now. So say at the start of this year, so the biggest datasets, multimodal dataset you could train on would be the LAION, right? Which is 5.8 billion images with text captions. It's been filtered for quality. So there should be high quality captions. And they're all extracted from the Common Crawl. And so there's a German organization that produced this, I believe. And that, I think, is the largest kind of image caption dataset that's publicly available.

Okay.

But just in the last few months, we've seen an interleaved dataset called MINT, which has been released by— it's a multi-contributor team, but Salesforce were behind it. So that's a trillion tokens, that one. And it includes not just HTML, but it includes PDFs, archive papers, things like that. And the OpenGV Lab, who are the team that produced the InternVL model, have just released a dataset they call Omnicorpus, which is, again, another interleaved dataset. It's 2.2 billion documents sourced from Common Crawl dumps. They say it's got 8 billion images in it and 1.6 trillion text tokens. So those are just much bigger than anything that was previously available. So I would say you're probably about right now by thinking a trillion tokens is about right. Organizations like Meta do have access. And when we talk about LLaMA 3B, we'll see that that is roughly 100 million token count used for division model there. But for open source researchers, like the last few months have really seen the arrival of these huge interleaved datasets. Which has kind of really jumped the pre-training dataset size that's available.

Scaling's a hell of a drug.

Exactly. Okay. So we've looked at the QuEN-VL model and we've looked at the multi-part pre-training stage where we can use our lower quality dataset at the point where we're training the vision transformer and the cross-attention layer. And we don't want to muck around with a language model at that point. Because that's been heavily trained in the past. It's doing great things. We don't want to mess that up. So we don't want to use low-quality images and captions and be training that thing. So we just trained the vision transformer and the cross-encoder. It's when we come to the multitask pre-training, when we put a lot more attention and care into the dataset, that now we're going to allow ourselves to train the language model. And that, I think, was just in response to COVID, that there reflection you made earlier about, you know, why do this, or the question about why do this in multiple stages like this. Yeah, it's very, you know, going up the quality scale, down the size, and untreasing more parameters as we go. And that recipe just works really well for the vision language models. I guess the last one for now that I wanted to cover is the InternVL series from the OpenGV Lab. And this is a team from Shanghai University. In China. So one of the challenges that's facing particularly open-source researchers building vision-language models is the training cost, particularly when you have to take a large model, unfreeze the parameters, and fine-tune it or do continued pre-training because you're now introducing this extra modality. So in a series of 3 models, or 4 models actually, They've looked at some ways that they can improve the capabilities of the open-source language models. And one of their papers was, in fact, called How Far Are We to GPT-4V? Closing the Gap. So that was for their 1.5 model. And I'll just touch on each of them, kind of one after the other, because I think each of them introduces something really interesting to the picture and tells us something about where vision language models are going. So their kind of first VLM, which is called InternVL, they observed that everything today is— we've been using a vision transformer, a ViT, that's been relatively small, and it's also been pre-trained separately from the language model that it'll eventually connect it to. So if we started, for example, with a CLIP net, so one that had been used in the CLIP model, It's, of course, been aligned to a language model encoder, but it's a different language model to the one the decoder will eventually want to connect it to. So what they did is they started with a fresh vision transformer and they scaled it up. So theirs is 6 billion parameters, which is large for a vision transformer, right? So the ViT-H, ViT-Huge, has around 600 million parameters in it. And I think the biggest of the standard vision transformers is the ViT-G, and that's about 1.8 billion parameters. So this again is several times that size. And what they do is they conduct a fresh contrastive pre-training of the model. They use a frozen LLaMA 7B, like the original LLaMA, as a decoder. And what they do is they take, in the same way you would train a CLIP model, You take images and text pairs, and you start to feed both of them into each, you know, the text into the text model, the images into the vision transformer. You then take the hidden states from your LLaMA. You know, you can then take an average over that embedding. That's going to be how you embed the text. And then at the end of the vision transformer, you're going to take the states from that, and you're going to, for example, pool that somehow. And then exactly as you did with the CLIP model, you're going to do a contrastive pre-training step, right? So you're going to try and get the correct caption pairs to be similar, the others to be far apart. So not so different, like, to CLIP from several years back. It's just we're using a decoder now, and it's a decoder, it's a LLaMA 7B. So it's going to be much more similar to the language models we then want to connect it to in our VLM. And we've also just scaled up the size. The vision transformer. Okay. So that was their first innovation here. And what they showed is that this leads to a really high-quality and really well-aligned vision transformer. And what they could do is they can actually forget about the LLaMA model that they used for the contrastive pre-training, and they can connect it to a totally different language model when they build a VLM, and it works really, really well.

Okay.

The first innovation, just train the vision transformers from scratch, Using something much more similar to the language model you're eventually going to connect it to when you're doing the contrastive pre-training. Cool. That was the first intern VL. The second one, the 1.5 model, they looked at one of the, well, image resolution. That was their big question there. So we talked way back at the beginning about, you know, images don't just come in 224 by 224 squares, right? Or 448 by 448 squares. They come in many different kinds of resolution. Yeah. So how can we try and, you know, if we've got a higher resolution image, how do we try and get more out of it? So what they did is they developed this strategy they called Dynamic High Resolution. What this is, it takes an image, it could be of an arbitrarily high aspect ratio and resolution, and they segment it into tiles of a fixed size, 448 by 448. And the number of tiles that they use are going to be based on the aspect ratio and the resolution of the image. Okay, so they try and match the tiling configurations, 4x4, 4x2, 1x2. They try and match that to the image itself in its natural resolution. Okay, then I'm going to encode each of the patches from the larger image separately and then concatenate a thumbnail of the entire image to the end of the sequence. So now you've got your thumbnail and you've got all of your patches of high resolution, a great big long set of visual tokens. At that point, they've got too many visual tokens. And so they use something called a pixel shuffle, which is just a strategy for trying to compress that down. And the way they do that is, if you imagine you've got patch from a vision tensor of an image, it's got the width, the height, and the depth. One of the things you can do, obviously the number of tokens you're going to get out of that at the end is determined by the width and the height of the patch. So what they can do instead is they can just resize that. So in other words, there's more going into the depth dimension of the tensor, and then you're able to split it up into fewer patches. And that's just called the pixel shuffle strategy. And because this dynamic high resolution ends up with so many tokens, that was the way to squeeze it down. So you use the pixel shuffle strategy. And quite a few of the latest VLMs have done the same thing. So Quen 2 VL does the same thing, does now dynamic high resolution. But this is quite an interesting innovation. So now you can essentially process images at whatever the natural resolution is, simply by generating more tokens. And of course, if you're using a self-attention or decoder-only architecture, you don't worry about whether you're putting in 100 visual tokens or 1,000, except for the expense of having to unroll them in the decoder. There's no kind of hard— you don't need to have a perceivable sampler that somehow squishes them all down to the same dimensionality at the end. So that's kind of the innovation from the 1.5 model. Return to the L.

And, um, I was doing that with searching for something, and I just wanted to flag that while we don't know too much about how a GPT-4V works, we can look at pricing, and that gives us a pretty good indicator that it is probably very similar under the hood because you can go even just on the OpenAI pricing page. And click the low resolution checkbox. And then what you see is 75 tokens are your base number of tokens for any image. That's the minimum you can get. That would seem to correspond to the thumbnail in this, uh, last scheme. And then if you uncheck the low resolution box and start to increase the height and width, what you see is that it starts with one tile.

Mm-hmm.

And the max I believe a tile can be is like 512 by 512. Once you get over either the height or width gets over 512, even if the other dimensions stay small, now you're into 2 tiles and you can still continue to get charged those 75 base tokens. So it seems like they are probably doing something under the hood where it's like, yep, you're always going to have that full image as a single thing in low res. And then the question is how many higher resolution tiles are you going to have? That's going to depend on how big the image is that you feed us. And basically you see this exact scheme reflected in the OpenAI pricing structure. So interesting. Yeah, I wanted to— this is probably on me because I was a little bit confirming that as you were describing it, but can you tell me the pixel shuffle thing again? I didn't quite grok it.

Yeah. So the explanation I gave was the best to my knowledge, because I didn't delve into this too much. So, but basically you have a height and a width and a number of channels in like any representation of an image. And the height and the width determine the number of tokens you're going to get out at the end. And so the idea is, can I just reshape the tensor? Such that there's more in the depth channel, and then I'm going to end up with fewer tokens at the end. Hmm.

So in other words, if I go back to the OpenAI pricing thing, if I create a tall but narrow image, if a single tile can be up to 512 by 512, and I have a 513 by 150 image that can fit into that space, but in its natural orientation, it doesn't. And basically what you're describing here sounds like a way of sort of reshaping that so that it kind of fits into the space, which I would assume under the hood with data augmentation and kind of, I mean, there's a ton of that kind of stuff, right? Where sort of weird I mean, the, from the first, right, of just putting everything into the same 224 by 224, there's a long history of these sort of like programmatic manipulations of images to put it in some form, which might be quite strange from a human visual system perspective, but which ultimately probably makes the AI more robust. And in this case, also.

Yeah.

you know, demonstrates that you can save tokens, save money, save compute all at the same time.

Yeah.

I mean, as best I understand it, it's about just reshaping the tensors themselves that you've encoded. So you're actually shifting stuff from the X and Y dimension to the depth dimension. And I, it felt to me when I was reading about it, and it was like similar to what goes on in like a U-Net. You know, where you're actually just reshaping the tensors again and again, making them deeper and narrower, if you like. You're not actually losing information. You're just kind of adjusting where it is. So you come out with something very different, different dimensions, but you haven't actually, you know, if you multiply the dimensions out, it's still the same size. I guess the last of the intern VL models to look at is the latest one, which is 2.5. And so, you know, we've looked at how we use dynamic high resolution. So that was in 1.5. Looked at how we scale up the vision transformer itself in the original 1.0 model, and also do the contrastive pre-training using a much bigger language model. So the third thing that they looked at of note in the last 2.5 is using what they call a progressive scaling strategy. So the idea is to try and train efficiently on a large number of tokens. And what they figured out is, because they built several classes of model, they built a 7 billion parameter version, and then a mid-sized one, and then a 78 billion parameter one at the end. What they did is they aligned the vision transformer to the smaller language model backbone. So they do the training process with a vision transformer and a smaller language model. And then they swap the smaller one out and they introduce the bigger, the next grade up language model, and then continue the training. And then they swap that out and they bring in the biggest one and continue the pre-training. And what they find by doing this, so progressively increasing the size of the language model backbone during pre-training, is that they reach convergence quite early on in the training process with a smaller language model.

Okay.

And at that point, the vision model, the vision transformer has learned a lot of what it needs to learn in being aligned to this class of decoder language model. So they can swap that out and they can put the next bigger one in and they can continue the training. And it turns out it's much more efficient to do this than it is to start with a large language model, start with the large vision encoder and try to align them both at scale from scratch. And in the paper, they reported that this strategy of what they call progressive scaling used— they reckon they read 120 billion tokens during the pre-training phase of the 2.5 model. And that's their nearest peer competitor, which is the Quen-2VL language model, had to process about 1.4 trillion tokens to kind of reach the same perplexity loss. So they reckon it's a very, very much more efficient way to do it. And so there you go. That's kind of 3 strategies to use: dynamic high resolution to how we process the images, how we're training the much bigger vision transformer, and then this kind of progressive scaling strategy. And this seems to be a winning recipe because if we look at the MMMU leaderboard at the moment, the InternVL 2.5, the 78 billion parameter class is sitting just behind 0.01. And that means it's beating GPT-4o, the version from May this year, which is on the leaderboard, the original 3.5 Sonnets, and Gemini 1.5 Pro on the MMMU. So it's an extremely successful recipe. It also seems to beat them, by the way, on visual question answering and OCR bench as well, which is— 2 other important kind of benchmarks.

That data scale of data thing is really interesting because it's not just that it is saving compute by starting with a smaller model. It's actually also that it's using something like 1/12 of the actual dataset size. So you're saving on 2 dimensions.

Yeah.

Yeah, that's really interesting. That reminds me of an episode, which was one of my favorites that we did with a couple of guys from Microsoft on TinyLM, which was language models of this, I think of like tens of millions of parameters, like really small, trained exclusively on these like short kind of kid story type documents. And they looked at what Do these things learn and in what order? And it was kind of like you could see that they were learning things like parts of speech and, you know, really kind of structural elements of language first. And then gradually started to seem to have an understanding of like nouns and, you know, what was what, and then started to become coherent. And at sort of the far end of their process, they started to see what I remember as micro reasoning skills, which was like the farthest they pushed this. And again, these are very small models, but you could get to the point where it was like, Sally doesn't like soup, so Jimmy offered her blank, right? And the earlier models would just put soup in again because it was like, well, Soup appeared once, soup is probably likely again. And at the end of their training, they would start to see these micro reasoning skills where it was like, well, it's got to be something else besides soup, given the full context of what has come. I don't know that they looked at a progressive scaling from those really small models up, but do you have any intuition for why? I guess what I'm kind of struggling for, and maybe there is no good answer right now, but it seems like we're saying something here that is like the small model is actually learning faster. The small model is more sample efficient.

Yeah. So let's imagine what we're trying to teach the models here, right? So the same way you just described with the small language models. Okay. So there are progressively more complicated understanding and modes of reasoning that you can learn as training continues, right? And we can imagine that larger models can have more capacity to learn more of these things. But a larger model, of course, has many more free parameters during training. So if you have a lot of free parameters, it's going to take you longer to find the right kind of basins in your gradient descent that actually represents good capabilities, reasoning capabilities and understanding capabilities. So the idea here, and I think this is why this is working, is that if you start with a smaller language model, you start aligning your vision transformer to that, there is going to be some cap on the degree of complexity of the tasks it can undertake. But because you've got fewer free parameters in the training setup, you're going to find kind of good solutions sooner. And at that point, you can then use the larger of your 2 language models. You've now got more free parameters, but you've already found a reasonable place in the search space that you're starting from. And so you just continue the pre-training from there, right? Now you've got more capability, potential capability in the larger. connected model, but you're already starting from a good place, right? So you don't have to search as widely. So that would be my intuition as to why this progressive scaling strategy works.

Yeah. I mean, that is quite interesting. There was a recent claim out of one of the Chinese companies. I forget which one it was exactly, but they basically said that they had trained a roughly frontier class model at like single-digit compute requirements of what they believe the leading developers in the West, quote unquote, the West had, the big monolithic West had used. And this sort of thing, you know, could be a really interesting way that that is happening. So yeah, that's a striking data point. I mean, the idea that you can do it with under 10%, again, it's not just a That the parameters are fewer and you save compute that way, but it's a compounding savings because you're also using far fewer data points. This would probably be relative to, you know, if they did Quin 2 full size for 1.4 trillion tokens, and then this progressive thing only took 120 billion tokens, it would be like low to mid single digit compute. So. Yeah, that's definitely really, it's worth a ponder.

Yeah. So we've come now to the InternVL 2.5, which is the top open source model out there on the MMU benchmark. And you may have noticed that since we talked about Flamingo, like everything else we've talked about has been based on this autoregressive self-attention architecture. And so one thing you may be wondering is, okay, does that mean this cross-attention architecture is dead? And the answer is no, even though most teams have opted for this kind of autoregressive architecture, just to be different, maybe. When the Llama 3 Vision models were released earlier this year, that was based— the Llama 3V was based on the cross-attention model. So they used a vision transfer, ViT-H14, You know what that is now. It's a 600 million parameter vision transformer. But I think they also introduced some new cross-attention layers actually into the vision transformer itself. So they made some modifications to it. The report given, like the report for the LLaMA 3V model, goes into a bit of detail about some of the modifications were made, but doesn't give you Quite the same level of detail that you might get in some of the other papers we've covered. So we know that, for example, they modified the Vision Transformer. And we know also when they did, they used some very large pre-training datasets. They did the same things a lot of other teams did, adding kind of machine-generated OCR into it. A lot of effort spent improving the safety filtering, desensitizing, deduplicating, and quality filtering. The multimodal datasets that they use. So that was a real big thing there. They also did some synthetic augmentation of their pre-training datasets. So it's kind of the same way we've seen the QuenVL team do. So this is like adding synthetic captions, generating tabular data, generating LaTeX documents. So adding a lot more of this stuff in at scale into the pre-training. And then when they trained the model itself, the LLaMA 3 model, they added new cross-attention layers into it. And they then froze the rest of the LLaMA 3 model. So they only trained the cross-attention layers and the vision transformer. And that is both during kind of pre-training and during supervised fine-tuning. And they also did DPO at the end as well, which I think is the first time that I've come across. DPO being used in the vision language models. But it's interesting that I've also seen it again recently. So it seems to be more of a thing now. So one of the advantages, I guess, of this is that if you're just training the cross-attention layers and you're not doing a full fine-tune of the LLaMA 3 model, you're preserving all of the capabilities of LLaMA 3, and you're not risking degrading them by introducing the vision component. And this might be why they decided to do this with cross-attention rather than using the self-attention. If we're doing the autoregressive architecture, we've really just got that projection matrix to play with. And once we've trained that as best we can to do the alignment of the text and the images, if we want to keep improving the model, we've basically got to unfreeze the attention mechanism and train that in the decoder. where they didn't have to do this because they were using quite a large, like a 90 billion parameter Llama 3. 25% of those parameters roughly being introduced for their fresh cross-attention layers, training on those layers. That's a lot of free parameters. So they just did that when they trained the model. And the Llama 3.2, the 90 billion parameter version of that, is the second placed open source model. on VMM and Neo at the moment. So it really shows that, so just behind the Intone BL 2.5 model that we talked about. So it kind of just goes to show that it actually really doesn't matter. You can build frontier open-source vision language models using either of those 2 recipes. And that was what they showed.

Is there any...

Performance difference or any sort of practical, I mean, I hear you on the, like, and having seen just a tiny bit of Llama development from the inside, not really the inside, but I participated in a little safety review project for Llama 3. And I mean, it just, it was a lot of people moving a lot of different directions is kind of how I would summarize what was going on there. You know, it's amazing on some level that the whole thing comes together. I suppose that's probably always the case, but. Out of OpenAI, out of Anthropic, you see these sort of small, focused teams, or at least that's the perception from the outside. The internals of Meta felt like 10 different projects going on at once. So I could easily understand and interpret this as being a reflection of just how much more sprawling the organisation is and multiple different goals. And maybe also thinking, geez, not everybody wants or needs vision, you know, let's create that modularity for the open source community that's going to use this downstream as well. All those things I get. Is there anything that people should have in mind about the different architectures leading to different results, or does it really seem to be just kind of either way works? And as long as you do a good job, you really can't tell the difference after the fact.

Yeah. So there are some, there are some differences in that there are some ways in which the architectures lead to different results. Obviously, on the design side, introducing the cross-attention blocks adds more free parameters to the vision-language model than just introducing a simple projection matrix and going down the autoregressive route. So with a larger model like LLaMA 3, there are probably enough— I guess they figure there's enough free parameters introduced by the cross-attention blocks. That we can train our kind of vision-language modality alignment and get a really good result just by training on the newly introduced blocks. Whereas, as I've said before, like the downside of using the kind of self-attention architecture is that if you are, you know, if you've gotten the best alignment you can just out of whatever MLP you use to connect the vision tokens to the language model, then you've got to unfreeze the language model itself. And at that point, you've got to worry about degrading of the capabilities. So it could be that the decision was taken with LLaMA 3, like, it's a big model. We want to preserve all of the capabilities of our language model. So let's not muck around with the language model. Let's introduce these new cross-attention layers and let's just see If we can align it for vision-language tasks. And if we do so, is it good? And the answer is, yeah, it was really good, provided you put a lot of attention into cleaning and curating the dataset, which they did. And they used a lot of tokens as well, a trillion tokens or so in the training budget. Which, if you've got a lot of parameters to tune that you've just introduced, kind of makes sense. So the recipe works. There is some indication that the cross-attention model is not so good for things like OCR, and maybe also not for some forms of multimodal reasoning. For OCR, the reason for that might be because you need to introduce these— you've got these new cross-attention layers. And so you've got to use something a bit like a perceival resampler. Or something else to kind of fix the size of your vision tokens so that you can actually look them up. And for this reason, something like a Perceptron Resampler is doing a bit of shuffling of the visual tokens at quite a fine grain. And this might be why it's affecting OCR performance. So a few teams have noted this, kind of cross-attention architecture is not quite so good when it comes to these These tasks that require really fine-grained understanding of small areas of the documents. So that's one downside. I've also read some authors speculating that multimodal understanding, and so reasoning, is actually better in the decoder-only, the self-attention architecture as well. And this might— it's not really known why, but it seems that putting the vision tokens Into this, mixing them up in the same sequence with the text tokens in a decoder-only architecture. The attention mechanism just finds it easier to kind of reason over that. But again, nobody really knows why. These are just 2 kind of minor findings.

Interesting.

And I think it's probably enough to steer people in general over towards the kind of decoder-only self-retention architecture, which I think is probably going to be the one that wins out. Then there's caveats, which we'll come on to in a second, might not quite be the case.

The only other thing I can offer at the moment is I think one of the better AI podcast episodes of the year was from Latent Space where Swyx had Yi Tei previously and now once again of Google, but with a sort of middle period where he was involved with Rekha. Hope I'm saying that right. And his take was basically that some of these architectures are a legacy of the fact that they were originally different teams. And that, again, probably plays out much more at a Microsoft or a Meta or a Google where they had for a long time an organizational architecture or hierarchy or whatever that had people focusing on different modalities. before the great unification of all architectures showed up. And some of that organisation persists even when there now is a sort of unification of, or at least potential unification of the architecture. And so, yeah, you can maybe see some of these echoes where it's like the language model team is moving on to the next thing, and now you're kind of taking the baton on this. And this architecture is friendly to that. But It does seem hard to imagine, and I know we've got, you know, joint pre-training coming up not too, uh, much deeper into the agenda. It does seem like at some point the bitter lesson has to come for this, right? I mean, there's no, no escaping that forever, presumably.

Yeah, it does feel like, you know, the Llama 3D model does feel very different to everything else that's come out of Meta or FAIR in the last couple of years. Like you say, they're focusing a lot on early fusion architectures, which we'll talk about in a minute. This does feel like a bit of an odd one out, going for the cross-attention approach. And not just an odd one out given Meta's research, but a bit of an odd one out given all of the models that have come out over the last year or so, where they've really been tending towards the autoregressive architecture. It could just be a feature of wanting to preserve the performance of the Llama 3 on language modeling tasks, just by introducing cross-attention layers and not having to muck around with the model's already perfectly well fine-tuned self-attention mechanism. Yeah.

Okay, cool.

So we've covered a handful of what I think are some of the more significant models And by no means have we covered all the significant visual language models, but we've looked a bit at the importance of some of the things that have been learned along the way, the importance of interleaved data at scale, the importance of data augmentation in the pre-training mix, and then staging your pre-training data as well. So staging your pre-training. So you start simple, gradually unfreeze more and more parameters of the combined model. Adding in all this augmented data of high quality as you go. We've looked at that. We've looked at some of these mechanisms for processing high-resolution images, so tiling things and then adding a little thumbnail at the end. Looked at the progressive scaling, so that is aligning your Vision Transformer and your language model backbone with a smaller language model. And then once you reach and kind of plateauing in your training run, switching it out for a larger one and carrying on. And we've also looked at the importance of task diversity in instruction tuning, which was shown by the LLaVA team, and just how important that is to getting your vision assistant at the end to be able to complete a wide array of vision-language tasks. And where we're at now is there's this kind of these 2 architectures, this self-attention or autoregressive architecture. Where we're just injecting the token— the visual tokens directly into the decoder stream along with the text tokens. And then we've got this kind of cross-attention architecture where we're actually injecting fresh cross-attention layers into our language model, and then we're using those to kind of look at the encoded visual tokens. And a couple of recent research teams have tried to do a more systematic comparison comparison of these 2 architectures and said, okay, which is best? And one of them was a team from Hugging Face, and they've built a series of a couple of models, which they call IDEFICS, I-D-E-F-I-C-S. And although their 2 papers were building these models, really what they were doing, I think the team was doing, is exploring what makes vision language models work well.

Well.

And that was really the thrust of their research. So what they've done, what the Hugging Face team have done in their 2-paper series, is try to explore what happens if you take a particular decoder and you take a particular vision transformer, and then you try out both of the architectures, okay, and see what happens. Yeah, so you've got the same training data, So in other words, you're just keeping the experimental conditions the same and just changing how you connect the 2 and looking at what happens. And so some of the key findings from their research, well, first off is that if you freeze the language model and you only train the newly initialized parameters, so these would be the cross-attention layers, uh, or they would be the projection matrix if you're using the autoregressive architecture, um, then the cross-attention architecture works a lot better, like gives you better results. That's perhaps not surprising since if you bring in some new cross-attention layers, you've got more parameters there to play with than if you just have simple projection in the autoregressive architecture. But then they say, so when they try to update the language model backbone, the autoregressive architecture can perform much better. So one of the things they noted in, I think it was the second of their papers, is if you try to do a full update of the attention mechanism in the language models, they discovered they had training instability. So it's really, really hard to get the training to work. So they just switched to using low-rank adapters and there's no issues. They were able to update the language model attention mechanisms. And in that case, The autoregressive architecture was performing much better. So that's interesting. Perhaps unsurprisingly, if they increase the size of the vision transformer or they increase the size of the language model, both of those lead to a better vision language model. But if you had to pick one for a fixed parameter count, they say you get more bang for your buck by increasing the size of the language model component. Than you do increasing the size of the Vision Transformer. So there you go. One other finding, and we have kind of referenced this a lot, if you do add a perceivable resampler, which you can add in both architectures, something like that is necessary in the cross-attention architecture because you've got to fix the size of the lookup vectors that you're performing cross-attention upon. But you can also have something like that in the autoregressive architecture. Architecture. And the reason you might want to do that is if you wanted to reduce the number of vision tokens that were actually being unrolled in the decoder. So you could do it for both. And they obviously find that if you introduce a perceiver resampler, it does speed up training, but it doesn't necessarily have a positive impact on, um, performance in the end. So that's interesting as well. And finally, They show that if you use the interleaved image-text documents, which the Flamingo team and other teams have found to be super important, they performed an ablation where they left those interleaved documents out, and they found it had a dramatic effect on performance of the model at the end. And that in particular, adding interleaved image-text documents like this seems to really benefit few-shot learning at the end. Those are some of the conclusions from their i-Defect series of papers. And they're not the only ones to have a look at this. There's a recent model called NVLM that came out from NVIDIA a few months ago. And there, what they did exactly the same thing. So they trained 3 different variants of vision language models. Well, actually 2 initially, 2 variants initially. They used a common backbone, so they used the language model from Quen. So that's the same one that's used in the Quen VL series. And the vision encoder, they used the intern vision encoder. So we talked about that earlier. That's, um, the vision encoder at 6 billion parameters, so entirely new, kind of trained from the ground up along with a decoder. So they trained this at a larger scale with a decoder transformer to see if they can get better performance from the vision transformer. So those are the 2 components that are used by NVIDIA. So they denote the 2 architectures D for the decoder-only version and X for the cross-attention version. And they compare them both. And they discover after training for a certain number of FLOPS that the decoder-only version has got the best multimodal understanding the best reasoning, reasoning over images, and the best OCR performance. But they also note that the cross-attention version was much more efficient to train. The reason for that, and we've kind of mentioned this earlier, is because you have to unroll the full sequence of image tokens in your decoder and apply the attention mechanism across all of them if you're training autoregressively. So it's perhaps unsurprisingly that you get lower training throughput. So that's interesting. They report that the Perceiva resampler affects OCR performance, so negatively, negatively affects it.

Hmm.

And the thinking here is the Perceiva resampler, it's the way it does resampling, it's probably shuffling some of the information in the fine-grained kind of spatial dimension. And that does seem to be hurting tasks like OCR, which require a very kind of high-resolution view of the image. So that's their hypothesis to what's going on there. So I mentioned they did 3 architectures, and this is because the NVIDIA team then looked at, well, what if we did a hybrid of the 2? So what if we had cross-attention and self-attention? So the idea here is trying to make it more efficient to train the model, and that is by removing all of these kind of high-resolution image tokens from needing to be unrolled in the decoder. So all of the high-resolution image tokens are now going to be presented to the model through the cross-attention mechanism, but they take the thumbnail. So remember, if we're dealing with high-resolution images, we typically tile the image, encode all the tiles to make the stream of tokens, but then also include a thumbnail, an overall view of the image as well. So they do inject that into the decoder stream. So this is their hybrid architecture. So it gives the self-attention mechanism of the language model a copy of the image to kind of reason over. So remember this finding that the decoder-only architectures seem to have better multimodal understanding and reasoning. Seems to be something about co-locating the image tokens and the text tokens in the decoder stream. That works really well. So it's still got that, but the decoder is then also able to look up via cross-attention the high-resolution tiles from the image when it needs to. So that's kind of— you might think of that as a compromise because it's clearly more efficient to train. And when I look through the results, you can see that the decoder-only version, it does beat the hybrid version on OCR tasks still, but the gap is smaller, as you might expect, than between the cross-attention and the decoder version. It beats it on chart understanding slightly. But what's really interesting is that the hybrid version actually beats both the decoder-only version and the cross-attention version on the MMMU validation set. Only slightly, but it does beat them both. So I think this is really interesting. kind of approach, this hybrid architecture. And it wouldn't surprise me if we saw this explored more in the future.

Is there any way in which it's strictly better? I guess I'm understanding that my super high-level summary of all of that would be, it seems like the decoder approach where the images go in right alongside the text at the beginning and get the full workup is the best performing. The cross-tension is a lot more efficient, but has some relative weaknesses and some that are particularly idiosyncratic, but it's easier to do. Hybrid tries to get the best of both worlds, but if I'm like OpenAI or DeepMind, and I'm like trying to make the best thing I can make. Is there any argument or result here that would suggest that this hybrid approach has a claim on, you know, in any way being the best thing you could make, or is it only appealing because it has the efficiency advantages?

Uh, I would say it's too early to tell. So The decoder-only architecture that NVIDIA put together, the D model, does win over the other 2 variants on, for example, chart understanding and OCR tasks. But on the validation split of the MMMU benchmark, the hybrid version beats both the cross-attention and the decoder-only version by about a percentage point, which is interesting. Right now, there aren't any other hybrid models out there that I'm aware of. So it's unclear yet whether this is generally a better approach, but in this particular instance, on that one benchmark, that was their best performing model. And indeed, right now, if you look at the MMMU benchmark, you'll find that the, we've got the intern VLM. model, which is the top open-source model. We've got the LLaMA 3.2 Vision model sitting right underneath it. And you'll find this on the MMMU leaderboard. You'll find the NVLM hybrid architecture sitting just below those 2 in terms of the open-source models. So I think it's a really interesting direction. From my perspective, the jury's still out as to how else you could vary this architecture. what might be the best. So no convincing evidence that decoder-only is hands down the best way to go. I think this is an interesting data point suggesting maybe there's more to the story.

Yeah. Interesting.

It does. I'm always amazed by how simple the transformer is. That's just a recurring point of amazement where I'm like, the Tangled mess that is my own brain and all the feedback loops and everything that basically you can get as far as we've got with none of that. So it, I guess, you know, I, on one hand have sort of a strong prior that that'll just keep being the best cuz it's been the best for a while. But then there's another part of me that's like, surely a more complicated architecture can work better or we would've presumably evolved in a simpler direction ourselves.

Yeah.

So, yeah, I don't know which one of those should dominate, but presumably the more complicated— I mean, the more I think about it, it seems like the more complicated architecture, just given how many possible versions of that there are, not to say this specific thing that has been tried here, but in general, it seems like more complicated architectures have to be better in some way, shape, or form. You got to find them though, and you got to make them performant on the compute. So those are huge advantages. Obviously, that the current remarkably simple architectures have. But yeah, that's cool. One other thing I was laughing at, I think you noticed at the very beginning of this section, was the paper from Hugging Face was called What Matters When Building Vision Language Models. And I would submit naming does matter somewhat. They came up with I-D-E-F-I-C-S.

Yep.

1 and 2, as you had to spell it out. What is that? I-Defects? I do. I, whatever, whatever that is, it's not popping off the page to me. And I think it's gonna have a hard time in the jungle of models out there of standing out without a little bit of a catchier name. So that would be, as a reviewer who never actually read the full paper, I would say at the title, you know, there, there's a question posed that they could have perhaps answered a little bit better with a better named model, but that's just a funny—

Yeah.

Reality across all of AI right now. Everybody, I feel like the O3 skipping O2 is just in some sense, like totally insane. And in some sense is kind of perfect for the moment that we're in. So they're certainly in, you know, good company for naming their models in a strange way. Yeah.

I mean, there may also be trademarking issues with O2.

Yeah, I heard that. Who's Who's got O2? I guess maybe they can't trademark it just because O2 is such a common thing. Is that the idea?

There's a mobile company called O2.

Oh, interesting.

Which could, could be, could be one of the reasons.

I, I was totally unaware of that. I was thinking that maybe because it's oxygen, you can't, uh, trademark something so commonplace, but I guess it maybe is running the other way. In any event, it, it is hilarious that they just introduced O1 and now we're on O3. And there is no O2. But yeah, somehow it does feel appropriate at the same time.

Yeah. So there's 2 other things I wanted to pull out from the NVIDIA paper in ascending order of interest for me. The first is a really interesting result halfway through where they looked at some of the leading open-source multimodal language models, the VLMs really. One was a LLaVA model, One's an intern VL model. And they noticed that if you apply text-only benchmarks, if you run text-only benchmarks, I don't remember which benchmarks they actually ran against the models, but you will— oh yeah, no, they ran MMLU, MATH, HumanEval, a couple of others. They found that in the VLM models, there had been a drop in text-only benchmark performance. As compared to the language model backbone originally. So for example, in the LLaVA-1 Vision model, whichever language model decoder they used originally, they knew what the benchmarks were across all these scores. And then they just repeated it with the vision language model and they found a drop. And they found this drop consistently amongst all the ones they tested, except for the LLaMA-3. Series. And if you'll recall, when we mentioned the LLaMA 3V series, they introduced all these cross-attention layers and freshly trained them, but they froze the rest of the LLaMA language model. So they didn't suffer that degrading. However, to kind of counter this, the NVIDIA team spent a long time building what they consider to be a really high-quality text-only supervised fine-tuning dataset. And as a consequence of this, what they saw was an improvement on all of the language-only benchmarks from the NVLM series of models. That is, as compared to what the decoder they used was scoring before. So that's very interesting, right? An improvement in the text-only performance after vision-language training. Possibly this could be down to just seeing the, you know, there's an additional large training data, training, pre-training dataset it's seen that presumably it hasn't seen before. But, you know, one does wonder whether there is something about the interleaving of the 2 different modalities now which is somehow causing the transformer or the overall model to be able to reason better. So I think that's a very interesting finding, and it will kind of be backed up by the fact that they saw this improvement particularly in maths. And they note that in their multimodal fine-tuning dataset, they had an awful lot of kind of maths questions in there, like geometry questions, for example. And they think it improved the model's ability to do numerical and mathematical reasoning, even in text-only form, text-only benchmarks. Even though this additional training data came in the form of images. So I think that's a very interesting finding.

Yeah, let me make sure I have that clear because there's 2 things there. One is it usually, the text-only performance usually degrades if you just do image datasets without maintaining text-only data in the mix. So they did that. And then in this finding where they see an improvement in the text-only math encoding benchmarks, that is the same thing where they are continuing to have some text-only data in that mix.

Yeah.

So the way to think about this is if you perform supervised fine-tuning for a vision language model, you'll see a degradation in text-only tasks. And we know this from large language models themselves. You know, if you do an additional round of fine-tuning on a particular task and you want to preserve the capabilities of other tasks, you have to mix in fine-tuning data just to preserve that. They're not the first team, and this is not the first model where text-only data has been included in the supervised fine-tuning, but they seem to really go to town on this and draw it out in their paper as something they really paid attention to. Having a large text-only fine-tuning dataset. As we mentioned, the LLaMA 3V models saw no degradation, but then they froze the language model backbone. And the NVLM models actually saw an increase in their text-only performance. And they saw this particularly on the maths datasets. And so their conclusion is the introduction of mathematical questions in image format has improved the model's mathematical reasoning overall, with a result that on text-only maths questions, it now gets higher scores than it did before, than the language model backbone did before.

So I think this kind of recalls the legendary Rune tweet of shape rotators and word cells. It's like the language model has, with the addition of the vision modality, has perhaps gained a shape rotator capability that it didn't have. when it was text only.

Yeah, I, I—

It certainly makes sense. I mean, if you had never seen a drawing of a triangle and only, you know, and I'm kind of imagining your SAT type math problems here. If you never saw that, you know, any of those diagrams and were forced to do it purely through text tokens, that would be weird. You know, there, I think there's a clear reason that we do that for ourselves. So it is interesting to see. a similar pattern popping up here.

Yeah, it's, it's perhaps not surprising. It's just really interesting to see it actually kind of verified in the research. And it brings me back to one of the questions that we posed at the very start, which was, you know, to what degree is multimodal understanding important for achieving the levels of intelligence, let's say AGI, that we want in the future? Well, here to me is like a small data point that is suggesting, look, there are real benefits to doing this. It doesn't say it's necessary. It does say it helps. So I think that's the interesting implication of the result. Yeah.

Yeah. That's definitely worth a little meditation. I mean, you can imagine that that can go a lot of different directions. That seems like it will be a huge trend. I mean, we're already seeing, of course, like more modalities beyond just image being added, video, audio, et cetera, et cetera. I've often wondered just how far that can go. Like, are we going to see natural language models that also get trained on biological sequence data? Because that's a whole other track that I've been quite obsessed with recently, where there's been all this training on sequence data. And in a way, it's really cool that there's not natural language there because I think it sheds really interesting light on how things are working when it's like it's picking this up from, you know, all the, all the things that it's learning about proteins and so on. It's, it's learning those from raw data in a way that is not mediated by human understanding in a lot of those scenarios. And I think that for me has been extremely clarifying when it comes to questions of Can these things learn new concepts on their own that humans don't know? It's very hard to determine that through natural language models because everything's kind of out there somewhere. And, you know, what's interpolation versus genuine out-of-distribution, whatever.

Right.

But when you see these higher-order concepts emerging from pure DNA sequences or pure, you know, amino acid sequences, It seems like, okay, there's something really there that is pretty undeniable. And then I wonder, do all these things sort of converge? I mean, it seems like the global maximum probably is a model that— and this may not be economical. There may be all sorts of reasons why it doesn't happen in the immediate future, but it seems like the global maximum has got to be something that is just literally trained on everything and has the text and the image and all the way out to weather data and just has this very robust sort of all-to-all understanding. And yeah, this is just like one little data point that suggests that that's true. But I feel like big picture, it's hard to see how that wouldn't ultimately be the case.

Yeah. Part of the story, we kind of said at the beginning that vision language models are a really interesting route into understanding how you actually exploit the relationships between 2 different modalities to improve reasoning. And some of the story we've told here is about what has been done over the last couple of years to get better and better at doing this. And I don't think we're at the end of the road yet. We've seen some interesting ideas so far, particularly this hybrid architecture from NVIDIA, which suggests there may be more droplets you can squeeze from the lemon. in terms of getting kind of more efficient cross-transfer of information between the modalities. So it's really interesting, and it's been a tremendous amount of progress in 2 years in this one area.

Yeah, this cross-attention thing would also be probably the way that it happens if you were going to try to integrate a protein language model or whatever with a natural language model. It seems like You could do the end-to-end thing, but for starters, you would probably grab 2 that work and try to make them talk to each other somewhere in the middle layers where all the representations are already there and you have a lot of confidence that you're working with things that have their own understanding respectively and just trying to bridge those in a useful way. Especially, yeah, I can imagine too, as you have lots of these modalities, it'd be an interesting question to try to figure out, even if the— and it's not even really that the end-to-end thing is strictly best because this hybrid one is really competitive. But I can also see just lots of reasons that you might, for convenience, right? And we talked a little bit earlier about how to some degree these architectures are legacies of team structures. And if you break that out now over lots of different modalities and you've got a whole different universe of people working on biological models, then it might be really hard to redo everything from scratch or get the data mix right, or all those sorts of things could be really hard. But if you have things that are working specialists across these domains, then I could see the cross-attention approach being a really natural way to bridge them without having to kind of go back to the drawing board as much.

Yeah. So a term we haven't introduced yet in this conversation, but which we could introduce now, is early fusion versus late fusion. And so what this is describing is at what point in your information processing architecture do the modalities come together? And in everything we've discussed so far, The answer is, well, we've got 2 separate encoders, right? We've got a vision transformer and we've got, I guess, a tokenizer for the language model. And then we're kind of— so we're encoding the 2 modalities separately and then we're fusing them in the architecture. But we'll discuss something in a second where we're looking at early fusion, which is, okay, can we get one thing to encode them both such that the modalities are aligned right at the start of their journey? Through the model. And I don't want to speak out of turn because I'm not a neuroscientist, but my bet would be if you asked a neuroscientist, how does the brain work, the answer would be there are early fusion and late fusion and probably multiple integration points for different modalities of data through the information processing pathway. And it may well be that the situation that AI eventually finds itself in is that you similarly have these kind of multiple points of fusion. We've already seen with the NVLM hybrid architecture, you know, we've got 2 points of fusion there. We've got a cross-modal cross-attention lookup, and we've got an image thumbnail being added to the decoder stream. So we've still already seen the first example of that. So one of the trends I would expect to see is that we find, you know, we have early and late fusion going on in future architectures, but I'm definitely not smart enough to say what they look like. Yeah.

When, when people talk about early and late fusion, to what degree is this a statement about the pre-training process versus the architecture? I've always been a little bit confused about that because there's sort of, you know, in the hybrid, the early fusion and the later fusion are still with like Separately trained, separately pre-trained modules, right, that are kind of learning each modality. And then it's like there's a question of, do I want to inject that image data at the beginning of the transformer or do the cross-attention thing in the middle? But then I don't know if it's like, feels like you could sort of make a 2 by 2 matrix of these, perhaps, where you could have, like, I don't know about the 4th box of that matrix, but joint pre-training is another thing. And I'm kind of trying to untangle that concept from the early and late fusion.

Yeah. So first off, I don't think there is a settled and canonical split in architectures where you can say late fusion means this, early fusion means this. Let's think of it more as a continuum. I would say everything we've looked at so far is late fusion, and that includes CLIP, because we are encoding both modalities separately and then aligning them. And so for that reason, they are later than what you could consider very, very early fusion, which would be we have one thing that encodes both our text and our vision at the same time. So we have one token space, if you like, right from the get-go.

Right.

And we will see an example of that before we finish today.

Okay, cool. Well, let's keep rolling.

Yeah. So I just wanted to say a couple of words on this before we move on to a really interesting benchmark, on a couple of the benchmarks that are very important at the moment that are also have been turned into, or are being used, the training splits in these benchmarks are being used as part of the fine-tuning datasets in lots of the models we see. And so the first one is the VQA benchmark. This is a reasonable-sized dataset. I think there's 250,000 images in it overall. I don't know what the train-validation split is, but there were images taken from COCO, so that's Common Objects in Context, and possibly from other sources too. But the idea is that each image is associated with an open-ended question about the image, which was generated by Amazon Mechanical Turk workers. And each question is supposed to require an understanding both of the vision, the language, and some common sense answers. And then you're given some, you know, if you're executing the benchmark, You're given multiple questions per image and multiple answer options per image. So they reckon there's about a million questions in the dataset overall. This has become very important, as I said, as part of the fine-tuning split. And so just to give you, kind of put this in context, let's get an example from the VQA dataset. So a photo of a woman who has got a pair of bananas drooping down from her upper lip like a mustache. And one of the questions is, what color are her eyes? And so I guess the expectation is that the VLM or the vision model is going to lock onto the yellow in the middle of the screen and answer yellow, which is not the right answer. But then it also asks, what is the mustache made of? So again, this is requiring it to know where a mustache sits on the face, what kind of shape a mustache has, and then what objects are performing that role. In this image. So that's kind of an example from the VQA dataset. So common sense and reasoning over images, very different notes to what's in MMMU. So if one was looking for reasoning over images, as you know, that's the thing you wanted out of your VLM, the MMMU benchmark is great. If what you were looking for was kind of an understanding of common objects, their relationships, you know, what's going on in an image, the VQA benchmark might be what you're looking at, the Visual Question Answering. There is also a kind of variant of this called the DocVQA benchmark. Again, it's got a large training set, so it's often found in the fine-tuning mix. That's about 50,000 questions that have been generated over about 12,000 images, and these have been extracted from a dataset described as industry documents. Okay, so this is things like PDF scans of documents from the workplaces, charts, graphs, like tables of data, invoices, business infographics, and handwritten notes, that kind of thing. And the tasks in the benchmark are to isolate and report kind of precise spans of text from the images. That answer a question. So for example, it could be, what is the number on this invoice? And then you're followed by a PDF scan of an invoice. Okay, so this is an interesting benchmark, and because this is the kind of thing that a lot of people want to use vision language models for, right, is processing scans of documents. And if that's the use case that one really cares about, then performance on the Doc VQA benchmark is the one to look at. And just as a final word on fine-tuning and instruction-tuning datasets, the Hugging Face team of the unpronounceable IDFX model fame have bundled 50 of these fine-tuning datasets up together.

Much better name this time.

Yeah, they've called it the Cauldron, which is a great name. So the Cauldron is obviously available on their platform. Form, and it's probably the easiest way to acquire a good fine-tuning dataset. And the reason I mention this is because we've talked earlier many times about how augmentation of images in datasets has been really key for actually learning the alignment between the modalities. If I was starting with a task which was going to require a vision-language model and I was struggling a little bit to get the performance I wanted, one of the things I would do is look at the Cauldron. I would find a task that seems to be similar to the one that I'm doing, and I would look at the augmentations and the prompt structure from that particular task dataset. And I would ask, is there any way I can do augmentations on my own images, or can I restructure my prompts such that it looks like this dataset? Because the— whether it's just at inference time or whether you're going to build your own SFT dataset, this would probably be the most informative and useful way to go about it. So, that's a cheat code, in my view.

Yeah, that's cool.

It's a great, good tip.

Shall we linger on the BLINK benchmark for a minute? Because this is fascinating. So, we talked about MMMU. I've just mentioned the VQA and DocVRA WeQA benchmarks, which I think those are super important. BLINK is a really interesting one that was produced by a mixed academic and team from Allen AI earlier this year, July, I think. So BLINK contains about just under 4,000 multiple choice questions, and it ranges over 14 common perceptual tasks. And these are tasks that they say humans can solve in a blink, but which should be difficult for VLMs. Across these 14 tasks, like, human performance is in the kind of mid-90%, and because they're all multiple choice questions, random guessing gets you just over a third. All right, the thinking behind the BLINK benchmark is that lots of the questions on MMMU are actually about reasoning, and the authors describe them as they almost reduce to a dense captioning task. Can you just extract a dense description of what you see in the image here? Another way of saying that is, if you were to replace the image with a rich description of what's in the image, a language model should still be able to answer a lot of the questions. And indeed, when we mentioned MMMU, that was actually one of the baselines that the MMMU team created was to caption with a LLaVA model. And then they saw that you could do much better than random guessing just from the captioning. So the interpretation of this that the BLINK team made is that a lot of what MMMU is testing is reasoning, and that less emphasis is being placed on the classic visual perception capabilities. And one piece of evidence for this could be that if you look at the jump in MMMU performance between GPT-4o and GPT-4o1, there's been a huge jump. And if that's all attributable to reasoning, it's suggesting that, you know, that is a lot of what the benchmark is measuring. So probably should talk a little bit about, or like introduce some of these categories, you know, what, what do the, the BLINK authors think is meant by perceptual reasoning? So I'll just cover a few of the categories. First, a couple that the VLM seems to do really well on, and then a few where this seems to do really poorly. Okay. With me so far?

Yeah, I like this because this definitely calls to mind some of the challenges that we've had with aesthetics, which I mentioned earlier as well. But, you know, again, in the Waymark context, it's not enough just to know the content of an image. We want to make our users look good. And the early versions of this were basically, first they were not dense captions. They were very sort of not very informative captions. You know, you'd get like in the earliest version that there, there was once upon a time when the Microsoft API was the best.

Mm-hmm.

And then Blip became the best for us for captioning, but it was still, you know, this is too years ago, back to the beginning of the outline, uh, you know, so sparse. Then we got these denser captioners. Those were much better for at least getting the relevant images content-wise, but still no signal for a while on what actually looks good.

Mm-hmm.

And that has definitely notably changed. And I think these subtasks within this BLINK dataset that you're you're gonna take us through are a really interesting way to interrogate how exactly they have changed and kind of reverse engineered to some extent too, like what the developers have done to add these capabilities when you look at these challenges. So, yeah, I think it's a very interesting kind of look behind the curtain.

Yeah. Well, if we peel back the curtain then on the The best solved task, BLINK task, is what they call the art style.

Okay.

At the time of recording, the best data that I can find is that the model that solves this the best is GPT-4o. Okay. We don't have, I think, 4o1 results in BLINK. Neither do we have the latest Gemini or Sonnet, I believe. So take this with a little pinch of salt when I say what was the best, but it gives you an understanding of, you know, as of June this year, what did the rankings look like? So the art task, the idea is you've got 3 paintings, for example, right? That you've got one's from a prompt, which is a sketch, and then you've got maybe 2 others from 2 other art styles. And the question is, which of the 2, numbers 2 and 3, seem to match the image in the prompt in terms of visual style? Okay. And for a human, this is very easy. Like, one can look at this and say, oh, this is— you've input a sketch. This seems to be some kind of Renaissance painting, the second one. And then the third one, I mean, I'm not quite sure what the style is, but it looks a bit like a sketch to me. So I'm going to say it's the second one. And I know nothing about art, and yet I can solve that very quickly. Fluoro does really well on this. It gets about 83%. And is the top-scoring model. Humans are at 95%, and random guessing would give you 50%. Okay, so art style seems to be solved, I would say, reasonably well by the current generation of LLMs. Again, another similar one is, is what they call visual similarity, which is actually solved rather well. So here you might get a series of 2 photographs and then a reference photo, and the question is, you know, which of the 2 photos is most similar. And in the paper, they show 2 waterfalls and then another waterfall. And one of them, the reference image, is taken from the same perspective as one of the images, and the other image of waterfall is taken from a different perspective. So I immediately pair the correct one in my head. Humans indeed get close to 97% on this. And here, GPT-4 Turbo was the winning model at 80%. So those are 2 of the well-solved What I think is really interesting is the worst solved blink tasks. So of these 14 categories, which ones seem to have the biggest gap at the moment? One of those is the IQ test. So if you've seen the kind of images you get in an IQ test, you know, the the example given in in the blink paper is one where you've got a simple diagram, you know, with a sequence of shapes in it, and then you're being asked which. Sequence seems to match this, you know. So I've got 3, 3 in a sequence, and then can you complete the next sequence? Uh, pick an image of these 4 which seems to complete it in the same way the first sequence is completed. Um, so that's— if you've seen IQ tests, you've seen lots of variants on this before. Humans get this right roughly 4/5 of the time. Okay, random guessing in this particular area of the benchmark is 4 options, so 25 And GPT-4 Turbo was the best performing model with 32.67%. So not a lot greater than random guessing. When I look at this blink task, I am reminded a lot of the Arc AGI questions. And I don't know if that's a lazy analogy or an ill-informed analogy on my part, but I seem to solve Arc AGI challenges in much the same way I solve these IQ tests. Am I doing, you know, guided program search based on perceptual priors? Because that kind of makes sense as an explanation as to why I can do it in a blink and the language model is struggling. So I think it's a really interesting one. And I think, you know, my interpretation is that this really backs up a lot of what François Chollet says in how he designed the ArcAGI benchmark. And why he thinks it's difficult.

Yeah, maybe just 2 interjections from my experience. One on the, even the well-solved tasks, I would say if you're actually gonna do something like this in an app, the way you ask definitely still matters, if only for avoiding unnecessary refusals. Like I, I found that GPT-4o, And Claude, you know, 3.5 probably similarly really doesn't want to tell you you're ugly. So, and that's, you know, presumably reflection of its reinforcement learning from human feedback. And it's, you know, Claude's virtue ethics, right? It's not, it's trying to be a good friend to you. So it doesn't, it sugarcoats sometimes if you say like, you know, is this image an attractive image, for example, it will sort of often hedge and be like, Well, it's sort of in the eye of the beholder. Um, so there's a couple different ways we've found to get around that. Kind of similar in some ways to like, you know, if you want a model to talk to you about a medical question and you're like, you know, getting the sort of, I'm not your doctor. I always get around that by saying, I'm preparing for a conversation with my doctor and I want to be as informed as possible. And then it will basically let its guard down and help you. And in these like image aesthetics questions, we found if we just said the, the tersest, you know, rate this image 1 to 5 beauty or whatever, it will sometimes balk at it. But if we say, is this an image that a small business would be proud to put forward in their marketing, then you are much, uh, better able to get a result. And I've also seen kind of similar things with even just pairwise type things. So, you know, I took a couple selfies of myself and my kid, and then, you know, one of them I made look kind of a whatever contorted face that was like clearly not a great picture. The other one was much more normal. And just saying like, is this a good picture? It really doesn't wanna shoot you super straight. But if I put both in and I said, which of these should I send to my wife?

Mm-hmm.

Then it will say, you know, I think you're better off with the second one or whatever. So play around with that stuff if you are trying to get it to work. Like everything else, you know, prompt engineering is definitely still a bit of a thing, especially if you're kind of asking it to be a judge. You know, it's comfortable judging in some ways, and it's really not comfortable judging in other ways, not to anthropomorphise it too much, but that intuition definitely can be helpful. On the ArcAGI thing, I was also just really struck, you know, back when that was dominating the discourse not too long ago, by how weak the, even the best models were when it came to simply describing the ARC images. So just to, you know, took a few screenshots right off their website, but then, man, I didn't ask it to solve the problem. My first thing was just, can you see what this is? Can you describe it? You know, and it was very not good at that actually.

Yep.

Not good at even the most basic stuff of like how many squares, what is the dimension of the grid? I was really amazed by how weak that was. And presumably that just reflects kind of a lack of that sort of thing in the training data. But it's still pretty surprising because it's pretty good at reading tables and they can do OCR reasonably well. And you would think that it would be able to count the squares, but That was really weak. It was interesting to see too, just in the last couple of days with the O3 results, that as far as I know, they weren't doing any image and literally just presenting, which is the actual underlying data beneath Arc is literally just arrays of arrays, right? I mean, they literally just give you numbers and all of the sort of colour representations that we've seen On top of that are basically a gloss for humans to make it easier for us. 'Cause we're obviously good at colors and stuff and recognizing these shapes when they're like in contrasting colors that plays to our strengths. You don't have to do that, you know, for the Arc thing. And it turns out that O3 seems to not be doing that and is literally just reasoning over text tokens. So, yeah. And yeah, interesting. In terms of what's your theory, would you, would you have a, any other sort of deeper theory on why these IQ tests are so poorly solved?

I am not smart enough to be a theoretician, but I just noticed, like, to me, it just reminds me—

Speculation. Let's call it a speculation. You're smart enough to be a speculator.

Reminds me so much of RKGI. The whole point behind the Blink benchmark is that human Humans can solve this in a snap, right? So there's something about the perceptual priors or the perceptual features that we extract as a natural part of looking at the image that means we can very quickly answer these IQ test questions. And it seems that understanding of images in VLMs just doesn't work the same way.

Yeah.

So maybe they haven't got the same kind of perceptual scaffolding that we do. And therefore, I can answer it very, very quickly because I don't need to do an exhaustive program search. I've got some priors which can help me zoom in on the correct rotation, if you like, of the image in the IQ challenge, therefore select the right answer. Otherwise, I have to do some kind of exhaustive reasoning over the possibilities. As, you know, Chollet would say, a program search. So, you know, the job of priors in this case is to constrain your search space, right? So you zoom in on the correct answer very, very quickly. So my interpretation would be, yeah, that they're probably not understanding the kind of perceptual elements of the image in the same way that you or I would. And that doesn't mean that you can't do very well on a benchmark like this just by adding reasoning. And my guess would be that, you know, 4.0, 4.0.3, given a lot of compute time, would do a lot better on this than, than 4.0, which is what has been measured. And simply because it can reason over the different possibilities, that would be a guess. But then again, you know, whereas with the RKGI benchmark, you're presented with essentially a matrix, you know, an array, that's the input actually comes in an as an array. And so you can kind of reason over that as a series of text tokens, whereas the VLM has got to process the image in the BLINK challenge. And so it's not quite clear that you get such a clean decomposition of the IQ test image into tokens which you can manipulate through reasoning in the same way. But I'm going to guess that reasoning plays a role in solving this and that you can essentially like brute force your way through problems in the BLINK dataset, perhaps by adding reasoning, but it just doesn't feel like the most efficient way to do it. And I think that's probably the point that the authors are trying to demonstrate. Yeah.

And I mean, you're not given any, in this case, you're given only the image, right? So that's a notable difference from the ArcGI where you are given something that is text token presentable, and you have the option to use images too, but you don't have to do that. Here, you have to confront the image and make sense of it. Is there anything in this recent line of work? I mean, again, I don't know a lot about the human visual system, and I also don't know a ton about old convolutional networks. But my general sense of the human system is that we sort of gradually, as the information is being more and more processed through layers of the brain, that we're going from very simple edge detection type things and angle detection and all these sort of features, basically, up to more and more semantic features. as we get to higher levels. And that definitely has been demonstrated in certain vision systems in the past.

They're labelled that way in the human brain.

Do you think that there's something about this sort of vision transformer that's not doing that? I haven't seen— seems like that work was a little older, and Obviously the paradigm has shifted and convolutional networks also can have much more engineered features where you can put these kind of specific priors into the structure itself of the convolutional, whatever, processing, for lack of the right precise term.

That's absolutely right. And it was pointed out, in fact, by the IDFX That's, you know, much more work has been put into looking at language models than has been put into looking at the actual architecture of vision transformers. And you're actually right. You know, one of the things about a vision transformer is that those inductive priors that we used to code into CNNs are missing. And in fact, that's why at smaller scales, probably, CNNs actually do better than— and this was when we first introduced the Vision Transformer, that was one of the findings from the Google team originally. Transformer takes over, you know, as the scale increases, but at smaller scales of size of model and smaller datasets, the CNNs do rather better. So there is an open question, I think, as to what should we be looking at in terms of the architecture of the Vision Transformers itself. Another observation that has been made, and I can't recall where from, is that a lot of vision transformers are now trained using these contrastive learning objectives. And could this be weaker than perhaps some other way of training? Because when we train a transformer, we do it a different way. We have a generative pre-training recipe, which is extremely effective. So could there be such a thing for a vision transformer? And that's such a great question that we might actually answer it in 10 minutes' time.

Can't wait. Let's keep going.

So I'll just touch on one other example of a task that's very poorly solved by current generation of vision language models in the BLINK benchmark. And that's called the relative reflectance task. So here, the idea is that you give the VLM a picture, and you put a couple of markers on the picture for 2 different areas and you ask which point has the darker surface color, or are the colors about the same? So you always get 3 answers. So random guessing would get you 33%. So in the paper, they show an image of a hotel bedroom, and it's got the kind of cream-colored headboard for the bed and a white wall behind it. But because of the impact of the light shining in through the window, you know, the pixels from the cream-colored headboard are actually a bit lighter than the ones on the wall. However, just looking at the image, I can tell you that the headboard is cream colored and the wall is going to be white. And so I know the headboard is darker. And I can answer that in a blink, as they say, because my brain is doing some adaption to the images, is accounting for the effect that the light is going to have and where it's coming from and where it's shining. So these are some of the perceptual priors I'm bringing to the problem. And it turns out that the VLMs have a really hard time solving this. Humans are kind of 95% of the right answers. And the VLMs, the top-performing one was a LLaVA model at the time they tested, and it was doing a shade under 40% on this benchmark. So just another kind of example that they might not be doing things the same way we are. I also think that we should point out perhaps the most interesting observation from the BLINK paper in my book. And I don't know if they explicitly mention this, but if you just look at the performance tables of the different models they tested, you can see they tested GPT-4V and Turbo and 4o. And you can see that 4o has improved in a number of the tasks, but it's actually regressed in a number of other tasks, which is really interesting. So for example, there's been a significant regression in counting. 4V was solving the counting tasks at about 60%. 4.0, they measure at 49%. So it had regressed in its ability to count objects in the images. It's really not clear, like, why this is. Is this an artifact of distillation? Is this an artifact of, like, fine-tuning? Not sure, but it's a result.

Yeah, on this I can only say these things are weird beasts and find similar things in other areas too. I mean, not so much on like major benchmarks. I don't know if this would necessarily count as a major benchmark. It's not usually, you know, one of the ones that they would report in kind of the headline table.

Mm-hmm.

So they don't let those typically degrade, but certainly every time there's a new model, somebody's got a complaint with it, right? With the super wide range of use cases that are out there.

Yeah.

I wonder if they're even measuring this kind of thing. I mean, they measure a lot, but are they specifically like looking at the Jigsaw score on the blink? And maybe almost by definition not, right? Because this paper came after some of these models, right? So, yeah.

June, I think. June, I think, this year.

Yeah.

Interesting.

But counting is a really interesting one, because on the surface, it feels like a really simple task to count the number of objects in an image.

Yeah.

And across the blink counting task, it's one of the kind of— it's in the middle of the pack in terms of how well it's solved. They measured the LLaVA 1.6 model, 34 billion parameter variant, leading the pack of models with 66% success rate. Human scores near 98%. In general, just a 4-choice question, so the random choice baseline is 25%. And I actually found another data point, a paper called The Effectiveness Assessment of Recent Large Vision-Language Models, again from June this year, which was finding that a number of open-source vision-language models, including the latest LLaVA, at that time 1.5, were outperforming GPT-4V on counting tasks. So there's this kind of a second data when they weren't using the BLINK dataset for this. They were, they had another kind of methodology for this. Very interesting to ask like why. And there's a team from DeepMind did a bit of work earlier this year. They were actually looking at getting diffusion models to produce the correct number of instances of an object in a picture. But as part of doing this, they did a scan across some of the commonly used image caption pre-training datasets. And they found that there are some captions in there which do denote the numbers of objects in the images, but they said they're very, very scarce and actually possibly not enough to learn how to bind the number correctly to the appropriate features they're extracting from the image. And this might be one of the problems, just the scarcity of this kind of task in the pre-training datasets. So it does make me wonder that a lot of the times earlier we've seen the story of improving the ability of tasks, task-specific capability of a vision language model has been a consequence of augmenting a dataset in order to be able to train it at modest scale on that particular task. And maybe that's the way forwards with things like counting. We just need an augmented dataset. To be added into the kind of the mix here, and that might improve things. But, you know, not all is lost if you're trying to train a VLM to do counting. If you look at the kind of Anthropic's notebook collection of kind of best practices for Claude, they show you how to use good prompting techniques in order to make the Claude think through the counting task and decompose the image and then ask what it can see. And that does work a lot better. But again, it's showing you that the fact that reasoning can compensate for some of these perceptual deficiencies, but the perceptual deficiencies are there.

Yeah.

Okay, cool.

So a couple of minutes ago, you were asking about, is the vision transformer part of the problem here in learning this kind of rich representation of the images? And you asked about, could has jettisoning the inductive priors that we brought to CNNs been a problem here? And I said, well, the other thing that I've heard people mention or read experts mention is that it's the contrastive learning objective that you train a vision transformer with that might be part of the problem. And actually, a team at Apple earlier this year took a look at this very question. And they said, can we do multimodal pre-training of a vision encoder in a different way? I'm not using a contrastive learning objective. So what they did is they took a vanilla vision transformer, okay? And they asked if they can change the recipe for pre-training it such that they use a kind of a generative training objective. So I'll explain the setup and then we'll see how it works, how they do it. So you start with a Vision Transformer and a Transformer decoder, both trained from scratch. And it's trained on a large mix of image-caption pairs. The captioning is a mixture of alt text scraped from the web and synthetic captions. So for example, generated by a LLaVA model, something similar. So the data is prepared by taking image patch tokens. So these are small snippets from the image and text tokens, and always presenting them in— so you've got your image tokens first and then your text tokens. Okay, simple tokenizer in place. Training is done using prefix attention. So prefix attention is where you basically randomly mask a subset of the visual tokens which are all appearing first. So it could be all, all of them are masked, or it could be, you know, a small number of them are masked. And then the decoder has to generate the rest of the sequence, which will include generating the missing visual tokens. Remember, these are soft tokens, they're not from a codebook, followed by all the text tokens. And at this point, it's using kind of standard left-to-right attention masking. Okay. The loss is only calculated though over the non-prefix tokens, because obviously you're going to feed in the masked ones first. So it's going to compute the loss over the production of visual tokens followed by text tokens. The idea here is you're doing generative pre-training. So you're asking the decoder to generate the missing image tokens. But you can't use the same loss function for text and image, okay? Because all the text tokens come from a codebook. So you can do a standard cross-entropy loss for the text tokens. For the image tokens, they very simply just use a mean squared error loss. So the decoder generates an image token. It's seen, you know, some fraction of the image already. It generates what it thinks is the next one for the next patch in the sequence, and they simply Compare it to the real, you know, the real token, and then just compute the MSE loss. Okay, so that's a decoder-only training, a generative training recipe for the vision transformer. And they're basically training the decoder and the VIT from scratch in one step using this pair of objectives. Recall that the idea here is to train a new, like, vision transformer. And so what they demonstrate in the paper is that they can jettison the transformer decoder that they've trained as part of this recipe and then connect their new vision transformer trained in this way to— and I think they used a LLaMA-3 model to create a vision language model. They connect the 2 in an autoregressive fashion using a simple NLP, and then they train it on the LLaVA supervised fine-tuning mixture, which we talked about earlier. And this means they can do a nice ablation. They can compare the power of the vision language model with a vision transformer created using this generative pre-training recipe with kind of a drop-in, for example, a vision transformer the same size but trained on a contrastive learning objective. And what they find is that they see improvements in all of the VLM benchmarks that they tested against, but particularly in captioning, And in visual question answering. So those are the 2 they see very significant improvements in. So this is really interesting. You know, one of the questions was, is it the contrastive learning objective that is limiting the power of the vision transformer element of the recipe? Answer seems to be yes. Switching to the well-proven kind of generative pre-training recipe that we know works in language models works really well the Vision Transformer as well. What I would be really interested to see is whether this AiM V2 vision encoder from Apple being injected into a vision language model has an effect on BLINK performance. I don't know if it will or won't, but I think it'd be really interesting to know if it did. I think what we're going to see is more experimentation in the next year or 2 in this space. I think we're going to see the vision transformer come under scrutiny, and we're going to see more really smart ways of trying to adapt and enhance it, maybe even revisit the way it works, reintroduce some of the inductive priors that we lost.

Yeah. So can we just linger for a second more on the difference between the contrastive training and this generative training? I think I got it, but give it to me one more time and make sure I Because this seems like a pretty important conceptual distinction.

Training on a contrastive objective, which pretty much everything we've looked at before now, apart from the very original vision transformer, has been done with, is you want to encode your image, you encode your caption or your paired text. Okay. You've got 2 encoded vectors at this point, uh, in a contrastive learning setup. You look across your batch and you say, I've got this one true pair within my batch and I want the cosine similarity of the 2 embeddings to be high, and I want the cosine similarity of all the non-true pairs in that batch to be low. And the contrastive learning objective kind of forces the embedding space, you know, distorts it if you like, such that you get that result. So you'll always get, you know, an image paired closely in the embedding space with its relevant caption. With similar text. For the— for this objective, we're using a generative pre-training objective here. And so what's happening instead is we're prepending each example with a load of visual tokens, we're masking a bunch of them, okay? And then we're saying— so in other words, we're not computing the loss amongst the first, let's say, 3/4 division tokens. And then we're simply asking a decoder to decode the rest of the visual tokens followed by the text tokens. Okay, we're jointly training the vision transformer and the decoder at the same time, and we're just measuring its success in reconstructing the visual tokens by the MSE between, you know, what, what the true token was, next token, and what it predicted. And then we're evaluating its performance on text in the same way. So what it should be learning to do do is to attend to the caption, the text that appears afterwards. It should be learning to attend to the vision, the image earlier, because it's using this decoding strategy. But what they found was if you put all of the image tokens first and then the text, a lot more of the onus is on trying to learn the remaining unmasked visual tokens, tokens from the image. And this appears to make them makes for a much stronger vision encoder than doing it the other way around.

Now, that original Vision Transformer, it was just trained purely on images, and that was just filling in masked tokens as well?

Yeah, the very first Vision Transformer was actually trained as a classifier.

Oh, classifier.

That's right. And I believe on ImageNet. And so again, The output sequence contains a classification token, and they simply took that, and then with a simple linear projection, were training it to predict which of one of a large number of categories the image belonged to. That was how the very first vision transformer was trained. But all of the language, all the VLMs we've looked at have basically used a contrastively trained vision transformer. And we talked about how that was done when we looked at CLIP.

Yeah.

Yeah.

So would this, with this Apple approach, would you call this early fusion or late fusion? Because I'm looking at the diagram in the paper and it's like, there is still this sort of cross-entropy loss and there's like a different encoder for the vision component. So if you think of the fusion question as being like where in the architecture it happens, it seems more late. But given that they're all being pre-trained together at the same time, in some sense, that seems like quite early.

Yeah, I would agree with you. It seems like a bit of both, doesn't it? I think they call it late because there's still a vision transformer that's separate from the language model it's connected to, but the alignment between the two has happened a lot earlier.

Yeah. Okay. So yeah, I like it. I don't think I have anything else on the Apple one. And I see we've got the, I wouldn't call it the final frontier, but the next frontier is next. I think these, these papers are really interesting too. So yeah.

Uh, I was wondering about just skipping over the chameleon one and just talking about the transfusion paper. And the reason I was thinking that is because I think Transfusion is just so much better, like, in terms of not only its capabilities but also its training efficiency. It feels like this is just going— and it's the way that DeepSeek built their model as well. I feel like this is the recipe.

But, um, I trust your judgment, whatever you think best.

Yeah, so, and then we just, we finish up with just like looking at the Frontier Labs offerings. We'll just go through that. Sounds good.

I'll try not to derail us too much.

No, don't worry. We've, we've, yeah, we've got plenty of time in hand. Okay. So we've mainly focused today on vision language models, which have been looking at understanding images and to some extent video. We haven't really mentioned video today, but a lot of the models we've we've described can handle videos because a sequence of frames from a video is not so different to a sequence of pictures. And so, you know, a lot of what you get for— you get for free, basically. This ability to understand video as a— whether you consider it a separate modality or not, it's kind of up to you. There are some video-specific benchmarks. There's a VideoMME benchmark that you can look at to see how well the different models handle sequences of videos. But what we haven't talked about is, you know, what's, what's been missing for everything so far is the ability to generate images. We've really looked at image understanding. Obviously, the simplest way to have a VLM or an AI in general generate an image is to have it generate a prompt and then to hand that prompt off to a diffusion model and have the diffusion model generate the image. And indeed, if we look at the Frontier Labs today and the services they offer, this is what Gemini and GPT-4o are doing with Imagen3 and DALL-E respectively. And we just put behind that in parentheses, actually, it seems from the original GPT-4o announcement earlier this year that it is a true multimodal model and it can directly generate image output. I mean, the O stands for Omni in it, so it can generate other modalities too. But this capability has not yet been released. It's been promised but not released due to safety and infrastructural concerns. I think it's what OpenAI said, but it's interesting nonetheless. And the question is, how might this be working? And there's actually been quite a lot of work come out of, like, notably FAIR, Meta, over the last year or two, looking at kind of true multimodal models. And if you're interested, kind of the sequence of papers to look at is CM3, which is— they call the Causal Masked Multimodal Model of the Internet. And this was followed by a model called CM3-LEON, which is pronounced chameleon, which was then helpfully followed by a model called Chameleon, spelled chameleon, not the same thing. But this was all part of Meta's exploration of kind of true multimodal models that can both input images and text and generate them both. But I thought the one I wanted to focus on today was, was the latest in this sequence from Meta, and it's called Transfusion. In the Transfusion recipe, they're actually looking at going beyond simply just images and text, but It introduces what I think is the most promising recipe for multimodal generation as well as multimodal understanding. Okay. So if we focus on the image and text part of this here, so the approach is to pre-train a single transformer on an even mix of image and text data. Okay. But they're going to use during pre-training. A different training objective for each of them. Okay, so previous papers in kind of this series, or previous models in this series from Meta, when attempting this, had actually quantized the image tokens before they entered into the model. Now it's probably just worth saying very quickly, what does that mean? How does it work? So we've talked before about how, you know, text tokens obviously can be looked up backwards in the codebook, okay? So they're quantized, right? But we've used the term image token, um, very loosely because we actually know they exist on a continuum, right? But they don't have to. So one way to quantize image tokens is using a vector quantization. Um, method. If you come across the, the VQGAN architecture, this is like one way it's done. So the idea is you take an image, you pass it through a variational kind of autoencoder layer to get some latent vectors, and you then have like a learnable codebook. So you can imagine this as if you've ever studied how k-means clustering or something works, you're kind of learning where your cluster centroids are in your vector space of part of your training recipe. Okay. So the idea is you then quantize to the closest of your quantized vectors in this kind of latent space, maybe you've got 10,000 of them. So you encode your image token, quantize it to the nearest thing, and then you decode it again using the variational autoencoder decoder part. And now you just perform a number of metrics to see, okay, I've decoded my image again, having encoded, quantized it, decoded it, and now I'm going to compare it to the original image and then run just a number of metrics, you know, and calculate a number of losses over that to see how close I was to my original image. And I keep changing my codebook, you know, my learnable codebook, until I'm getting images that look very close to what I put in. And now I've got a codebook that I can use to quantize any image tokens that I see. So that's kind of how they were doing it previously. And they actually jettisoned this approach completely for transfusion. And they said, you know, we're not actually going to quantize any of our image tokens at all. We're simply going to pass them through a variational autoencoder, encoder part. And this kind of turns them into like latent patches, if you like. They're going to then pass it through either a multilayer perceptron or through a U-Net downsampling block. This gets them their latent vectors, and they're simply going to use those as tokens and inject them into their Transformer. Okay, so they're kind of continuous, and then the text tokens are going to be handled in the usual way. So they're going to be turned through a tokenizer into their vectors, and now they're going to feed this stuff into the Transformer. And the idea is you're training using a next token prediction, but you're going to do different losses, okay? Different losses for text. So text region is kind of easy. We're going to handle this with a simple linear layer and use a cross-entropy the same way we always train with a Transformer. But when we detect that we're outputting image tokens, we're going to process it through the corresponding U-Net up path and then through the variational autoencoder decoder to actually start generating an image. And then we're going to use the diffusion loss objective there to actually train, train that part. So we've got 2 different things going on here. And Transfusion really is, if you actually look at the architecture diagram, it is like a latent diffusion model that's been pulled in half. It's got this kind of transformer in the middle, and then you've got both sides of— if you look at a Stable Diffusion architecture, for example, it's It's like that pulled apart with a Transformer stuck in the middle with the text being handled in a new way. The very important thing here is how you do the attention masking when you train it. So the training objective, they use causal attention masking for all the text. So if you're decoding a text token, doing the next token prediction, you can look at everything to the left. For images, if you're generating an image token, you've got bidirectional attention. So you can look at all of the other tokens in the image when you're reconstructing one. This means every patch can attend to every other patch within the same image, but you can only attend to textual patches of other images that appeared previously in the sequence. Does that make sense?

I think so.

So more complicated masking to set up, like obviously more complicated decoding regime. You need to know whether you're decoding text or you're decoding images. You've got the kind of U-Net and VAE structure around it. You've properly here got an amalgamation of the 2 architectures. It sounds very complicated, like very complicated to set up, very complicated to make work. But what they discovered when reporting Transfusion results is really interesting. Compared to the previous way of doing it, so actually quantizing all of your image tokens in their previous Chameleon series, they say that they're producing images of a similar quality after only training for about a third as many flops, which is like truly impressive. So that really suggests they're onto something here. In the previous series of papers, the Chameleon papers, they described having real difficulty actually getting the pre-training to work stably. And they had to introduce a number of kind of what they called architectural innovations, which feel like kind of sticking plasters and changes to get it to train properly. Nothing like that was reported in the Transfusion paper, suggesting they had a kind of a smoother time of it. And then very interestingly, they found that on just text-to-text tasks, this new recipe was matching kind of the training losses that they saw in their previous series, the Chameleon series at half the flops. So we're getting not only better images being produced, but also better text. So it sounds like it's a really efficient recipe, just rather complicated to set up. Not much more to say about it other than this recipe, I think, was very similar to the one followed by the DeepSeek team. So that's kind of another recent model. It's fairly new, so late 2024, the Transfusion paper. But this feels like this is something we're going to see a lot more exploration of these kind of hybrid architectures in the very near future. And the team did some early experiments to ask if they can actually adapt this same recipe across kind of new combinations of modality. Can they look at audio, for example? They did some small scale experiments in the paper, and they suggest the recipe will work for those as well.

One thing I'm not very clear on is why ever quantize the image tokens in the first place. I recall reading about that some time ago, and I was like, but why? It just seems strange.

Because if you do that, then you can simply train a single decoder transformer to produce both images and text documents. You've just got a larger codebook or 2 codebooks, and that means you can use much simpler training objectives. You can, for example, look at the cross-entropy for both of them. You don't need all this other machinery in there to do the auto-encoding and then the U-Net downsampling and then upsampling. It's, you know, I'm surprised that when they mention that, you know, they achieved the same training losses at half or a third as many of the FLOPS. I thought it was really interesting because it sounds to me like with all that other machinery, that actually there's— it's going to be much more compute intensive to build one of these hybrid architectures like Transfusion. So I'm assuming that is all factored in when they talk about the losses seen at the number of FLOPS. It is a more complicated model.

I'm scanning through both of these papers and it is remarkable on the Chameleon one, which is the one that has the discrete or quantized image tokens. It is remarkable that the image outputs do look pretty good. Like you would think that, does it say? quickly how big the vocabulary size is of this?

They say in the paper. It's not that big.

It's only codebook size of 8,192. And the 512 by 512 image gets broken into 1,024 discrete tokens, which means, you know, they— yeah, that's weird. I don't know. That's like you've got 250,000 pixels, right? 500 times 500, you got 4 zeros with a 25 in front of it. So, 250,000 pixels. 1,000 discrete tokens. So you've got 250 pixels per token in this chameleon thing, which—

16 by 16 patches then, by the look of it. So if you take 512 divided by 16, you get 32. Square that, you get 1024. So, 16 by 16 patch. So, when you think about it, 16 by 16 pixels, and then you've got 8,000 tokens to cover that. How many different variations of a 16 by 16 patch do I need?

I would still think a lot more. I don't know. It just seems like each pixel is 3 colors with a range of 256. values. So just the number of possible colors for a single pixel is 256 cubed, right? And just from that alone, I would think, boy, 8,000 tokens to represent a 16 by 16 little patch would— just does not seem like enough. But I mean, it's hard to argue with the results. They The images do look remarkably natural. I'm quite surprised that that works. I would, I would've intuited that it would've been just much like more artifacty, that you would be able to see these, like, you know, where these tokens, you know, come together. Uh, yeah, I would expect you would see the seams of this and generally speaking, I don't see 'em. That's quite, that's It's quite crazy. Um, yeah.

They certainly reported for Transfusion that, uh, they did some evaluation benchmarks and compared it to DALL-E 2 and Stable Diffusion XL. And they found that they were outperforming both of those models and that they were reaching Llama 1 performance on text-only tasks. So that was their 7 billion parameter Transfusion model. Again, and I don't know if this interpretation is warranted from this, but it's really interesting to see such like fantastic performance. I know these are, you know, it's more recent architecture. There's probably a lot of tips and tricks they're using in training it which weren't used for SDXL or DALL-E 2. One also wonders, you know, what role has this mixed modality learning played In improving the ability of the model to generate images. I'm not sure, but it's, it's an interesting question given that we've certainly seen it happen the other way around.

Yeah, the image editing on this is really impressive too. I'm now switching over to scroll through the Transfusion paper, and I think the demos of this that are most compelling to me Are basically instruct-style editing. There was an image, uh, there was a model, not, oh geez, when was this? I guess it was almost 2 years ago now. InstructPix2Pix was the one that basically you could give an image and give a command and it would attempt to edit that image according to your command. And I did one of an ultrasound image of my son who's now a year and a half.

And.

Said, make it look like a newborn baby instead of an ultrasound. And it kind of did that. Not, not too bad. Um, it was a funny experiment anyway. And yet those were limited, to say the least, in terms of what you could do and how good the quality was and how much fidelity it, it actually had to your instructions. It was, you know, kind of all over the map.

Mm-hmm.

Here I'm like, man, this is exactly what we need for the Waymark application because we're talking very precise local edits that are not changing the overall composition, not making the image feel like it's a totally different image, but doing the sort of precise cleanup that you might see from an actual pro image editor doing it in Photoshop or what have you.

Yeah.

So things like, you know, a couple examples here that stand out to me most. One is change the graffiti on the side of a truck into calligraphy writing. And the before and after is just amazing. I mean, you see the one, you know, before is like tagged, of course, with graffiti. And then it's just like, man, it's perfectly in situ calligraphy that's been printed onto the side of the truck. Examples of removing things, you know, replace one object with another, change the color of this thing.

This is pretty impressive.

Yeah, it is pretty impressive. This has never been released, right? Was it? Do you know why they haven't released this?

No, to my knowledge, none of the series has been released, and I'm not sure why. One does wonder what image sources they used for the different models. They do say, at least in some of the papers, they used only kind of open source or publicly available image sets. I don't know if that's true for all of them. I'm not sure for everything they use. They may also— their text-to-text, they may also have some in-house text-to-text kind of datasets that they used, which for whatever reason they don't want to release, but I'm not 100% sure. But you're right, it certainly beats like the SpaghettiWeb ComfyUI.

Yeah.

canvases that I end up building to do image editing, just be able to write it in a single sentence.

Yeah, you can really see the future here, I think, and in some of the GPT-4o demos as well. It's funny, people ask us all the time about, with Waymark application, do you use Stable Diffusion or DALL-E or whatever? And we've actually found that those things are not very useful for our users because they aren't realistic to, you know, small business wants to present itself in a, you know, positive light, but in like a realistic light. They don't want it to feel like, wait, this is nothing like what I saw on TV when you actually show up. So the lack of control and the difficulty of grounding the purely generative models has been a real challenge. And you've of course can do image prompting. But a lot of times those also haven't worked super great. They sort of noise your input image and then take it in kind of a different direction. And it's like, oh, I actually wanted something that was more like local specific, like maintain the integrity of this, but change it in this one, you know, very specific way. And that has been hard to do. I also can imagine how there's other techniques now that are, that are popping up for this too, but character consistency has always been a real challenge.

Yeah.

For people that are trying, just trying to create original content. The Waymark creative team has done some really cool stuff. We did an episode, I hopefully have another one soon, but we did an episode on basically a short film that they made with all DALL-E 2 images. This has been a while.

Yeah.

Now there's a part 2 that uses newer models and all sorts of new techniques. Character consistency was a huge problem in those, uh, early models. Scene consistency was another huge problem. They, I mean, they came up with elaborate prompting techniques and all sorts of ways to try to get around that. And I think did, you know, a really remarkable job with what they had at the time. But when I look at this, I'm just like, man, it, it, a lot of it falls out of this very quickly where you can just say, you know, change this, do this, put this guy in a different scene. You know, next thing you know, you're kind of off to the races on a lot of the different things that you want.

Yeah.

want to do that have been hard. So yeah, presumably that'll be coming at some point from an API provider, GPT-4o or otherwise, or maybe they'll finally get around to releasing this. But it is, um, I noticed too, Lily Yu, who's a former guest, I had her on to talk about Megabyte. It was called basically a byte-level transformer and Interesting to see her name pop up on another one of these kind of continuous space transformer projects.

Yeah. Cool.

So this has been a fantastic deep dive and walkthrough. Where does this leave us now?

Well, maybe we could just wrap up just with a quick summary of what the kind of frontier labs are offering and kind of what's winning on the different benchmarks. So I found I had to compile this data from a number of different sources because it wasn't all available all in one place. So I've done the best I can, but if one is building with one of these models and you know that you want something that does well on NLMU or on, for example, DocVQA, so that's, you know, extracting information from images, Or maybe even if you're interested about what does well on Blink, I tried to compile this into a simple table. And I can just walk you through now what I think is winning in each case.

Cool. So yeah, this is really useful.

So what I looked at, um, I looked at, um, the Groq 2 beta. I looked at the latest version of Claude Sonnet 3.5 new. Gemini 1.5 Pro because I was unable to find results yet for Gemini 2. 401 Preview, I could only get one result for, and that was the MMMU, but 401 I could get results for all of them from. And then just a couple of open-source models as well. Okay, so if we look at leaderboard here and we say, okay, what is currently doing best on MMMU if I really cared about reasoning over images? And 401 is standing head and shoulders above everything else with 78% there. And if I wanted to look at what was next down from that, we would find Claude 3.5 Sonnet New and the InternVL 2.5 model, which is currently available, I believe, through an API. We'd find those down at 70 and a bit percent each. So Gemini 1.5 Pro is coming a few percentage points below those 2. So that's kind of where we're sitting on MMMU at the moment. Be interesting to see what Gemini 2 does there. But the Grok 2 beta is sitting at 66% on MMMU. If we were to look at—

One quick note on that too, because this intern, I keep forgetting as we've gone through this who made this model, the OpenGV Lab, which is Yeah. Uh, so it's another, I mean, I think we touched on this a little bit earlier too, but it's another good reminder that the Chinese models, the Chinese labs are not super far behind. They are in this case, right there, head to head with Anthropic, 5 months earlier.

Uh, the LLaVA 1 vision model is from ByteDance. The Quen model is from Alibaba. Um, Yeah. And they, they all perform very well.

And this is not, yeah, the, the reason I mentioned the 5-month earlier thing too is cuz sometimes people will be like, oh, well, they're, you know, doing that. They're starting with, you know, an American thing or they're, you know, they're starting with LLaMA and not acknowledging it, or they're, you know, training on the outputs of the, uh, American models, whatever the, the sort of rationalization often is. And I'm sure sometimes that stuff is happening, but here, It is still, as of now, and I guess we've got caveats around like O3 and whatever and full O1 where not all the data points are available yet, but it is striking that this Shanghai AI group has 0.3% lower than Claude 3.5 Sonnet knew a full 5 months ahead of time. So they're definitely not, um, definitely not training on Claude outputs to achieve that. And that is higher than GPT-4o by just a point, but still it's like, um, I think it's, it is very much worth keeping in mind that, uh, yeah, that's good progress.

Yeah.

The gap here is like basically zero. And arguably you could even say, and if you squint at it, you could even say the, the Chinese labs are maybe a little bit ahead, but I would say.

Yeah.

Overall rounding, you should probably say they're about the same.

Yeah. And we also note as well that the parameter counts in their models are all around the 70 to 80 billion parameter mark. Presumably, they're going to build larger and larger, and presumably they'll get better and better as a result. But lots of what we talked about earlier in the podcast about how fast the training recipes and the datasets sets are developing, you know, it probably goes to account for a lot of this. Just the more recent the model, like, they get better every single time because people are learning so much about how to effectively do this. I think it's also interesting to look at the DocVQA scores here. You know, all of the models that I mentioned are doing above 90% on that benchmark. The best I can find is, again, it's the Quen 2 VL, Quen VL 2 model, which is currently doing 96.5%. So that's open source, not proprietary model. I don't have results for the new Claude 3.5 or for 401 on DocVQA. So that's just to point that out. But that model is doing better, for example, than Groq 2. It's doing better than Gemini 1.5 and GPT-4o on DocVQA. So kind of leading there. And Blink is very interesting because we haven't got results for everything here, but kind of at the top of the leaderboards we've got 4.0, which has about 63.2%. I have seen a result that's higher than that, but I've seen 3 results for it. 2 of them are at 63.2% on Blink, so I'm going to go with that one. The OpenGV Labs in turnVL 2.5 is clocking in at 63.8%, so doing ever so slightly better. Gemini, 61%, and Claude 3.5 Sonic New, that's 56.5%. So those are the best and most up-to-date scores I can find on those 3 benchmarks. It's also worth just looking at the mini class here. So here I've put in kind of the Flash and mini versions. of some of these models. They're obviously a little bit lower down most of the benchmarks. They're smaller models, but it's interesting to know, you know, if I was to pick one up, what would I use? If we look at MMMU, we've got results for Gemini 2.0 Flash, which is clocking in at 70.7%, and that makes it stand head and shoulders above everything else in the mini class, by which I'm including 4o Mini, the intern VL 8 billion parameter, I haven't got result for 401 Mini. Groq 2 Mini is at 63.2%. So best bet, if you want kind of reasoning over images, would seem to be Gemini 2.0 Flash at the moment. DocVQA got very few results from here, but the intern VL 2.5, 8 billion parameter model, is clocking in at 95.1%, which seems pretty damn good to me. Interesting enough, that's exactly the same result as it gets in its full 78 billion parameter form. Blink, we're just going to finish on Blink for the mini class of models. The best score I can find, and I couldn't get one for Groq, for Gemini 2.0, or for 401 Mini, but I could find it for GPT-4o Mini, and that's clocking in at 51.9%. Which means that the intern VL 2.5 model is getting slightly better at 54.8%. The surprise winner coming in from left field in the mini class is Microsoft's Phi, or Phi, depending on how you want to pronounce it, 3.5 vision model, which is a 4 billion parameter class model and scoring 58.3% on the BLINK benchmark. Not doing so well on MMMU. Doing really well on BLINK, which is really interesting. So I combed the technical report for the V-Vision model, and just a couple of interesting details from it, and, and not enough to make conclusions on. For a 4 billion parameter class model, they used half a trillion pre-training tokens from the mixed pre-training dataset, and that's a big dataset for a small model. We think about the intern VL 2.5 model. They said they trained on 128 billion tokens. The Quen 2 model, 1.5 trillion tokens. But those are all many times the size. So it's a fairly large training dataset for a small model. The other really interesting thing was that they mentioned their SFT dataset, which they described as a combination of datasets from a significant component of which was built in-house by Microsoft. 33 billion tokens. So this is instruction fine-tuning data. That's an extremely large dataset, especially for a small model. I mean, I can't recall seeing one that large anywhere else in the research. And they also mention performing DPO, which only a couple of the other models like LLaMA 3V explicitly mention a DPO step. So not sure why it does so incredibly well, but anecdotally, I have seen plenty of commentary online with people saying, wow, Phi 3.5 model is really good at visual understanding. So really interesting, perhaps something we should have dwelt on a bit more in the podcast. And that is about everything that we wanted to cover today. I mean, We've kind of told the story of the last couple of years of VLMs. What are we going to see in the future? I think we'll see a lot more of these kind of true multimodal models following the Transfusion recipe. I do expect to see the scale, the parameter count of open-source VLMs increase further, particularly now as we know about this kind of progressive upsizing of language model backbones works so well. We should expect to see more innovation in either pre-training vision transformers, or maybe even their replacement at some point, and a continuation of this kind of production of new fine-tuning datasets that contain programmatic or human augmentations. So I would, for example, expect to see new fine-tuning datasets there. What we haven't seen too much of as well is— we just mentioned DPO at the very end there— not too much exploration of the role that alignment post-training can play in vision language models. So I expect to see that explored as well. And that concludes not quite so whistle-stop tour through the last 2 years of vision language models.

Amazing. Well, the depth of research that you put in to make this possible is outstanding and definitely much appreciated. And I learned a lot from it. So as you know, that's how I tend to score of these episodes for myself, and I come away definitely with a much better understanding of the various options and strengths and weaknesses and even a few prompting techniques along the way. How does this relate to what you typically do? And maybe in the last couple minutes, just tell us a little bit about like your normal work and the sort of stuff you do commercially.

Yeah, so a lot of this is relevant to some of the work that I'm doing. So, Veritai is the small consultancy that my colleagues and I set up a couple of years back. We do a lot of AI strategy work. So, before that, we used to do a lot of data science and analytics strategy in my background, and we've moved on to thinking about how do you develop and build an AI strategy, and especially if you're an SME. So, I think that's our sweet spot, although what we do does work, I think, in larger organizations if you're kind of working at the department level. So, we do a lot of that, but we also do a lot of prototyping work for people, so, proof of concepts. You have some ideas, you know, in your strategy, things you want to follow, things you want to— things the company wants to try out. So, we have a very kind of structured way of performing these experiments cheaply and finding out what's easy and what's difficult. Because as you probably know, a lot of working with AI today is, you know, it's your mileage may vary. So, we kind of know what the best practice is, but it's very hard to know from the outset whether you're going to get great results given the data or compute constraints that your client may have. And so, we've kind of developed this rapid experimentation methodology. But in some of the domains we're working in, and we do some in medicine, particularly looking at medical terminologies and how they get used by language models and how language models can work with them. But also, I know I've spent a lot of time looking at open-source intelligence for a few clients recently, and there a lot of kind of the data that we're trying to interpret is multimodal in nature. So there's lots of trying to understand, you know, how does this image seem to correspond with this claim that's being made? You know, can we infer who might be in the image or what the image is about, given that there's some surrounding text or context we understand. And so, in that, you know, it's in that domain that we've certainly been working with some of the VLMs here.

Yeah, I hear a couple sort of reflections, probably, of some common experiences that we've had. I don't do a ton of this sort of stuff, but occasionally, you know, mostly I've done it just for my own company, which I've mentioned a dozen times already. But occasionally I'll take on a project for somebody that asks me for help with something. And it's an interesting juxtaposition often of like, and I'm sure you've found various versions of this, a lot of times you can like answer somebody's question really quite quickly.

Mm-hmm.

And I'm always reminded of Tyler Cowen's answer for when he's asked how long it took him to read a given book because he's famously fast reader. People ask him that question, and he always answers with his age. And I think this tour through vision language models is a good reflection of how much depth and sort of obsessive quality of research has to go into the ability to then turn around quickly and be like, I think I know what to do here for your random situation.

Yeah.

And I think we can put together a proof of concept on that pretty quickly. But that's because you've done this, like, extensive exploration of everything that's out there and have a really already fine-tuned intuition for which direction to go.

Yeah, the way we do strategy projects relies on this a lot as well. It's like, you have to come with prepared minds, you have to come having been immersed and soaked up everything that's happening. So that when you see the, you know, when people say, this is the problem that exists here, you can pattern match it to something. And, you know, I am sure, like you, I've got a, like, a big archive, searchable archive, like my own personal kind of rag index that I maintain at home of just everything that I ever come across— papers, newsletters, Substacks, whatever it is sits in there. And I just try and pattern match it so that I have some grasp of, okay, this is how we're gonna, you know, suggest the options for how you might solve this problem, A, B, or C. And yeah, but you do need to be steeped in it, I think, in order to then spot the opportunities to use the technique, the technology, the model, whatever it is, when you actually see it in practice.

Yeah.

Cool.

You want to tell us where we can find you and the company online?

Yeah, you can find the company website at Veritai, which is veratai.co.uk, veratai.co.uk. Or you can find me on LinkedIn, Will Hardman. And I am writing a fair bit at the moment about AI strategy. And next year we'll be writing about various other things and maybe about vision language models as well.

Cool. Well, I'll be sure to connect with you there and encourage the audience to do the same. This has been, uh, Fantastic walkthrough of vision language models. I know a lot of work has gone into it, but if you want to tackle another topic like this, I would love to do it. Um, now I will say thank you for this one. And officially, Will Hardman from Veritai, thank you for being part of the Cognitive Revolution.

Thanks for having me on, Nathan.