Building Multi-Modal Search with Weaviate | Build Effective RAGs

Hey, everyone. So My name is J.P. i work at Weaviate. My role basically is as an educator. So I get to do really fun things like this, meet all of you and talk to you about this. Really exciting and a field that's evolving very, very quickly. I think we were here a couple of months ago to talk about semantic search and retrieval augment generation with weaviate at datacamp just like late last year. And since then, one thing that's really come to the attention, for attention for everyone is this multimodality. And that goes in two directions. Right. So one which we'll talk a lot about today is search. So let's go through that. But I'll talk a little bit about retrieval augmented generation with multimodality involved as well. And I can see some people coming in the chat in different parts of the world. It's really, really exciting to meet all of you. So I am. I do apologize that we can't code along today, but hopefully you'll get to see a lot of things you can do with this material and how it works under the hood as well, because I find that personally, when I'm coding along and trying to follow along with video, it can be quite, quite challenging. So hopefully this is a rich and still rewarding experience for you. Let's get in onto it. Let's see if I can move the slides. Cool. So what I put together in the data set is I've got about 25, just under 30,000 images and. Oh, sorry, there are some images. Okay. So I do apologize if you're having some issues with the video, but I guess Reese, let me know please, if you want me to jump back or anything like that. So I've got a Data set, about 30,000 images or so, and really the goal of the session today is to talk to you about, hey, what is multimodal search at a really high level and how does that work? And hopefully by seeing the code that I've put together and sort of queries as well as, you know, really to take you end to end from building a database to using it and querying it, you'll get a good idea of what you can do with it and. And hopefully it'll also spur on your imagination and your sort of thoughts about how you can apply this in your field, in your domain and so on. The other thing I wanted to touch on before we sort of get into more of the specifics is, is why multimodality? Right. So. And my colleague Zen likes to talk about this when he, you know, he's had a baby recently. How do we learn as humans? And up until now we've used a lot of text and computers. But if you think about how children learn, it's not just about reading, right. It's really, you know, seeing things, hearing things, touchy smell and all your senses. And that's because there's so much information that's out there that's encoded in these other media that's not just text. And no, this isn't quite, you know, general AI, where it's going to come and get us or anything like that, but they do unlock a lot of information that just wasn't accessible before in terms of just being able to read text. And hopefully you get to see more of a sense of what I mean by that as we go through today. But yeah, that's really the key to why multimodality has been very, very popular. Cool. So I'll do 10 second introduction for myself. So I'm an educator. I work at weaviate and weaviate is both a vector database and it's the name of the company as well. So before, as I mentioned, we talked about how vector databases enable searching by similarities and what's called retrieval augment generation. If you were here before, you might have heard of RAG already. But basically it's a technique to augment some of the things that large language models like GPT4 and Llama and so on can't do all that well. And those things are really recalling specific facts or even up to date information that it doesn't or didn't have access to when it was built. So let's talk a little bit more about that to set the scene and talk about vector databases. So this is a really, really high level conceptual diagram of vector database, right? And this is just like any other database in this diagram. So, so we've got a vector database on the left and typically the workflow is you build it somehow and once you have it, you send a query and retrieve things. The key difference though is that a vector database has what's called a vector index, right? So, and what that does is to group these objects in your database by how similar they are to each other. So you can see in our little diagram we've got items like what's generated chicken, wolf, dog, so those animals at one side and you might have food like bananas and apples away from each other. And what that allows you to do is that if you want to find something like if you put in a query for the word kitten, the database would understand, or the model at least would understand that that's close to objects like chicken, wolf, dog and cat and so on. And it'll allow you to find objects by similarity. And what we talked about last time was that with retrieval augmented generation, we're able to augment sorry to complement large language models with the capabilities of a vector database. And you might have heard of phenomena like what's called hallucination with large language models. So that's when you ask a language model, hey, who won the latest big sporting event, who is the current xyz or something really specific about your field or your company. The language model just doesn't have access to that information. And even if it did, it might go out of date. So what you do then is to couple it with vector database so that you can feed it that particular bit of information. But then you let the language model do what it's really good at, which is using language, rephrasing information or extracting information, just using your natural query. So that really short circuits a lot of the things you can do from. Also that really makes it efficient to extract information or knowledge or wisdom even from your data. Right, so that's rag. And this is setting the scene. Now today we're going to talk about multimodality. And what that means is here's that picture of a vector database and a vector index again, where we put similar things next to each other so that you can retrieve them faster. But instead of just having text and sorting text objects by their similarity, you can have objects of other modalities. Right? So that could be images, which is the example that I'll use primarily today. Whether it's an image of a chicken that'll end up next to where you would put in the word chicken and same thing with bananas and apples and so on, as well as anything else really you can think of. And again, depending on how we go with time, there's this other direction that you can go with multimodality, which is to use multimodal objects as inputs. So some of you may have used some of the latest, latest language models. With those models you can actually feed in things like images as part of your input, which along with your chat or text prompts, which is really cool. And I'll show you some of the examples of where I've done that. Okie dokie. And I believe we'll have opportunities to have a chat or answer questions either halfway through, depending on how we go, or towards the end. So, so please, if you do have any questions, please Ask them along and then we'll do our best to answer them as we go. Fabulous. Okay, so let's have a look at the model. So, as Rich has mentioned, because of the kind of size of these models, and as I'll show you later, some of these models aren't really even quite productionized, which is a really interesting phenomenon. So I'll show you some of that. But what I've done here for this workshop is to build a notebook that you can follow along later on, which will give you a really good idea of the things you can do. And look at the end of the day, the modality that we do have access to the most images and text on the Internet. And so what we've built this demo with primarily is using a model or at least a family of models called Clip. This is an open source model that's actually developed by OpenAI a few years ago and it's quite popular and really, really interesting insight into building this model was this. That. So if you are familiar with machine learning and different fields of machine learning, quite often the data that's really difficult to come by or expensive to obtain is your label data for training. Right? So it goes for classifiers to anything else where you need the ground truth labels. You can train your neural networks or classifiers or so on. The really cool thing about Clip was that they were able to say, oh, actually there's this huge corpus of images out there on the Internet that has labels because we all have these images with captions on the Internet. So what they've done is to build a model that leverages that and gets the model to basically understand or at least to reduce the loss function as a basis of between the text caption for an image and an image. So here what I'm doing is loading what's called the sentence transformer version of that model. So we're loading actually two models here. One is the actual Clip model and that was what we'll be using to load images and turn that into an image embedding. So again, that's putting our objects and turning that into a series of numbers to put into vector space as you saw in the diagram at the start with the picture of a chicken or the words and so on. And the second model that I'd be loading is this sentence transformers model for Clip. So what that does is essentially if you try to think of it as translating text into the same set of similar set of numbers or number systems so that they are comparable to each other, you can read more about these models. So these models are publicly available and I've got links right here to Hugging Face. I'll just make them a little bit larger so you can look them up and get information about them on Hugging Face Model hub if you like. Cool. Let's go back to our image. I might just move that a little bit. Cool. And I've got a few helper functions to help us along. So this is so that I can specify an image that's local or on the Internet. And in weaviate the way we store image data is as base 64 strings. So this is again a helper function. Not terribly exciting to change the binary representation to a base 64 text representation or string representation, excuse me, not quite text. And then I've got another helper function too so we can show you some cool pictures along Jupyter notebook. And this just uses matplotlib to display the search results. And if there is a query as well, I've got a source image parameter set up so you can pass also that I or you can pass along a response from weaviate as well as the source image and you'll just get plotted nicely. This will make a little bit more sense once you see it in action and the next couple of things probably a little bit less exciting. One thing I'm doing is I'm getting an image set of images of movie posters. I thought, hey, what's really interesting data set for everybody. At least for me I'm a bit of a movie person. So there's a lovely API provided by entity called tmdb. So I was able to obtain a set of images for X number of movies through there. So that's really cool. And here I'm just going through the path where I've downloaded and just getting the file names so I can iterate through them. So hopefully that makes sense so far. Next what I'm doing is spinning up an instance of our database, right? We've got a bunch of data, we are going to turn them into embeddings or their vector representations. So what I'm going to do is to spin up a weaviate instance and. And there's, you know, it's a database, so the needs of its users are quite different. So there's multiple ways to use weaviate and spin up weaviate database instance. The way I've chosen to do so here is to use what's called embedded weaviate. So it just means I don't have to leave my Python environment to create my database and it Just connects that straight away. So that's what I've done here. But there are other options such as connecting to a local instance or connecting to a SaaS server. And yeah, if you're not familiar with Weaviate, Weaviate is what we call AI native vector database. So if you're familiar with hopefully something like SQL or SQL databases, they organize data kind of by either their textual data or what have you. But with Weavier it allows you to index or organize your data by similarity of the vectors. And as you'll see, that allows retrieval or finding kind of relevant data by similarity. Cool. The next thing to do is to create what's called a collection. If you think of a collection. Again, if you're coming from a SQL world of nomenclature, a collection is like a SQL table. So all I'm doing is setting up the structure or the data schema or data structure for me to store objects into. And since I'm going to be. So don't worry so much about the syntax of the specifics of what I'm doing is kind of what I always tell people when I'm running these streams. And that's because you can always look them up in the official documentation for whatever library you're looking at. And second thing is, I think what's really helpful is to decouple the, oh, what function is he using to what is happening under the hood so you can understand and follow what's happening. And to me at least, that makes it easier for me to replicate it in the future by if I understand what. Because if I understand what's happening, then I can look up the syntax and apply it as I go. So what I'm doing just here is to specify a couple of options and this you don't need to worry so much about because I'm just saying to we've yet, hey, don't worry about creating vectors of our data, because I'm going to be creating those vectors with the models we just loaded up. And this is for our RAG retrieval augmented generation configuration. Right. So this will allow us to connect to a language model and ask it questions later on. And then with these properties, what I'm doing is to define a template for our data. So really, if you're coming from a SQL world, what I'm doing is creating a collection, which is basically a very similar thing to creating a table in SQL or if using something like MongoDB, it has the same equivalent idea of a collection. And a property in weaviate is like a column in SQL Right. So I'm defining what properties or what attributes of each object will I be storing and what data type does that have? So this is text or kind of string type data, and this is an image blob. So I can store the beast, the stringified representation of our data in base 64 representation. And from that point, as you might imagine, what I'm just going to do is to iterate through each object and import it. So it's probably in terms of number of lines of code, fewer lines of code than you might expect, because a lot of these ideas of, hey, how do I turn my image of a cat or a movie poster, whatever, into a form that's indexed by similarity, is taken care for you by the people who build the model. And the job of how do I then store it so that I can retrieve it later is taken care of by the folks who build the database. So you'll see the code as we go through it here that a lot of these ideas are abstracted away, so you don't really have to worry about that aspect of it too much. So this portion where I get the collection is to just create a Python object called MM or multimodal collection, so I can insert objects into it. So again, if you think of it as like fetching the table in SQL so that I can add rows to the table, that's what I'm doing here. And then I'm just creating a. I'm just iterating through those objects, I'm going to add them one by one. So I'm just going to create a counter so I can report the progress as I go, have a start time. And this batch process you see here with the context manager, it just makes sure that you don't have to say save X number of objects and then flush it, send it to your database every so often. So if you are doing this manually, you might say, oh, iterate through 10, 50, 100 objects or whatever, send it to my database, recreate that list and go on. But this batch process just takes care of that for you, so I don't have to worry about it. Next, what I'm doing is going through my image data set. So remember, this is our list of the pods that we created a little while ago, and this is just my name for my data set, so that if I have more than one of these collections, as you'll see in my demo, I do have another one that I will be searching through and then from then on, it's just another in a for loop. So what I'm doing is we using the image model that we loaded earlier and then encoding that is turning our image into a series of numbers. This is the image representation that we've loaded. And I'm just going to turn that return set of numbers. I believe it comes back as a tensor or something like that, so that it's GPU compatible. You don't need to worry too much about that. But basically the thing you need to know is that what we're getting back at the end of this is just a list of floats, right? So you list a floating point numbers. And then what I'm doing here is to create a base 64 representation of my image. And that's just for that. I. So that I can store it into my database and so I can retrieve it later and use it for, you know, viewing it or whatever else. At that point, I've got all the information that I need for my image. I've got a file path because I'm using that to load the image. I've got the base 64 representation of that image and then I know where the image came from because that's what the source is. Now here, this is key because I'm generating the vectors using my model. I'm gonna have to tell weaviate, hey, this is the vector representation. You don't have to worry about it. And here, this is just me creating a particular ID so that I know what that looks like. So I'm just creating a unique ID for that object based on the path of the image because I know that the path is going to be unique. And I've got another nice quality of life thing, like every 500 objects, just tell me, you know, I've added total, whatever number of objects and it's taken X amount of time for those 500 objects. When I was running this at home on my M1 computer, obviously without GPU acceleration, sadly, I think it was taking about, I want to say about 30 seconds per 500 objects. So it's not extremely fast, but it's also not incredibly slow either without a G, without any GPU acceleration. So, yeah, it's quite a good, quite a good set of usage. Cool. So hopefully you don't have any. Hopefully that part was clear. So what I've done so far is to load basically whatever set of objects convert each object into, into a set of numbers. Right. So those are vectors, and those vectors represent something, some sort of conceptual idea about those objects. And then what I've done is to add all of those objects into my Vector database so that I can search by those meanings. Cool. So since I've done that, let's have a look at what happens if I run queries against them and load them up. So here I'm fetching that object like I did before, and then let's try some different things. So here's one example where I'm now searching those images by text. And what is really, really cool about this is that I didn't encode any other information or I didn't tell the clip model anything else about these images other than to say, oh, these, like, here's an image, turn this into a set of vectors. Now what I'm going to do is you'll remember that we loaded two models, one an image model up here, right? So we loaded that image model and then we loaded this text model. And I'll just go back to show you what exactly I mean by that is we had this image model and this text model, right? So one was called Clip Fit and then one called Clip Fit Vit Multilingual. Cool. And then I've gone, not far enough. And then what I'm going to do is to get this query, and obviously this query can be anything. I want to encode the query into another vector, another series of numbers. And if I perform a search by that, and this is the search I'm running is called a near vector search and is quite indicative of what it's trying to do, which is that, hey, I've got a vector that I'm searching for, right? What are the most closest objects to that? Right? So if I run that, this is what I get. Sorry, If I run that, this is what I get. So, you know, that's not so bad. So I looked for man with Suitcase. I get posters for Midnight in Paris, the Gray Man, American Gigolo. So that's Richard Gere in a suit, that's Owen Wilson and kind of like slacks. So it does give the idea of someone who might have a briefcase. But we haven't quite found anyone with a briefcase. That's not so bad. And also remember that in this particular example, I've got a very small data set of 100 or so objects, I believe. And if I now search objects by image, I can do similar things. What I've got to start with is a data set of images. And here I've searched for the string magnet suitcase, and these are the objects I got returned. I had returned. And now what I'm going to do is actually query it by another image so I can find images using a source image. And when I do that now I'm going to have to use my image model to create vectors for that image. Just remember again that there are two distinct models that will translate into the same, translate the objects, whether it's an image or a text, into a same kind of vector that are compatible with each other. And I'm going to again perform a near vector search just like I've done before. And what happens? So this is the image on the left that I searched for, right. So I said to Weaviate, hey, find me objects that are the closest to the vectors of this. And I get these images back, which is kind of interesting in that the color profiles for one on the left are quite different to the second image. The second image is the closest one here. But the overall layout and the kind of feel, if you, if you wouldn't mind me being so vague, is very, very similar. Right. So they both have people dressed quite well lined up against each other in a row. There's this very, I wouldn't say very similar, but you know, a group of people standing next to each other, well dressed, with a caption up top, right. So you can see, start to see what it might be doing. And this is where the model that you're using comes in becomes critical. So certain models might be trained to look for things by similarity based on their color profiles, for example. Right. But with the clip model, what it's doing is, if you remember, it's trained using captions for images. Right. And captions tend to describe what's happening within the image itself. So if you imagine a caption for the first image, you can imagine that it being very similar to the caption for the second image, third image, and so on in decreasing order. So similarly, I could perform a search for red cars. And this search would basically look for images whose captions would have been similar to those to that of the words red cars. And here's kind of what I guess. So I get one for the movie Cars, which is cool. And I get, you know, this Jurassic park poster, and I suspect that has something to do with there being a big red circle in the middle and this Hugh Jackman movie and there's some sort of mechanical connection there. Again, just keep in mind that in this particular example I'm working with a limited data set and I'll show you one later where I've got a larger data set as well. So that's the very, very high level concept or the idea of and demonstration, excuse me, of what happens if you do run a clip model. So if you run a multimodal search with a clip model, again what we're doing is to load these two models, one for translating images into a series of numbers or called a vector, and another text model which will translate text into a series of numbers which will be a vector. But the really the key thing is that they are compatible with each other and you can search one or the other using each of the these modalities, whether it's an image or text, they kind of become, if you think of it as a translator between some sort of neutral third party language that can speak image and can speak text, that's really what these multimodal models can do. Rhys, how are we doing for questions? I don't know if that's a good time to see if there's any questions we can answer.

Sure. I think you went through some of the questions from the audience already. There is a question from Yu Qian saying can you explain vector indexes? So if you've got a group of different objects, do they all have the same vector index or is it a different index for each one?

Oh, that's an interesting question. Yeah, so there's a lot kind of packed into that question there. And I think we're maybe talking about an index like we might talk about in a SQL table index. Right. So when we talk about vector indexes, if you think of it similarly to, I don't know if you're familiar with the concept of a inverted index which you might have in a database. So if you're looking for in a SQL land, because I think that a lot of the audience would be familiar with SQL and you, let's say you filter your million rows for rows containing the word I don't know cat. I like cats. What your database does typically is not to look for that word cat in every row manually at that time. What you would typically do if you do a search like that is when you build that database, ask the database to perform what's called indexing and that is to build a table, I guess not literally a table usually of objects of those rows that you predetermine or you can pre calculate which words appear in which rows. Right. So when you look for the word cat then you look up that table and say oh, these rows 3, 45, 10, 24, what have you have those words. So that's the kind of, that's called an inverted index because you go from the having the full strength to all these words appear, but now it's reverted. So that you have these words appear and then you go back to, you know, these rows now are relevant. Sorry about the long answer. So when I talk about a vector index. Yeah, vector index here, it does a similar thing. What it does is to organize those objects by meaning and it really just literally puts these objects in close proximity to each other, at least as far as those numbers go. So that when we talk about a vector index and we say we're looking for an object in that vector index, you say actually, actually just saying, oh, what would Kitten be right in this vector index? Or where would it be in this vector index in terms of a vector, like a number representation? And you literally, almost literally putting it there and you're saying what objects are close to it? So going back to your question of like, do they have the same index? Kind of yes and no. They will have very similar vectors. Right. And this is obviously a 3D representation. So if you think about that as having, you know, X, Y, Z numbers, they'll be similar. In reality they're like something like you know, 1 to 2000 kind of numbers, floating point numbers and you can compare them. And what the vector index does is to really speed up that calculation. Because if you, I don't know if you've ever done numpy or you know, scipy similarity calculations over like 50,000 rows, it's slow. So you wouldn't want to do that in production with like millions or hundreds of millions of rows. So that's what a vector index does. And that's kind of like the magic of the vector database abstracting these tasks for you. Sorry about the long answer, Richie, but I tend to do that.

No, it's fascinating. Let me just make sure I've understood this. So the first part, you're talking about indexes on SQL tables. And an index is just a way to pre calculate things so you get better performance when you're doing lookups and SQL queries. And the second part, the vector index, so you got your text or your images and then you use a process called embedding to convert these into numbers. And the vector index is that. So that, that's the, the individual. You've got a vector for each of the objects and the vector index is like, is like a list of vectors then or.

Yeah, you know, I like to. Another analogy that some people use is like you've got these, you know, as you said, embeddings, which is exactly what you said, right? Which is like this is the meaning captured in series of numbers and it's organized like it might be in a library or in a supermarket or what have you. Right. So you go to a particular aisle. Etienne loves this analogy. And then you go, oh, I've got this canned soup. What I find, I go to the soup area or the canned stuff area. Right. So it's vector embeddings, Sorry, Vector index does a similar thing. So if you have a vector vector, you just go set area where the similar vectors exist and you can find those similar ones faster.

All right, super. Next question comes from Jane. So in this case we've got a limited data set. So what happens if there are no results? So does it ever not return result or is it always going to try and return whatever the nearest thing, even if it's a bad match?

Yeah, that's a really good intuition, Jane. Because it's based on similarity, it doesn't inherently filter anything out. It's a degree of similarity between whatever metrics like 1 and 0 or between 0 and 2 PI or what have you. And that's exactly right that you. It's a good idea to set some sort of threshold or at least have a look at what the thresholds are. If we look at these results that I pulled up because I'm performing this on a smaller. Before you do the full one, the results aren't like amazing. None of these guys have suitcases. I mean sure that men that's great, but it's not the most amazing thing. So one way I'm limiting the results in this particular query is by this limit parameter. So I've got a number three. Another way you can limit this is by a maximum threshold distance. Right. So you could do that. And then in production setting or what have you, you would kind of calibrate that by saying, oh, given this model that I'm using and given my particular data set, I find that results over let's say 0.2 distances is too far cosine similarity wise. So then you can calibrate those results so that you don't serve again in production setting kind of results that are the best match but perhaps aren't quite good enough for your audience. So yeah, that's a really good question. But setting of threshold or maximum similarity or minimum distance might be something that's really good for that purpose.

Okay, yeah. So you really want to be able to make sure that you are only returning good results, ideally. Excellent. So I have many questions for you for anyone in the audience. We do have plenty of time for questions, so please do ask JP in the chat now. So I think My first question is just can you talk me through when you might want to use the different use cases? So you, you showed an example of having a text input and returning images. You showed an example of like having an image input and returning images. When would you use each of these things?

Yeah, I think, you know, when I think the larger kind of search engines already do something like this. And if you think about the cases when you're, you know, we're looking at like redecorating at the house, right? And you might want to look for living rooms that might have a certain look, and that's really cool thing to do for us to be then to go and find pictures of a room or layout that we kind of like, at least decorate it the way we like and look for similar images that way. Right. So that's kind of a really easy, you know, pretty trivial use case that I'm using at home. But if you, you can also imagine other cases in domains like, let's say marketing, you want to see what material for a certain field might look like. If you search for romantic, like movies, let's say you are literally making posters for a movie. And you might say, oh, what kind of, you can look for settings in cities, sitting, urban settings, what kind of colors did they use? What kind of scenes have been used in that domain. And you can almost see in other cases where if you have a very, very large data set and that's disorganized, you know, you used to have to go through the process of organizing that yourself and putting manual tags on it, have what have you. But if you just have 100,000 unorganized family photos now, you can just say, tell you what, find me pictures. Oh, I know I was playing basketball with Jim like, you know, a couple of months ago. I don't know where that picture is, but I could look for that. And that applies to basically any domain where you might have a data set that you don't want to label. And you certainly couldn't label for all the potential use cases. So you could do that in your marketing domain. And yeah, really, the images is really one thing that I find. Takes a lot of time to label and organize and searching through that with similar images or just keywords that you're interested in is really useful. And you see more examples of that the next, I think in the worksheet.

Yeah, certainly. I don't know anyone who goes through with every photo they've taken and like, writes a text description of like, what it involves. They can later go back and Find that, that photo and yeah, certainly those, some of those shopping use cases seem pretty cool. Like I can certainly imagine you see someone on the street, like they got some cool clothing, take a photo and then you can look up what that clothing is, go and buy it yourself. Yeah, lots of cool options there. All right, so next question from. Comes from Mohammed. So Mohammed asked, what are multi languages? So if you're not just limited to English, you want to speak in French, Arabic, maybe any other languages, how can you do search using different language types?

Yeah, that's a really good question, Mohammed. Thanks for that. You know, that's really good. That's a really good opportunity for me to actually say that this model is multimodal and multilingual. Right. So the original Clip model, which I have the blog post here from OpenAI, this is when they announced it, they had a cool blog post. I would recommend that you have a read of it because it's quite approachable. Wasn't multilingual, so it basically spoke English and it spoke image. Right. So that's what the two languages it spoke. But the improvement, that's one improvement that's been made by the lovely folks at Sent who build the Sentence Transformers models is that they've improved them to be multilingual. So you don't have to translate your French query into English or Japanese. I speak Korean, any of that. You can just have a one data set and just look them up in your native tongue or text. It's really, really cool. And just while I'm on that topic, some of the text vectorizer models like Cohere has this really cool model that speaks, I think 100 plus languages and that's really amazing.

Okay, so you don't need like an initial translation large language model to get to one language and then have a second model for interpreting images. That's cool. This all built into the same thing.

Yeah, absolutely.

I guess on that note, so we're using Clip here, but have a look on like hugging face or something. There's just hundreds and hundreds of different models available. So how does Clip fit into a sort of larger ecosystem of these models?

How much time do we have? That's a really good question. And people do ask that question a lot. Not just for multimodal models, but also for text model, you know, vectorizer models itself. I think Clip is a really, really good starting point. And I think with these models and with these vectorizers, my recommendation is to get started with a popular model. So with text that might be, you know, Coheres, Vectorize a model or OpenAI's or you know, what have you or one of the more popular sentence Transformers models. And with a multimodal model, I think Clip is a really, really good starting point because it was much better really than the competition when it came out. And it's well known enough so that you kind of, you know, you kind of know what you're getting. I think that's a really good point and that's a really good segue. Richard, to me to just mention this model called Bind or Image Bind. And this is, this is an interesting one because there's no like PIP package to install this with. You kind of just if you want to use it, you have to clone the repo and install it manually. But it does speak these different modalities. So it speaks images, text, audio, depth, thermal and IMU data, which I believe is accelerometer data. So if you did have, you know, I don't. But if you have a bunch of thermal images that you want to put into same embedding space and search through, you might have a lot more audio that you want to search through. You can do that with this model. So some of your requirements will definitely dictate your model choice. And just like with text vectorizers there are domain specific fine tuned models. Right. So for text it might be like medical or patent specific models that you might choose. But my default advice really is to go with a popular one because they're kind of popular for a reason.

Okay. I'm imagining all sorts of like paranormal investigation use cases of this, like you know, the ghost hunter TV shows as, ooh, it's gone cold here. The night vision cameras probably more likely to be used by I guess plumbers or something like that. Maybe like construction people.

Yeah, I think, yeah, yeah. I'm just imagining my next plumber and asking them about the multimodal embedding model.

That's the thing. AI is everywhere. You can't get away from it. Okay. So yeah, I imagine there's going to be some layers of applications just to make it easier to use before like your plumbers start using this sort of stuff. They're not gonna be cloning GitHub repos.

Yeah, I mean, you know, I think that's the thing about models like this is that I think there's definitely an aspect of they feel more like research projects than others. Certainly the ImageBind model. So I think Clip is been long enough and it's been kind of productionized enough by different places. I would feel safer recommending Those.

So some of these content types I hadn't really thought of as being things that you might want to search. Are there any other weird content types that are available to work with these models? Like some models doing something odd?

Oh, that's a really good question. I often wonder, like, so Zen is a resident data scientist who I, you know, always question about these things and I go, who's got a library of accelerometer data that, you know, I'm searching through? But I guess if you have, if you're doing research on things like car state, like vehicle safety, you would have a lot of that data and then you can perhaps correlate that with different types of impact or different types of images and outcomes and things like that. So I think those would be really cool. And one of the things that's been really interesting to see from my uninformed perspective is what's called mixture of experts models, where a model is actually six different, 10 different models under the hood. And I think it's going to be interesting to see how those tie in with the ideas like this of multimodality. So you don't have one model doing one thing, but if you can have one model under the hood saying, oh, actually I'm going to now rely on this part of quote unquote, the brain to do these things. So, yeah, I think it's really exciting.

Okay, that mixture of experts idea sounds pretty interesting. I'm not really that familiar with it. Do you want to expand on what this involves?

Yeah, let me see if I can do some live Googling. So there are these open models, like open source language models. Sorry, I'm very not good at reading and talking. And some of them, like this mixtral model, relies on this idea of mixture of experts and basically the idea that it's very broadly speaking, instead of having one large model that just tries to do everything, it tries to be good at math, it tries to talk to you about coding, it tries to do whatever. What happens with this mixture of models is that it's actually under the hood multiple. If you imagine little elves or helpers under the hood. So that when you ask the language model to say, hey, can you debug my code? It'll go to a different part of its model architecture or this model model than if you asked it to do something else, like correct my. Can you, you know, proofread my essay or something?

Okay, that's kind of cool. I suppose, like when you think about human brain, you've got lots of different subsystems. You got like the vision and hearing and then stuff, I don't know, emotions and thinking and hunger or whatever. And so your brain is actually lots of different experts just sort of clumped together inside your head. So I guess this approach does seem make some sort of sense when you relate it back to like even human biology.

Yeah. And it's a really intriguing idea of you don't need to get one thing like build one thing that can do, you know, 45 different things. Right. If you have like an organization, like you know, a company or whatever, you don't have one person doing everything too. So you don't need to build a kind of monolith that does everything. It's really cool.

Okay, nice. And are there any domain specific models for specific use cases?

Yeah, I'm not. So at least in terms of the text models, there are very domain specific fine tuned things. So I've seen medical patent based things or whatever. And of course when you get to the classical machine learning models, there are these classifiers that are like say hey, this image is a cat or not a cat. Right. So you can fine tune kind of models to do that. Like whether it's a language model or an embedding generation model. But as far as open source ones, I'm not aware of too many like that in the image domain, but I'm sure there are out there. And fine tuning models is kind of becoming more and more approachable or sort of accessible. That's the word I'm looking for. So if you have a, if you have your data to do so, it certainly is becoming easier. Okay.

I'm sure there's going to be like some medical thing where it's like, oh, we're looking at like similarity between proteins or something or similar between chemicals. There's got to be something along those lines. Maybe task for the audience is see what you can find out there. All right, so I think. Last question for me before we wrap up. So in this case all the data was the same type in this goal, so it was all images. What happens if you have different types of data together? So maybe you have some audio, you have some text, some images, some video, some, some depth, whatever it was, or accelerometer data, you got all these different types. How does search work then?

It's basically the same way. So I guess in. I've got a few more queries in this example. But you know, if I ran this query, this is the one I was referring to with the larger kind of text. So I've got additional images in here from the 80 20k data set, which is like 2020 or 20 odd thousand data sets. So you can see things like, oh, if I search for a crowd of people, I get these images and they're kind of not very discriminated, discriminatory. And then I go, oh, what about a poster of crowded people? Now I get more of these images, posters and different posters of an action movie. So you kind of can drill them down a little bit more and more. But if I had ingested actual text objects into the database, some of these responses would be text, some of the responses would be images, and if I had audio, some of these responses would be audio. So that's the cool thing about these multimodal embeddings is that once you have the data ingested into vector format, you just get the best match. Now if you search using an image, you're probably skewed more towards getting images back just because of the way the embeddings are done. But at the same time you're not necessarily only going to get images back, right? And it does make kind of more sense that if you're searching using an image, you get more images back than text and vice versa. So basically all it does is just all these models do is just put them into a mixed bag of objects, catalogs, if you will. You know, going back to our supermarket analogy, and you just get the best results back and really it's just a function of what your input is and that makes them really, really cool. So I thought I would mention one more thing which was using a multimodal rag. And here I've got an example where I use the OpenAI API to ask GPT4 Vision some questions. So I won't spend a lot of time talking about what I'm doing. But basically, if you've used GPT4 API before, even ChatGPT, the idea is the same, right? You ask it questions or prompt it with your query and it gives you some answers back. But here what I'm doing is I'm running a search like this on weaviate and then I'm actually feeding that back into a language model, like including the images, and I can ask it questions. So I can say, hey, get me some science fiction movie posters set in space with an interesting vibe. So here are the posters that I found and then I can say, oh, what themes do I have? Oh, that one's. And then I can ask it things like what elements do these posters have in common? What might make them successful posters? Describe them, blah, blah, Blah. And you can actually start to see that it's really broken down each image in detail. And going back to our earlier discussions about particular use cases in different domains, these models are now sophisticated enough so that you're not just getting a general idea about these images. It's able to process each image, understand the elements in it and give you back, you know, feedback. So if you had, let's say 15 images of your marketing last latest marketing collateral, you could ask it your model questions and say, oh, what might be some each potential shortcomings of this? How might we break away from it? It would be able to understand and process those queries given the context of the images that you're passing to it. So yeah, I would encourage you to check that out yourself. This is API based, so this would be relatively low compute, have relatively low compute requirements too, if you want to just pass some images through to it along with some prompts. And multimodality is really, really cool, both for search as well as for prompting.

All right, super. Thank you for that. This has been such an informative session and I'm glad we had a chance to have a good chat about this. So lots of great concepts covered there. Now, before everyone dashes off, we've got some very exciting webinars for you coming up. So later this week tomorrow we've got Naomi Robbins, who's a bit of a data viz legend, so she's talking about creating effective graphs. On Thursday, we've got three very senior people from pharmaceutical companies talking about how you make use of data and AI within the pharmaceutical industry. And then next week, on Tuesday, the next code along is on getting started with data pipelines. So please do go to datacamp.comwebinars register for those. I hope to see you all again in future sessions. So thank you once again. J.P. uh, it was really great to have you here.

Thanks very much. It was lovely. Brilliant.

And thank you to Reese Moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today. See you all again in.