Building Multi-Modal Search with Vector Databases

Sa.

Hi everyone, my name is Diana Chan Morgan and I run all things community here at Deep Learning AI. Today we are so excited to host an amazing workshop about leveraging the power of vector databases like weaviate in conjunction with multimodal embedding models to power at scale production ready applications capable of understanding and searching text, images, audio and video data. What all of you can expect to take away from the workshop today is how machine learning models can embed multimodal data. How vector databases like weaviate enable real time semantic search. How vector databases can be used to scale the use of these models to billions at scale code implementations of performing any to any modality search applications enabled by at scale multimodal search and retrieval. So we will be dropping a link in the chat where you can ask all the questions and vote upon which questions you want answered by the speakers. This session will also be recorded and the slides will be sent out afterwards. And to introduce our event partner, weaviate. Weaviate is an open source vector database. It allows you to store data objects and vector embeddings from your favorite ML models and scale seamlessly into billions of data objects. They just released our short course with us last week with Sebastian, one of our speakers. Vector Databases from Embeddings to Applications. I'd love to introduce our first speaker, Sebastian. Sebastian is the head of DevRel at Web8 and an EX Expert and Vector database technology. He's passionate about helping developers build and productize AI based applications that take advantage of the latest cutting edge developments in machine learning. Hey Sebastian, happy to have you here today.

Hey Diana. Yeah, thanks for having me. Yeah, I'm super excited about this session, so. Great.

Absolutely. And our next speaker is Zane. Zane is a senior developer advocate at weaviate. He's an engineer and data scientist by training who pursued his undergraduate and graduate work at the University of Toronto building artificially intelligent assistive technologies. He then founded his company developing a digital health platform that leveraged machine learning. He's passionate about open source software, education, community and machine learning. Hey Zane, thanks for joining.

Hey everybody. Super excited to be here and really looking forward to this workshop. Getting our hands dirty with multimodal data performing search over it. Couldn't be more excited to be here with you guys.

Absolutely. Well, I'll have Sebastian start to take it away.

Perfect.

All right. Okay. So we thought that before we even jump into multimodal, might be good to split it maybe into two parts to give you a little bit of intro into vector search and kind of like the ideas behind it and kind of like basically warm you up to the main portion of the session, which zen later on we'll deliver. But before I get into that, like Diane mentioned earlier, we launched a pretty cool course with Andrew, which is basically titled Vector Databases from Embeddings to Applications. So please enroll. I'm super proud of working with the deep learning team on this and with Andrew. So yeah, the more the merrier. So super excited about this. So let me do a quick intro now to like vectors, vector search, vector databases and all this kind of stuff. And trust me, I'm going to take just few minutes of slides because I actually, I prefer to lead in the code and that's going to be like my favorite part of the session. But what I want to talk about is like, and I've been talking about this actually for years, where in the world? Or like at least on the Internet, everything we do starts with search, right? Whether you want to look for music or you want to watch movies, or you do shopping or you're looking for information, all of it starts with search. And we've been doing that for decades. So what's the problem, right? Everything fine. Search works. The thing is that with some of the classical methods, there's some challenges with traditional search, if you run a query like why do airplanes fly? You may get the response like this, which basically talks about like, hey, you know, flight expensive error. Because it does this, this and that, but doesn't really explain why do airplanes fly? But it matches on keywords, right? Like technically that should be the right result, but practically it doesn't answer the question. So yeah, we asked for one thing, but we really got like a completely different thing. And it's more of an ad in this case than the answer to our question. Where semantic search comes to the rescue, it kind of looks at the whole thing from a different perspective. It's not about like matching the words that are in our content, but about matching the meaning. What is it that we mean by this question? And what is it that we need? And with semantic search, you are more likely to get a response like this that maybe come from NASA that basically explains that to make air move faster over the top of the wing, so and so happens, and this is more of an answer, and this is not about matching the keyword, but what's that we are looking after? And in summary, I think this meme explains pretty well, like with a lot of cool stuff that is happening in the ML space, we're getting like amazing new tools of doing our jobs better so how does this work? So let me explain to you, like maybe not a 5 year old, but a little bit like a 10 year old, like a little bit of math might be required, but not too much. And the idea is like this. So we have those amazing machine learning models that can take content of any type, pretty much depending on the model. And then if you fit it in, what you get back on the other side, you get a vector embedding. And basically a vector embedding is like a bunch of numbers, an array of numbers. That is the way of machine learning understanding the meaning behind the original data, right? And if we took like our original examples where we had like the two responses and let's say run it through the machinery model and then take the second one, would get like a bunch of different vector embeddings and each of them will be already somewhat different. And if we then took those and moved it into like a multi dimensional space, what you would probably notice very often is that the vector embeddings that represent the similar kind of data will be stored together. So like what we're trying to visualize here, like a cat and a dog, they'll be a lot closer to each other. Like chicken will be somewhere somewhat further away. While for example, things like apples and bananas will be in completely different space. And this is a simplified version of how those vector embeddings look because we can only like imagined spaces in like three dimensions. But those vector embeddings go into like a thousand or fifteen hundred or even more dimensions, which basically means that it can capture more information and more meaning. So to summarize the whole thing, of how the whole thing works and how the query works is as follows. So if we took our original query, why do airplanes fly? And we already have our vector space embedded with our data. If we then feed in that query through the machine learning model, we'll get a vector embedding again. So this one actually represents our query. And now from here we can map it to our vector space. And basically the area around here that we're pointing to is probably where the most likely article or answer is to our original question. And very likely we'll find our NASA article finding it. And this is how basically it works in a nutshell. So let me show you how all of this works in practice. So I have a notebook that is prepared and I think we can also share a link to it. So I'd like to walk you through this. So a little thing. So what I'll be using, I'll be using Web8, our open source vector database, which is pretty awesome. And recently we were working on a brand new Python client which changes how we interact with weaviate. It's still under beta, but I think it's so close to be released that I thought it was ready for us to actually show you it in action. And I would love if you were actually following encoding along with me as I go through this. So of course, as one of the first steps you have to do is go and install the client and then this will basically install the latest version of the VA client, which is still under beta. So that's why we need this like four star. So once we have that, basically there's different modes or different ways to deploy weaviate. We could use weaviate embedded, you could use self hosted. So self hosted is like. Could actually run it with a Docker composer. And then we also have a cloud deployment. The cloud deployment is not compatible just yet with the latest V4 client. It's going to come next week. So for today I'm just going to focus on running this on my machine. If you want to use Embedded, then you can follow me along. If you prefer using Docker Compose, there's already a Docker file in here that if you're familiar with Docker, you could just go and kick it off. But I am going to stick to this one the way it works. The thing that we need to do first is this where you need to import weaviate. I'm also importing OS because I don't want to actually share my keys with you in plain text. But usually the idea is what we need to do is just call something like Weaviate Connect 2 and then basically you can say either connect to embedded or connect to local. And for the purpose of this presentation, I'll be using OpenAI and Cohere. So if you want to use one of those, you could just replace it here as a plain text. Or it's better if you use Environ variables and that should be it. Right? So if I run this code. Oh yeah, I need to connect to my jupyter notebook. Fine. What this will do the first time you run it, weaviate will go or the client will go and download a binary, like an embedded binary from weavit for you and then you run it for you locally on your machine. So it's pretty much no instance required for as long as you have the weavit client. So I already have an instance running and from now on I can interact with weaviate through this client. Right. So as part of this demo, I want us to use a couple of data sets. They're super simple. So one is like just 10 objects and then it has things like category and question and answer. Right. Like with some content for us to play with. And then the other one is got like a thousand objects. So we can run like a second test and it's got like some additional properties as well. So we'll get to that. So if I run this code, basically what we do is just like load the data from given a path and then convert into a JSON. So this is like a previous one locally of one of the objects. Perfect. Okay, so we have our data pretty much ready. So what we need to do is create a new collection with let's say classical databases. A collection will be a table. In our case we call them collections because they're not actually tables underneath. So what we do is call collection, client create. Okay. Sometimes I need to reload my VS code and reconnect just to get my tooltips showing. And hopefully you'll be able to see them too. Let's see. So yes, there we are. So basically as part of this, in order to create the most basic collection, we just have to have a name, right? Let's call it Questions. And then if I create a collections like this, it will be one of those collections that you can bring in your own vectors. But in this case I want to take advantage of weavit's module system which allows you to which actually can help you vectorize your data. So what I will do, I'll add a quick import weavit class called weavit classes, where we have a lot of helper classes that help you actually write your code a lot faster. So if I just follow this, I want to configure my vectorizer and I want to use text 2vec cohere. Right. So cohere. They have like an amazing multimodal model. I'm just going to use the default one and let me call it just Questions. Fine. What happened here? Oh, I was playing with it earlier. I already have this collection. So let me. I should have cleaned up my environment, but that's fine. So we can do very quick one. Like if the collection exists called Questions, then we can just delete it.

All right.

Iam collections delete. Fine, let's rerun this. And that basically creates like a brand new collection for us. And then from this moment, in order to add data into with it, I can just go into my Client collections. And what I will do is get our questions kind of object I can call it, let's say I call it Q. That's called questions. And then I can say questions.data. insert many. And you remember earlier we loaded this data, right? So I'm going to just grab this 10 objects and then straight up insert them here. And then something interesting is going to happen. So as part of this, what we do is insert the data. So let me show you a quick preview. So response. And this get. This gets actually pretty interesting. So response questions the query. And then maybe I can fetch some objects and I'm going to grab let's say four objects and if I just print that response, could see like we already got like a bunch of objects that we got back. And then I can actually drill in specifically to objects and maybe just drill, just print the properties of the first one. Right. So this is one of the objects that we got in. The other interesting thing is, and you're going to like this so often, we need to kind of like take care of vectorization. In this case, let's call item. The vectors are already there. So what I can do, not client queries, queries, questions, not queries. Got questions. Query, touch object by id. And then I'm just going to grab One of the IDs that we had from before. And what I can do is say include vector. So this object already has a vector, but I can print it like this metadata vector. And you can see that's like that this is the vector of the first object. So the important thing to understand is when did we get this vector, how did that happen? And it actually happened here. So when we went and decided to insert this object, we went like, okay, I'm going to add all these objects into the database, but also I'll go and use Cohere to vectorize, since that's how we define the collection. And that's kind of like the idea. The other thing is, and let me show you like a different example. So I'm going to recreate this collection. Oops. And this time I'm going to use OpenAI. So let me just go text 2vec OpenAI. And technically, I mean if you want, we could even just specify like a specific model, like ADA version, model, version. Yeah, and then kind of do your two, etc. Etc. But that's kind of like the model we use by default anyway, so I'm just going to skip that for now. But that's how more or less we would do it. And we could also add like a generative config. And this generative config is interesting because that's how you can for example and we have like support for OpenAI Palm Cohere. You could also run on Azure, but in this case I'm just going to use OpenAI and we could even just specify like which model we want to use. Like we could say hey, I want to use GPT4, etc.

Etc.

Right. And so let's recreate this collection and that basically can load the data. Sorry, that's like a brand new collection that will work for us and let's add a new one. So this time if I import, I can for example try importing again. So it's pretty much the same code, it's going to work with 1000. Technically if you have a bigger batches of objects, if you have a million objects, I wouldn't necessarily recommend running just insert many and just pass in a million objects because you can run all problems. But in this case for 1000 of text objects I should be able to just run this. Then this actually takes 1000 objects and then vectorize them using my API key and then those objects will be pretty much available to query very soon and I can even verify client Oops. So okay, so we have our questions collection here again. So I oops questions, I can even very quickly verify how many objects we have in there. So let's see, let's just print the response that should give us. Yeah, so we have a thousand objects there that should be all vectorized, etc. Okay, so this is the first part, this is how you kind of get in get the data into weaviate and you can perform all sorts of operations on it. Right? So if you have questions data you can delete objects, you can inserting of course, but like you can update objects, you can, let's say I could grab 1 by id and then delete it, et cetera, et cetera. So that's kind of the idea. So the second part is querying and this is where things get even more interesting. So I just need to reconnect to our client because this is a separate jupyter notebook and hopefully I will get like all the reports. So I am going to use the same code as before where we can connect to our questions and using the questions I should be able to say query near text. And what did we have in our data? So we have like this object vectorized, right? So so maybe we could search for something like death or. Yeah, a bit of A morbid one. Yeah. Let's not look for Abraham Lincoln. Lincoln. But yeah, like we have something about rhyming pigment. So it's for now just search for pigment. Right. And this should do the trick. So wait, that's not the right thing. Okay, so query equals exactly pigments. And we want to return maybe the first five objects and then maybe you can print response object zero. Technically I should do like a loop, but I'm lazy. I find copying like this faster. All right, maybe not. That was a terrible way. I should have done a loop. All right, let's do a loop.

Okay.

For item. In this font object. Yeah. Print item properties so then we can see what we got back. Okay, so this basically code here where I spend more time on printing than actually the query. It, it shows like how you can run a vector search that uses this as the query and then we get this response back and then we could look for color and then you search through that as well. There's as well like something. Yeah, and you can see like reference to color green. We could have like, we have wax. And this is interesting. I can actually search a query using in different language. So in Polish or wax is vosk. And we should probably get something like, which is air. So let's see what sort of. Oh, we're using open AI. Not coherent anymore. But it's. It's still kind of like a find information that it's. It's sort of relevant, like musical instruments. And then it finds like a durable wind instrument, et cetera. Right. So it finds info on all sort of stuff. So that's how we can run like a vector search using just text. We could also add a filter. And so we could for example, say find something where value is. Let's say because value is like maybe number of points at least 500. Right. So in this case, to do that, let's import we beate. This should be Python, not markdown. Okay, fine. I'll add a new code. Move it here. Okay, so I'm going to import weavit classes again as WVC and to add a filter, it's actually pretty straightforward. So we can say filters and we are searching on value. Okay, so search on value. And we want something that is greater than 500. And then this time the results we're getting back are at least 500 or more. Well, they're more than 500. We also can use hybrid search, which is a pretty interesting concept where basically instead of like new text, we could do hybrid and hybrid search. What it does, it combines A keyword search using BM25 and vector search into one query. And we have this alpha parameter. So if alpha is 1, we basically say we are going to use 100% of the score from the vector search. If it's 0, we want to use 100% of the keyword search. But maybe we want to say, hey, I want to use zero, like 70% of the vector search versus the keyword search. So in this case we will try to both match the keywords and the vectors. And that also runs and performs a search. Where hybrid is useful is when maybe the model is not necessarily trained for the type of data that you're searching for, but you still want to be able to match exactly on something or maybe like you have serial numbers or all sort of other things. And the final one that I want to maybe show you, which is pretty exciting, is retail augmented generation. So as in here, you saw when I was creating the OpenAI collection, I added this GPT4 to that collection. So what we could actually say is use a different query. So it's generate dot. Yeah, near text. That's the syntax. So in this case, what we can do is say like, okay, I want to get, let's say four objects and then I can add a single prompt. And then the prompt can be say something like explain or write a short tweet about. And then we want. Yeah, about the question. And I think that should do the trick. So when I run this. Well, it basically what we do, it runs in two steps. First it runs the query and retrieves those four objects. And then after that it passes basically the question value together with the rest of the prompt and then generates a response. Where did that go? Okay, so we should. Let me just grab. I'm just gonna go to documentation. That's usually the safest thing. So with the generation on single prompt. Oh yes. Instead of going for properties, I should have printed generated on the item. Oops, not on the on hybrid. Fine. So I just need to add this. So let's rerun it. You only took six seconds last time, so it shouldn't be too bad. And there you go. So basically this took the original information from our question property and then turned it into a tweet. So for each of those, there's another one that is called group task. So maybe I'll just do very quickly. So group task, basically, instead of doing it per object, it can basically generate a single response for all of it. So we could just say like, hey, explain what this content is about. And then in this case, the response should contain just generated already as a one global thing. So one last code example and that should kind of take again, like the this find like these four objects and then create like a single summary across all of it. So this content is about the collection of questions and answers from the game show Jeopardy. Each entry includes the air date, episode, and then it goes on and on and kind of like explains what this content is all about, which is actually pretty awesome. So, yeah, I hope you were able to follow through. I know I was like, flying through it, but I guess like, one of the benefits of watching it, especially Maybe later on YouTube, you may be able to pause at the right moments, et cetera. But yeah, I hope this gives you a pretty good basis to the kind of things we can do. And then hopefully I can pass the baton to Zen. But yeah, if you want to learn as well about weaviate in our documentation, you can definitely go to Weavy Docs and start with the quick start, because that also takes you through a similar workflow. But yeah, that's me for this part of the session.

All right, thank you, Sebastian. So let me share my screen now. Go ahead here. All right, so that should be coming up. Sebastian, can you verify you see that?

Yes, I can see part two, Vector databases.

All right, we're in business. Okay, so. So Sebastian spent the last half an hour talking about how you can get started with vector databases and how you can use them to do this semantic search over text. And he even showed you this idea of sending the output of the vector database to a large language model for it to reason off of. And I was having conversations with people in the chat, talking about the unique applications of vector databases and comparing them to other types of databases. One of the most unique aspects of vector databases is exactly what you have in screen here, the fact that they can handle a lot more than text. The last 30 minutes we have together, I want to spend talking about going from text data, understanding text data, searching over text data, to doing a lot more than that, understanding audio data, video data, images, and then searching over it. And if we have time, at the end, I'll even add a little bit of multimodal calling. The new the GPT4 Vision model for it to understand the images that we're retrieving from our vector database. Let's start off from fundamentals and let's see how we can pass in more than just text to our vector database. What we really need is a model that understands multiple types of data. What I've Got on screen here is not just a sentence, lion is the king of the savannas, but also the image of a lion, maybe the sound of a lion roaring. I've got a video of lions running around. We need models that can understand each of these modalities and can then convert these into corresponding vectors. And the trick here is that because the input data points, the types of objects that I'm passing in, are quite similar in meaning. I either have the image of the lion or the lion roaring or a sentence describing a lion. The corresponding vectors should also be similar. And this goes back to what Sebastian was talking about, where no matter what type of data you pass into the vector database, if it's semantically related, if it has a similar meaning, the vector data points should be close together in vector space. And so, as you can see here, the vector for the sentence is quite similar to the vector for the actual image of the line. And that is also similar to the sound of the lion roaring. And in order to do this, you need to have models that can understand each one of these modalities. There's a couple of ways to do it. One way to do it is to have one model that understands each modality separately, and then you unify them. And so once you have this capability to understand and extract meaning from different modalities, then what you essentially have is one unified vector space. Regardless of what type of modality you bring in, if the model has been trained to understand and embed it into vectors, you can now have this unified vector space where your videos, images, audio files, text files all live together. And this translation from human understandable versions of data, videos that you can watch, audio files that we can listen to, translation from this human understandable version of data to machine understandable vectors still preserves the meaning behind the vectors. It still preserves the meaning behind the data. The images of the chicken and the video of the chicken and the sound of a chicken should all be in close proximity in vector space. This is where things get really exciting, because now you can literally take all the files on your computer, whether they be video images, audio text files, and you can embed them into the same vector space. And the reason why this is pretty exciting is because now once you have the ability to embed all of these files into one unified vector space, you can now do any to any search. And so what do I mean by any to any search? You can take any of these modalities and you can turn them into questions. Let's say you want to return media that is similar to the concept that's encoded in this sentence, or it's similar to the image that I've shown here or the video that I've shown here. All of these can become queries, as Sebastian was showing earlier, and they can go into your vector space and you can perform vector search using these multimodal queries. And not only that, but because you have images, audio, video in your vector database already, what comes out can be any of these data modalities as well. So you can retrieve audio files, you can retrieve images, video files and text documents as a relevant object to any of these modalities that you pass in. This is quite unique because vector databases, because they represent data through vectors, it doesn't matter what the original modality of the data was. If the machine learning model understands it, then it generates a vector and the vector database is optimized to search over it. And so, with that short intro to multimodality, let me show you how you can take what Sebastian taught and you can use that to build up a multimodal search engine using weaviate. So this is part two of the workshop. You can also get access to this notebook on the same repository that Sebastian shared. There's a second folder there. To set this up, we're going to be using Docker. So what I would recommend is go ahead and if you don't already have it, download and install Docker. And then once you've got that, you're going to go into your terminal, you're going to get the Docker image. This is going to give you a Docker file, and then you're going to go ahead and run that Docker file, you're going to compose it, and then it's up and running. So I've already got this up and running. So as you can see here, over here, I've got this up and running. I've got the deep learning AI environment running, and it's got both my containers up. We'll see the utility of both of those in just a second. So let me go back here. That will be as much Docker as you need. There's a couple of other dependencies here as well. So Sebastian has already set you up with this one, the new Weaviate Python client for this notebook. You'll also need the OpenAI Python API. So we're going to install those, and then we're going to go ahead and connect to our locally running version of weaviate. So in this case, this code is very similar to what Sebastian showed you. I'm going to go ahead, because Docker is running in the background. I've got weaviate running locally on my computer. I'm going to go ahead and connect to it, verify that it's connected, and then I'm going to go ahead and get some metadata about the instance that's running. So here, one thing that you'll notice is that I've got this running locally, and then the module that I'm using is a multidevec bind, and this is particularly important. This goes back to what I was talking about earlier, where in order to embed and store media files, you need to have a model that can translate those media files into vectors. And that's exactly what the module system with weaviate does. Right. So the multidevec bind is one of our modules and this module. So I'll go to the documentation here. This module allows you to pass in any one of these modalities that are listed here. So most importantly, text, documents, images, video and audio. And that's what I'll be showing off today. And all you need to specify is in your dockerfile, you need to specify which module you want to get up and running. We can see that we have the correct module, you've got the correct version of weaviate, so we're all good to go. The next thing we're going to do, as Sebastian showed, is we're going to create a client. So here we have a quick check that sees if the Animals collection is already there, delete it, and we're going to recreate it from scratch. This is very similar to Sebastian's code, where I'm going to go ahead and create a new collection called Animals. And then I've got code that's slightly different here. So what I'm going to specify is a vector config, because I need to tell weaviate how to vectorize my multimedia data once it has passed it. So here I specify that it wanted to use the multidevec bind module. This is how I can pass in audio, images and video. And specifically, I need to tell it where these files will live in the database, so which properties these files will be called from. So the audio fields, I tell it that the property name will be Audio, the image fields will live in a property name called Image, and the video fields it can access through the property name video. And this is quite important because if you think back to how multimodal embeddings work, actually underneath the hood in ImageBind, we have multiple models, and it needs to understand how to take the correct modality of data and route it to the correct embedding model and then generate the embeddings. So here we set it up for success by telling it where to access the right types of files and vectorize them accordingly using the right model. We're going to go ahead and get that running, and that creates our collection for us. We've got a helper function that you'll understand the importance of in just a second. Anytime you pass in multimedia, audio, images, video files into weaviate, it needs to be base 64 encoded. So this is going to help us do that for all of our file types. Then we're going to go ahead. So I've got a repository of images, audio files and video files, and I'm going to insert them into the collection that I just created in real time. So this is quite similar to the code that Sebastian showed as well. What I'm going to do is loop through all of my image files. I'm going to add them to a list, to an empty list, a Python list, and I'm going to pass it the name of the file, the path on my computer where that file resides, and I'm going to pass it the actual image. So I'm going to base 64, encode the image, and then I'm going to pass it in. I'm also going to tell weaviate what type of data I'm passing it by giving it a media type property. Over here. We're going to go ahead and take that list and we're going to insert many objects here. As this is going on, what's happening underneath the hood is weaviate gets the image. It knows that I wanted to use the multidevec bind module. It's going to pass that image to the inference container that's running on my laptop locally. It goes ahead and vectorizes all of these images, and then it returns and it tells me successfully, I've added all of these objects to weaviate, and these are the unique identifiers for every single object. In order to make sure I like to be vigilant, I'm just going to go ahead and run a aggregate query that shows me whether all of the objects have been added. Here I've got nine image files. I can run this aggregate overall query that'll tell me how many objects I've got stored in my database right now, and that checks out so far. So we can now keep on going. I've now got images in my database and I've got Vectors for those images as well. Now I'm going to go in and insert audio files. So in this case, my audio files are also. So here if I show you, I've got about, I think, six or seven audio files. Yeah, I've got six audio files, everything from birds to apes to roosters, dogs and cats. I'm going to take each one of those files, I'm going to loop over them and I'm going to add them to the database as well. The only thing that's different now is that I have to specify the right property because the multidevec bind module understands that these are audio files. These because of the name of the property, I have to use the correct property name here. Outside of that, all I'm doing is looping through all of my audio files. I'm adding them to a Python list and then once that's ready, I'm just going to go ahead and insert all of them into weaviate and behind the hood. This is actually going to use batching for a small amount of files that's not that relevant. But if you're inserting millions of files, then that becomes a lot more relevant. So I'm going to go ahead and run this. It goes through the exact same loop, except now it's going to use a different model. It's going to use the audio model to perform inference, create the vectors and then store the vectors into weaviate. Now all these vectors are stored into weaviate as well. Just to do another quick sanity check here, we had nine images and we've got six audio files. We've got a total of 15 objects in the database stored altogether now. So now comes the interesting part. I'm going to take video files and I think I have four or five video files here in my repository. I've got six video files here. Similarly, concept of animals, I'm going to loop through them and I'm going to specify that I'm inserting videos. Now, everything else is the same. I store the name of the video, the path to the video where it's stored. I and then I tell eviate this is a video type and this is mainly for filtering later on when I want to do query. So I'm going to go ahead and run this and this one. As you'll notice, I'm doing something different here. I'm inserting one video at a time as opposed to inserting all of them together. So I'm not using insert many, I'm using one video at a time. And the reason for that is because video files are large, it takes time for the model, especially running on my Mac, to generate the vectors for those video files. So I don't want the database to timeout. I want it to know that everything is okay. It just takes longer to create vectors for these files. So I add them in one by one. And what's happening underneath the hood here is it takes the video file and it passes it through the video model. Actually an image bind. The video model is very similar to the image model. The only difference is that it looks at per second of video and then it breaks it down into frames. It generates vector embeddings for those frames and then it, and then it combines them to output 1, 1 single vector for every, for every video. So as this is happening, I can talk a little bit about the next steps here. So now that I have my database, I've got vectors for each one of these types of modalities. So I have the video of the Meerkat Digging cat playing. I've got the audio of these animals and I've got the images of these animals all living in the same vector space. And what I said that this enables is the ability to do this any to any search. And that's the really exciting part because now I can pass in audio files as questions or queries and I can get back the most semantically relevant data points. So I can get back images or video files. And that's really easy to do in bv actually. It's a single query where you can pass in the file that you want to use as a question and then you get back the relevant objects. You can control how many objects you want. You can control which properties of the objects you want as well. Right. So if I just want to the name of the file, the path of the file and the type of media, I can specify that and it will only give me those data points. We're almost done here. We're just vectorizing the last video, by the way, I should say a couple of words. So if you're using this in production, what I would recommend is running the model, the inference model, the large embedding model on a computer that has a lot of GPU so that you can inference a lot quicker. And then I would recommend running weaviate on a, on a computer that has access to ram, because that's what weaviate needs. So as we've been speaking, I've inserted my six videos in here as well. So now we should have a total of 21 objects, right? Nine images, six audio files, and six video files. So if we do a quick sanity check here, we've got 21 objects in here belonging to multiple modalities, all living in the same vector space. So now I can go in and actually print out everything that lives in my. In my vector database. If I wanted the unique ID for these as well, I could loop through it, but I don't really want those. So I'll just take out the name of the images and I'll print those out. As you can see here, I've got all of these different multimedia files in the database. So now I'm going to. I've got a couple of helper functions here that allow me to visualize these multimedia files, right? So I'll be able to play videos and images directly in the notebook. And I've also got a helper function here that I can pass it a dictionary and it will visualize whatever modality is in there. This is also one of the reasons why I had the media type encoded in the database as metadata, because I can then parse it out and then show you images, video and audio data points as well. I'm going to run these helper functions here. Oh, this one I don't. I'm just going to run this, but we've already got the base 64 functions there as well, so that should be. That should be fine. Okay, so now we get to the searching portion, right? This any to any search, which is probably the most exciting part. I'm going to go ahead and point a variable to the collections animals collection and then we go in and we can query this. And the interesting part here is because we're using a multimodal module with weaviate, I can go in and do any type of search. I can search with audio files, images, I can search with text files, with video files, or let's say somebody just brings me a vector and says I want to know what object is close to this vector. I can also do a with near vector search and I'll demo all the multimodal searches that I spoke about earlier. So here I'll do a near text search, and in this one, all I need to do is pass in a natural language query, as Sebastian was showing earlier, right? So if I search for dog with stick, it's going to go in, it's going to vectorize this query, and then it's going to go in and return to me the three closest objects and I can go ahead and plot those all out. The most semantically similar concept Here is this video of a dog running with a stick. And then I've also got audio files here. This is the audio of a dog barking. I'm not sure if you'll be able to hear it, but yeah, you can play around with this once you have the notebook. And then this is the image of the dog. And in order to go from text search to image search, it's as easy as passing in the image that you want to use as a question. So here, if I've got the. If this is my query image, this image of a cat, I can pass this and I can convert it to a base 64 image. I can do a with near image search and then I can ask it to return to me the name of the closest images, where they're stored and what type of files they are. And I can run this query and this vectorizes the image, generates a vector for me, and then I can see what objects are close to this image. So here I got the image of this cat. It's pretty close. These are all images of cats essentially, and it matches pretty well just based on image similarity. And then you can even do audio search. So here, this is the audio of a dog barking. You can play around with this later on. I'm going to pass this in as a query. So when I do with near audio, I'm going to convert it to base 64 and then everything else remains similar. I'm going to limit this to one, so I only get one output and then we can have a look at what that one output is. And so this is another audio file of a dog barking. So this is one second, that was two seconds. We can also do video search. Right? So if I've got this video of a meerkat looking around, I can use this as the input query like so, Again, same step, very simple, convert it to base 64 and then return the closest objects that I've got here. This one takes slightly longer because again, videos take longer to vectorize and embed. The machine learning model running on my computer needs to crunch a lot more numbers to get the vectors. So one of the last things that I wanted to show you, and then I've got a section on multimodal Rack. Okay, so that's completed. So let me show you what came back. So here I've got this video of a meerkat. So this is all truncated, but you'll be able to play around with this later. Again, very semantically similar, I've got this image of the meerkat and then I've also got this image, image of a meerkat standing on a log. And so this is pretty cute. So what I decided to do was introduce this concept of multimodal rag. And multimodal rag is a very new concept. There was a paper about it earlier this year, but Sebastian talked about rag. I want to talk a little bit about multimodal rag. So let me take two minutes and talk about what that is. So how does all of this multimedia searching actually help with large language models? Everybody is interested in large language models. How can it improve these large language models? Well, let me start off with this. Sebastian talked about retrieving text and then passing it off as context. But isn't it true that a picture can encode a lot more meaning than text? So then the question is, why stop at just retrieving text? If now I've shown you that you can retrieve audio files, video files and images, we can go ahead and retrieve all of these different types of modalities. Now, your rag workflow looks a little different where you can go ahead and store all of these images, text, audio files in weaviate and you can also retrieve them. Let's say I retrieve an image like that image of the meerkat on the stump. Now I can pass that off to a large multimodal model, a large language model that can understand images, and I can get it to answer questions on top of those images. And this is the very basic concept of multimodal rag. If I'm retrieving different media from my vector database and then I'm sandwiching it into the prompt and getting the large language model to answer questions on those images, I've performed multimodal Rag. That's the very last demo that I'm going to show you. This is the most, probably the most exciting part. So what we'll do here is we'll retrieve this image of this meerkat standing on a log from weaviate, and I'm going to pass it off to the Large Multimodal Model Chat GPT4 Vision from OpenAI. And then I'm going to get it to output text. And then I'm also going to use Dall E3 to get it to repaint, recreate this image, but just more cuter, right? So maybe I want to get this framed and put it up on the wall. So I want to show you how to do that. So I'm going to perform a near text query and I want to get out this particular image, so I'll describe it. Right. Meerkat on a log and I'll only get back one. And I'm also going to pass in a filter here that only returns images because for now, ChatGPT4Vision only understands images. It doesn't understand audio files. So I only want to filter for images. So I'm going to run this query and then I get back this meerkat image that I wanted to pass passed to GPT4 Vision. This code I got from OpenAI. I have a new API key that allows me to access GPT4 Vision, the preview, and I give it a prompt that says, this is an image of my pet. Please give me a cute and vivid description. And I pass in the base 64 image as well to this call to OpenAI and then I'll show you the description that it returns for me, the answer on this prompt. So we're going to run this, it's going to make an API call to OpenAI and we'll see in a second what it comes back with. It gives me this really nice description of the meerkat, charming site, your pet meerkat. And then it describes the meerkat. So this is one application of Multimodal Rack, right? I've given it an image and I've got it to describe the image for me. I'm going to go one step further. I'm now going to take this description and I'm going to pass it off to a model that can understand text and generate images. So that's Dall E3, which was released last week. So now I'm going to call Dall E and I'm going to tell it to create a image, a higher resolution image based off of this description that I have here. So this is potentially something that I can frame and put up on my wall. Then I'll show you what that image looks like right here. Again, this takes a little bit longer because it's generating the image after I run the code. All right, so this is purely off of the text description. I've got the meerkat on the log and I can print this out. I can get this turned into a poster and can go up on my wall. And this is the power of vector databases used synergistically with generative models that can understand all the modalities that you can retrieve from vector databases and you can do this at scale. So here I've shown you a quick kind of toy demo where I've got 21 files, but these could be billions of files, and you can retrieve them and pass them off as context. And so this is what I was super excited to show you all. If you have any questions, we'll take those now. If not, check us out. We think a lot about this. We work a lot on this. It's completely open source. Join our Slack community. Thank you very much, folks.

Absolutely. Well, thank you so much, Sebastian, Zane. I'm sure our community learns so much from you guys and really appreciate the thorough workshop you put together. I think we have time to answer maybe two or three questions before we end today. I know Judah in the audience and Sebastian, I saw you hop in and answer some questions in the live chat. But I think our number one question right now is if you were to compare the cost of keyword search versus vector search, how much is vector search more expensive on average?

Interesting. Is that like in a monetary way or. Yeah, that's. That's an interesting question. I don't have a clear, like a perfect answer. Zen, do you have a good take?

Yeah, I guess the. So that's a pretty. It is a difficult question, but I think what you have to account for is that when you're doing vector search you have this added component of having a machine learning model that you need to perform inference with. And so as you saw in the multimodal space, that inference can be very costly. It might require a lot more GPUs. It requires its own infrastructure. So from that component it might be more costly for keyword search. For classical search, you don't need GPUs, you don't need machine learning models.

Although you wouldn't necessarily use keyword search on an image. Right. Because that's the.

That's the.

Unless you do labeling or something.

Yeah, exactly. So what are you. If you're setting up this computer that has GPU that can understand images, video, the added benefit that you pay for is that now you have this ability to semantically search over any modality really, that you can encode as a vector. It would be very hard for you to do keyword search over images or video or audio, unless you have some detailed description or metadata of it, which then you can use to filter. But that's a very interesting question.

Absolutely. I think the last question for today, how do you keep data current? Are there optimal methods for updating the Vector DB database once a day nightly different than updating more frequently?

Yeah, I would say so. It depends on your needs within your business. So if you like, basically. So that's one of the benefits of weavit is it's got full CRUD support That is live. So basically, as you go, even if your data changes every second, you could continuously keep updating the data and then the vector space or the vector index gets updated immediately. So the question here is it depends on what does your business need? So if you always need the data immediately to be all the time, I don't know, you're running a social media platform. In this case, you need to be constantly. If you're running a local library, maybe you could do it once a week. So it all depends on what do you need in terms of that frequency. And yeah, there's no one perfect answer. But the beauty of webit is that you could very easily keep it in track like live. If you want to create a system that continuously sends updates, that's not a problem at all.

Absolutely. Well, I want to once again thank Sebastian and Zane and Duda for speaking today, answering questions, helping us learn more about weaviate and vector databases. Once again, please don't forget to register for their short course. I'm sure you'll not be disappointed and learn so much more than what we also learned in this workshop today. We also dropped a community survey in the chat. So happy to take topic suggestions for our next event, maybe even another course of web 8. We're always open to feedback and suggestions and we hope to see you next time here with us at Deep Learning AI. See you guys soon. Thank you.

Bye.

Thanks a lot.

Thanks for having me.