New course with Weaviate: Building Multimodal Search and RAG

I'm delighted to introduce Building Multimodal Search and rag, built in partnership with weaviate and taught by Sebastian Vittanetz. Most retrieval augmented generation systems are just about text, text, text. Multimodal retrieval augmented generation lets you retrieve context that can be text, slides, images, video or other modalities. In this course we'll also cover a couple of key techniques. First, a deep learning method called contrastive learning, which is key to computing multimodal embeddings. And we'll also learn how to instruction two models that can process both text and images, in this case using tokenized text and tokenized images as input.

Thanks Andrew. I'm really excited to be back again in here to record this course. So let's say a user asks question about a company's annual financial report. You'd like your system to retrieve not just relevant text documents but also graphs, tables, slides and so on, and use all of this context to generate a more useful answer.

And an important step to accomplishing this is making sure that the text, images, size and so on that are about similar topics have similar embeddings so that when they stored in a vector database, a query for a specific concept can retrieve or all of that relevant context. This means we need embedding models that can embed similar concepts into similar vectors regardless of the modality. So as a toy example, we'd like a picture of a cat to be embedded close to a passage of text about cats, which should also be close to a video of cats playing. This way, a query about cats can retrieve all of this data. In this course you build a full write application where you search for a related image based on your text query and after retrieving it you ask the questions based on the content of the image. There are lots of technical details behind making this work well, and I'm delighted that Sebastian, who is head of developer relations at weaviate, will be teaching all this. Sebastian is an expert in vector databases and has been working on developer education for a decade.

In this course you'll first learn how to teach a computer the concept of understanding multimodal data like text, image, audio and video. Then you build a text to any as well as any 2N search engine. So for example, you could input video and retrieve relevant images. You also learn about instruction tuning of large language models that only process text and using additional image tokens to turn them into large multimodal models that understand images and text. And next you focus on multimodal rack.