In this Seed to Harvest podcast episode, host Paige Finn Doherty (founding partner at Genius Ventures) presents an extended, co‑authored overview of multimodal AI. She defines multimodal AI as systems that ingest and generate multiple data types—text, images, audio, video—drawing a parallel to human multisensory perception and emphasizing the core idea of convergence. The episode traces the field’s academic origins (early neural networks, CNNs, RNNs, transformers) and the private‑sector acceleration by firms such as Google, IBM, Microsoft, OpenAI, Anthropic, and Meta. It contrasts text‑only large language models (e.g., ChatGPT, Bard, Claude) with multimodal models, highlighting the expanded input‑output possibilities. Technical depth is provided: preprocessing pipelines for each modality, model families (CNNs, transformers), massive compute requirements, and the need for diverse, accurately labeled datasets. The discussion flags major obstacles—data privacy, bias, integration complexity, and the labor‑intensive labeling process. Looking ahead to 2024, Doherty forecasts a breakout year driven by incumbent breakthroughs (GPT‑Vision, Gemini, Anthropic’s new models) spawning a vibrant startup ecosystem across verticals like healthcare, edtech, and entertainment. She spotlights emerging trends: wearables and spatial computing as multimodal sensor platforms, video‑generation and deep‑fake risks, and the rise of specialized vertical hardware to close modality gaps. Throughout, she maintains a cautiously optimistic tone, urging experimentation while acknowledging the early‑stage nature of the technology. The episode concludes with a call for community discussion and thanks to editor Tate Doherty.
Chapters
We anticipate transformative breakthroughs in 2024 with multimodal AI making significant strides in various domains. So due to its complexity of interpreting and outputting different modalities, multimodal AI is much more technically involved than unimodal AI. It's unlikely a perfect solution will be the first one announced as we're still super early. So I think experimentation is really important. Hello everyone. Welcome back to Seed to Harvest. I'm your host, Paige Finn Doherty, founding partner behind Genius Ventures. I'm really excited today to share a piece that I wrote with our investment associate, Andy Strand. So today's episode is on an introduction to multimodal AI, explaining both its roots in academia and the private sector, what it is, what's happening, what the difference is between text-based AI models and multimodal. The core concept of multimodal AI, which we believe to be convergence, discussing the technical complexities and challenges, including the need for diverse datasets, computational power, and addressing data privacy concerns. And despite these obstacles, we anticipate transformative breakthroughs in 2024 with multimodal AI making significant strides in various domains. Anyways, welcome thoughts, ideas, discussions around the topic. Remain as ever cautiously optimistic about the future, and I hope you enjoy this episode. Thank you for tuning in. So first, let's define what is multimodal AI. So my definition, multimodal AI combines, interprets, and outputs multiple data types. So if you think about the initial, like, text-based large language models, those are simply text-based. Multimodal AI, you can think about text, images, audio, video, all of these become part of the input and output of multimodal AI. And this elevated integration of different modalities allows multimodal AI to respond to vastly complex and multifaceted scenarios and create systems that are more intuitive, interactive, and capable of handling real-world complexities with unprecedented accuracy, much as how a human would. And this was a big part of what I studied at San Diego State. Actually, one of my favorite classes was a speech processing class in which we used applied machine learning to both input and output audio files, which was just fascinating. It's a relatively new space of research, and it has its origins in both academia and the private sector, multimodal AI specifically. And its journey can be traced back to the exploration of fundamental neural networks and machine learning applications, which primarily focused on single-mode data processing. And this groundwork is amazing. It established the foundational building blocks used in later multimodal research, such as the development for convolutional neural networks for image processing and recurrent neural networks for sequence data like text. And audio. Um, and then the private sector played a crucial role in accelerating the development and application of multimodal AI. Companies like Google, IBM, and Microsoft invested heavily in research and development, leading to breakthroughs like Google's multimodal neural machine translation system, which integrated text and images for translation. And this system was a significant advancement as it demonstrated the practicality and effectiveness of combining different data types for complex AI tests. And later, multimodal AI benefited from the advent of the transformer model. It's kind of weird, but transformer model is like one of my favorite models. But anyways, the difference between text-based AI and multimodal AI. So on November 30th, 2022, most mainstream consumers were first introduced to OpenAI's text-based AI, ChatGPT, which at the time primarily specialized in processing and generating text. So ChatGPT, along with competitors such as Google's Bard and Anthropic's Claude, are language-based models, meaning they primarily understand and produce text-based content, which means the input and the output are both text-based. Um, so what are these models good at? These text-based models are highly effective for tasks like conversation, text content generation, rudimentary data interpretation, answering questions, and text completion. And these models learn from immensely vast libraries of text data, allowing them to generate responses that are contextually relevant. On the other hand, multimodal AI goes beyond just the singular input of text. It integrates and interprets multiple types of data inputs, including text, images, audio, video simultaneously. Additionally, the output can be multimodal in nature. For instance, text-based input can result in video-based output. This unique capability elevates the experience dramatically, allowing multimodal AI to have a more comprehensive understanding of complex scenarios, which again aims to mimic how, how humans perceive and interact with the world using multiple senses. So. Multimodal AI, to me, primarily is about one concept, convergence. And it sounds very like sci-fi-y, but basically it blends different forms of data inputs to generate a more holistic picture of the world as output. And what's particularly unique about this approach is that it comes closer to mimicking how we as humans process information, integrating sights, sounds, and other senses to form a complete picture of our surroundings, be able to take action on those surroundings. And our own senses ultimately work in harmony, not in isolation. And this multisensory integration allows us to perceive the world in a rich, multidimensional way. Our brains don't just process these sensory inputs, they weave them into a cohesion of experience, memory, and emotion. So multimodal AI is really interesting to me because it seeks to emulate this exact human capability. It converges a series of diverse data streams, visual images, auditory sounds, textual information to understand and interact with the world and produce an output. So just as humans use their senses to navigate their environment, multimodal AI uses its multiple modalities to process information more historically and accurately. Which brings us to the challenge in multimodal AI, which is inherent technical complexity. So due to its complexity of interpreting and outputting different modalities, multimodal AI is much more technically involved than unimodal AI. And as a result, proper and accurate execution is really, really difficult. From a technical standpoint, it's important to understand that multimodal AI provides output through a series of unique data-specific procedures as opposed to just one single process. And so the process begins with the initial acquisition input of different data forms, be it text, images, audio, or video. Each data subtype is preprocessed to fit a respective analyzable format, with images undergoing normalization and resizing, text being tokenized and vectorized, and audio converted into analytical representations like spectrograms. You can think of those kind of like fingerprints of audio. And each data stream is then processed through various models tailored to that specific modality. And then multimodal AI systems generally leverage neural networks, particularly deep convolutional neural networks or transformers for processing sequential data like text and speech. These networks are notorious for requiring immense computational power, often necessitating the use of advanced GPUs or TPUs for parallel processing and quick data handling. And so the challenge in Multimodal AI lies not only in managing diverse data types, but also in synthesizing them to produce an accurate and meaningful output. This requires not only strong algorithms and an intense level of computational resources, but also intricately designed models that can capture the nuances of each data type while providing a coherent and unique output. It's an art as much as it is a science. Partially due to its technical complexity, multimodal AI still faces roadblocks in obtaining widespread mainstream adoption. Notwithstanding immense levels of computing power, multimodal AI systems also require vast and diverse datasets to be effectively trained, and these datasets must be accurately labeled, a process that can be both time-consuming and labor-intensive. And with the recent press around the biases present in Gemini's model. This is a great example for how important testing is in these models and how much of an impact that opinions can have in these models being trained. So another major obstacle is the complexity involved in integrating and synchronizing different data modalities. So each type of data, be it visual, textual, or auditory, has its unique characteristics requiring specialized processing techniques. So developing models that can effectively handle this while maintaining accuracy and efficiency is a significant technical challenge because it takes time. Another challenge is data privacy. So challenges such as data privacy, ethical considerations, and the need for robust and unbiased datasets will also need to be addressed. As multimodal AI systems become more integrated into our everyday life, the way that they handle and process our personal data becomes critical concern. So unsurprisingly, data privacy issues arise from the extensive collection and analysis of personal information, which, if not properly managed, could lead to breaches or even misuse of said sensitive data. Ensuring the privacy and security of user data requires stringent protocols and possibly even new regulatory frameworks. So the good news, obstacles aside, multimodal AI is poised to be transformative with the potential to significantly reshape numerous aspects of technology. Technology and life, with 2024 shaping up to be the breakout year for multimodal AI. As computational capabilities continue to advance rapidly, we can expect multimodal AI systems to become more sophisticated, enabling more seamless and intuitive interactions between humans and machine. Applications can be widespread— widespread entertainment, healthcare, edtech, and more can all benefit from multimodal AI. Next, I'm going to explore some predictions for multimodal AI in 2024, where this technology is expected to make significant strides in various domains, reflecting an acceleration of AI technology that we expect will be substantially elevated from where we stand today. So these following trends build on our earlier piece, Investing in Early Stage AI. You can find it by Googling Investing in Early Stage AI. It's one of the first results that pops up on Google. Or I'll link it in the description. So the first trend that we're watching is incumbent breakthroughs, which will result in a new startup ecosystem. So whether it's OpenAI's GPT Vision, Google's Gemini model, or Anthropic's new models, incumbents are investing tremendously into multimodal AI. With companies able to build atop these multimodal AI models, the possibilities for disruption are endless as startups can leverage these incumbents' research breakthroughs and can utilize their APIs. Expect a wave of companies innovating in verticals far and wide, from healthcare to edtech to education and much, much more. The tangible convergence. So Meta launched their collaboration with Ray-Ban, Humane released their AI pin, and OpenAI has been said to be pursuing an entry into the wearable space. And my belief is that wearables will increasingly become integrated with advanced multimodal AI, transforming how we interact with our digital and physical environments. One of the things that's really interesting is I went to a talk done by one of the heads of Microsoft's research. And one of the things that I learned was that actually, if you think about like our sensory inputs, wearables are really interesting because they can serve as a multidimensional sensory input machine for a multimodal AI model. Which then can also use that wearable to have a reaction as well. And the third trend that we're looking at is video generation multimodal AI. And we've made an investment in this space in a company called ProfGym that's been tremendously impactful in both education and workforce training. One of the things that I'm really interested in is how multimodal AI-generated deepfakes play out in Um, the political landscape with the upcoming election. I think deepfakes have constantly been a concern in the evolving landscape of AI, even from the earliest days. Um, and so while traditional deepfakes mostly involve manipulating images or videos, multimodal AI deepfakes take this a step further by integrating various data types, making them more sophisticated and extremely difficult to detect. But we're definitely closely watching for AI-driven applications of media literacy and content moderation. And 4 is spatial sensors, which is the inevitable intersection of AR, VR, and multimodal AI. So behind Genius Ventures, we've been investing in consumer hardware-enabled software as part of the practice, like Averon, The Last Gameboard, Heart Display, and Lotus Labs since 2021. And I would say most VCs shy away from hardware, but we believe deeply in its ability to transform the world as we know it and produce meaningful returns. So what happens when hardware comes in the loop? Our prediction is that specialized vertical hardware companies will fill the gaps. It's really challenging to build high-level software for things like this without verticalized hardware. And so I think that companies that are really good at one modality or the other, touch, audio, whatever, I'm really interested in innovative solutions in the space. So curious to see if there will be like a next generation homebrew computer. club where folks jam on innovative hardware ideas. So I think it's unlikely a perfect solution will be the first one announced as we're still super early. So I think experimentation is really important. One of the things I thought that was really interesting was the consideration of spatial computing versus multimodal AI. So if you think back to a lot of the trends like generative AI or Web3, there were existing things that were happening in tandem, and then that language arose and kind of took us to a whole nother level of hype cycle. And so I'm really interested to see if, like, for example, Apple's submission guidelines for the Vision Pro App Store specifically instruct developers to not describe applications as AR, VR, XR, or MR. Instead, they're requiring the term spatial computing. So I anticipate this term of spatial computing to be used far and wide in 2024, often directly alongside or as a substitute for multimodal AI. In conclusion, hopefully this was a helpful introduction to multimodal AI. I'm really passionate about this space, and hopefully this was a helpful explanation of its roots in both academia and the private sector. And Yeah, I welcome thoughts, ideas, discussions around this topic. If you have any interesting articles, I would love to read them. Feel free to DM them to me on Twitter. As always, I remain cautiously optimistic about the future, and I hope you enjoyed this. And a very special thank you to Seed to Harvest podcast editor Tate Doherty for his amazing work on this episode.
