AGENTS are the Real Future of AI + NExT-GPT: AI's Multimodal Masterpiece!

There's a new open source framework for creating and deploying language agents that is going to blow your mind. It's called Agents. Go figure. And it stands for Autonomous General Purpose Environment Aware Natural Language Taskable Systems. But honestly, this is one of the most exciting things I've ever seen in AI. So what are language agents and why do we need them? Well, language agents are systems that can understand and communicate in natural language, such as English, Chinese or Spanish. They can perform various tasks and interact with different environments, humans and other agents using natural language interfaces. You've probably seen some examples of language agents in action, such as chatbots, virtual assistants or conversational AI. These systems are becoming more and more popular and useful in various domains such as customer service, consulting, programming, education, entertainment and many more. However, making and using language agents is tough. They need expertise, tools and resources. Many free resources are limited, hard to use, or too specific. For experienced developers, combining different parts to make new agents is tricky. For others, understanding and using these systems is challenging. It's also tough to ensure agents act right, control their actions and improve them. This is where Agents becomes useful. Agents is a revolutionary framework that aims to make language agents more accessible, flexible, robust and controllable. It was developed by a team of researchers from AI Waves Inc. Zhejiang University and ETH Zurich. They published a paper about it on ArcSIF and they also released the code on GitHub. It is based on three key memory, web navigation and tool usage, and multi agent interaction. Now I'll break down these ideas for you. So first let's talk about memory. For language agents, memory is important because they need to save and use information for their tasks. Take booking a flight as an example. The agent needs to remember things like what you like, how much you want to spend and when you're traveling. Agents has both short term and long term memory. The short term memory holds info for what's happening right now, while the long term memory keeps info for many tasks over a longer period. This way an agent can learn from past experiences to do better in the future. Next, there's web navigation and using tools. This helps language agents get more information and skills from the Internet. Say you want an agent to write you a poem. It might look online for ideas or examples, or if you have a math question, it might use WolframAlpha to find the answer. Agents can use popular tools like Google Search, wikipedia, Wolfram Alpha, OpenAI, Codex and more. Developers can also add other tools if they need to. When agents look something up online, it doesn't just bring back links, it can pull out facts, pictures, news stories, and more. Lastly, there's working with other agents. This is key because sometimes agents need to work together or even compete. For instance, if you want an agent to play a game with or against you, it should be able to team up or go head to head with other agents. Or if there's a big job that needs different skills or steps, an agent might hand off parts of it to others. Agents allows for this with a system that lets agents decide when and how to work with each other. There's also a main agent in charge of making sure everyone follows the rules and does what the user wants. Speaking of goals, this brings me to the final and most amazing feature of agents controllability and symbolic plans. Controllability means that the user can have fine grained control over what the agent does and how it does it. Symbolic plans are the way to achieve that. Symbolic plans are high level instructions that specify the desired outcome and the steps to achieve it using natural language and logic. The agent will then use this plan to guide its actions and interactions and produce a blog post that satisfies your requirements. You can also modify or refine the plan as you go along and give feedback to the agent to improve its performance. This is a revolutionary paradigm shift in language agents because it allows the user to have more control and transparency over the agent's behavior and also allows the agent to be more flexible and adaptive to the user's needs. Agents is the first framework that supports this kind of controllability and symbolic planning for general purpose language agents. Alright, now let's shift our focus to another innovative leap in AI. So researchers from the National University of Singapore created a new AI technology that could revolutionize the way we interact with computers and the Internet. It's called NextGPT and it's an end to end general purpose any to any multimodal large language model. I know it might sound complex, but don't worry because I'm going to explain what it is, how it works and why it's so amazing. So a multimodal large language model is basically an AI that can understand and generate content in multiple modalities. Like when we chat with friends, we use more than just words. We smile, change our voice, use hand motions, show pictures, and sometimes even play music. All these help us share more feelings and make our talks more real. Now think about talking to computers like that. Well, that's exactly what some researchers and companies have been trying to achieve in recent years. They have developed various MM LLMs that can perform tasks like image captioning Video summarization, speech recognition, text to speech synthesis and so on. However, there is one big limitation that most of these MM LLMs have. They can understand different types of inputs like text, photos or voice, but usually respond only with text. But NEXT GPT is here to fix that problem. NextGPT is a new technology developed by researchers from Next, the School of Computing at the National University of Singapore. It stands for Next Generation Extreme Task Oriented GPT Generative Pre Trained Transformer. It is an end to end general purpose any to any MMLM system that can perceive inputs and generate outputs in arbitrary combinations of text, images, videos and audio. It's based on three main components, multimodal adapters, LLMs and diffusion decoders. Multimodal adapters are modules that change different types of inputs into language like forms that LLMs can understand. They also turn LLM outputs into other formats using diffusion models. LLMs are the core agents of NextGPT. They use these language like forms from different sources to understand and think about the input. They then give back text, answers or special signals. These signals tell the decoding parts if they should create multimodal content or not. By connecting LLMs with multimodal adapters and diffusion decoders, NEXT GPT achieves universal multimodal understanding and any to any modality input and output. This means that it can take inputs in any combination of text, images, videos and audio and produce outputs in any combination as well. What's more, NextGPT is tuned with only a small amount of parameter 1% of certain projection layers. This means that it can leverage the existing well trained, highly performing encoders and decoders without having to retrain them from scratch. This not only benefits low cost training, but also facilitates convenient expansion to more potential modalities in the future. Now NextGPT also has another unique feature and its ability to switch between different modes of communication intelligently thanks to something called modality switching, instruction tuning or MOSAIT for short. It means the AI can take a task, break it down into several steps involving different forms of communication and carry it out efficiently in everyday life. There are so many ways this concept can be used. It could reinvent chatbots and virtual assistants, making them more intuitive and interactive. It can be a game changer in education and entertainment, creating immersive and engaging content based on individual inputs. Even in the realm of research and innovation, it could assist in exploring new ideas and solutions more dynamically. In simpler words, NextGPT is set to revolutionize our digital experiences, making interactions with technology more natural, fluid and intuitive, enhancing everything from virtual assistants to educational tools. So what do you think? Let me know your thoughts in comment section below. And if you liked this video, please give it a thumbs up and subscribe to my channel for more awesome AI content. Thank you so much for watching and I'll see you in the next one.