Microsoft's New 'PROJECT RUMI' Takes Everyone By SURPRISE! (Now Announced!)

Microsoft has recently unveiled an innovative breakthrough in the realm of artificial intelligence, setting the stage for a transformation in how we engage with large language models like ChatGPT. This pioneering endeavor is termed Project Rumi. This is not just another incremental step in AI advancements. Rather, it's a leap forward, differentiating itself from prior efforts by leading AI research teams. Until now, interacting with AI language models has largely been a text based experience. We input questions or prompts and the AI responds based on its training. But what if an AI could go beyond just understanding the words you type? What if it could sense the underlying emotions behind those words? Enter Project Rumi. The core idea behind this initiative is multimodal paralinguistic project prompting. At a glance this might sound like a complex tech jargon, but it can be distilled into a simpler concept, enabling large language models to not only process textual information, but to also gauge and respond to the user's emotions. This means that when you communicate with such a model, it would not only understand the words, but also the sentiment and feelings behind them. So if you go over to Microsoft's research page, they do a lot of explaining to show how it works. They they also include a video in which they demonstrate this new software. They state large language models are great, but LLMs also have limitations. They may not always understand the context and nuances of a conversation. Their performance also depends on the quality and specificity of the user's input or prompt. The data that the user inputs into the LLM is a lexical entry which does not comprehensively represent the nuances of human to human interaction. It is in fact missing all the paralinguistic information, intonation, gestures, facial expressions, and everything besides the actual words that contribute to the meaning and intentions of the speaker. This can lead to misinterpretation, misunderstanding or inappropriate responses from the LLM. Project RUMI incorporates paralinguistic input into prompt based interactions with LLMs with the objective of improving the quality of communication. Providing this context is critical to enhancing LLM's capabilities in this AI as a copilot era. Our current system leverages separately trained vision and audio based models to detect and analyze non verbal cues extracted from data streams. The models assess sentiment from cognitive and physiological data in real time, generating appropriate paralinguistic tokens to augment standard lexical prompt input to existing LLMs such as GPT4. This multimodal muti step architecture integrates seamlessly with all pre trained text based LLMs to provide additional information on the user's sentiment and intention that is not captured by text based models. Augmenting the prompt with the richness and subtlety of human communication to bring human AI interaction to a new level. The illustrated diagram that accompanies the description offers a comprehensive view of how a user interacts with the advanced system. This visual representation breaks down the various inputs into two primary physical sensors and non contact systems. Physical Sensors these are tactile devices that come in direct contact with the user to gather real time physiological data. The diagram showcases three primary EEG this sensor measures and records the electrical activity of the brain. It provides valuable insights into a user's cognitive processes, alertness and mental state, allowing the system to gauge the user's concentration, relaxation or potential cognitive workload. Perspiration Sensors Also known as galvanic skin response sensors, these measure the electrical conductance of the skin which varies with its moisture level. As sweat production can be an indicator of emotional or physiological arousal, these sensors offer cues about the user's emotional state, be it stress, excitement or fear. Heart Rate Monitor as the name suggests, this device measures the user's heart rate. Fluctuations in heart rate can indicate a range of emotions or reactions from relaxation and calmness to anxiety or excitement. These are systems that gather data without physically touching the user, relying on visual and auditory cues instead. The diagram details three Primary Camera this system captures visual information, analyzing facial expressions and micro expressions. These expressions can reveal a plethora of emotions from happiness and surprise to sadness and anger, enabling the system to deduce the user's emotional state accurately. Eye Tracking this advanced system follows the user's eye movements and gaze direction. By doing so, it can infer elements like focus, interest and even the emotional response to what the user is viewing. For instance, rapid eye movement might suggest nervousness or excitement, while a fixed gaze might indicate concentration or deep thought. Speech Analysis by analyzing the user's voice, this system can detect variations in tone, pitch and speed which can be indicative of emotions like happiness, frustration, uncertainty or confidence. Together, this combination of physical sensors and non contact systems and paints a holistic picture of the user's emotional and cognitive state, allowing the system to tailor its responses and interactions accordingly. This fusion of technology offers a promising leap towards a more intuitive and empathetic human AI interaction. So Microsoft also includes a video in which they demonstrate how effective this software is. They have integrated it into what seems to be a beta version of Bing Chat. So from this first screenshot you can see this is where the user records a video or audio that is then transcribed and input into the AI as text. Then the data in terms of what emotion the user is feeling is displayed as a pie chart. You can see that some of this user's emotion were happy. There is also apparently some disgust in the audio which the AI has picked up on and in the video the AI describes the state of the user as neutral. Further on in the video they describe how they break down the audio and realize what is being said. The audio is broken down into two parts, first the text and then the features. They use two transformers to do this, the first being Hubert. The Hubert Transformer is a self supervised speech representation learning model inspired by the BERT architecture which is originally designed for natural language processing tasks. In simple terms, Hubert is used to transform raw speech data into a more language like structure by discovering discrete hidden units which can be compared to words or tokens in a text sentence. The main goal of Hubert is to improve performance on various downstream tasks such as speech recognition, generation and compression. It achieves this by consuming masked continuous speech features and predicting predetermined cluster assignments. The model architecture of Hubert is similar to WAV2VC 2.0 with base and large versions having 95 million and 317 million parameters respectively. They also use the Transformer Distalbert Distalbert is a smaller, faster and lighter version of the BERT transformer model designed for natural language processing tasks. It is created using a technique called knowledge distillation which reduces the size of the original bert model by 40% while retaining 97% of its performance. This makes Distilbert more efficient and suitable for on device applications and situations with limited computational resources. Distilbert can be used for various natural language processing tasks such as text classification. Overall, project RUMI is quite surprising. We are so used to just chatting with large language models, but now they may be able to gauge how we feel and think, which means there will be a much deeper level of understanding which overall should improve the responses.