Meta New Models - SAM 2.1, Spirit LM, MEXMA and More

Okay, so Meta just released a number of different models focused on a variety of different applications. All this happened on a Friday afternoon in a blog post sharing new research models and data set from Meta. Fair is meta's fundamental AI research lab and their focus is on achieving advanced Machine intelligence or ami, unlike some of the other foundation labs which are trying to achieve AGI. But I think the difference is more in semantics. Now one of the main highlights of these newly released models is the new model which they're calling metaspiritlm, that is their speech to speech system. More on this later in the video. In this video I want to look at these new models and will share some of my initial thoughts. So we're going to start with SAM 2.1, which stands for Segment Anything Model 2.1. It's not a new model, it's an updated version of their segmentation model that was released a few months ago. SAM or the first version was specifically focused on segmentation of images into different objects. SAM2 was able to do object segmentation in videos and track these segmented objects across multiple different frames. Now SAM 2.1 is not a new model, it's just an updated checkpoint or an improved version of SAM 2. It's great to see that Meta is not only releasing these advanced segmentation models, but also open sourcing them. The second model is a MetaSpiritLM, an open source language model for seamless speech and text integration. It's their new system that can accept both speech and text as input and can generate both text and speech as output. Now why do you need a system that can accept speech as input? Because there are systems which will use something like Whisper to convert speech into text, but it will lose all the expressiveness of the speech. Similarly, the output speech is usually very robotic and and loses expressiveness, but very similar to GPT4O or Mushi. This can process speech directly as an input and then generate speech as an output. The way this does it is there's a encoder in front of the LLM.

A cute clever cat. Once upon a time there was a beautiful cat named Peter. His mother taught him how to climb a tree. I feel very excited about the upcoming trip to Paris. I've never been before. Our friends are about to stay for two weeks and I want to see as much as possible.

The quality of a generated speech is not that great, but I think It's a relatively small 7 billion model. Hopefully with the larger models we will be able to get much better quality of the output speech. Look at some of the technical details of this new system. But I think the most important part is going to be the license, which in this case is non commercial research license. There are two different models. One is called the Spirit LM base. The other one is the expressive version. Now since it's processing speech tokens, so there are three additional speech tokenizers. One is for processing speech inputs, the other one is for processing pitch and understanding the style. That is the third tokenizer. All of this is encoded and then fed into the LLM. For the LLM part they're using the Llama 2.7Billion model as the base model. I wish they they would have used the newer versions, but depending on the input or the model that you have selected, it's going to tokenize this, the text plus the speech and then they're going to be additional special tokens for the style and pitch of your input speech as well as the generated output speech. Now currently I'm downloading the weights of the model. Once the download is finished, I will experiment with it and probably will create a subsequent video. But in order to use this model you do have to accept the terms and conditions and you need to apply for access to the weights. The rest of the models are more technical in nature, but they are significant because they show an improvement in both the inference and the training speed and training cost as well as some very interesting ideas on how you can better train these models. So the first one we are going to look at is layer skip enhancing large language model performance with accelerated generation times. So this is a new research which shows how to improve the inference speed of the models. And the way they recommend is that you can use a subset of an LLM layers for the generation of responses and then use the subsequent layers for verification and corrections. Now they have fine tuned checkpoints with this layer skip for Llama three, Llama two and code Llama. And they show that with the layer skip architecture you can get a performance boost up to 1.7 times the code weights and a technical report is available. The second is a SALSA validating security for post quantum cryptography standards. So this is basically a research project that is focused on how can you improve the security specifically against AI based attacks. Now personally I'm more interested in LLMs. So they have another project, Metalingua, accelerating research with efficient model training. It's a lightweight and self contained code base designed to train language models at scale. This is essentially their own small framework for improving the performance when it comes to training of large language models. And it's based on different Pytorch components and seems to show pretty great performance boost with lower computational resources. The next project that they are open sourcing is Meta Open Materials 2024 facilitating inorganic materials discovery with open source data and models. So in this case they are releasing both the models and a data set for doing material science related experiments. One of the I think most famous model is for protein folding, but it's a closed source model. In this case, Meta is open sourcing the data sets plus the model weights to enable open research on inorganic material. Now the next two are again related to LLMs. So the first one is Maxima Token Level objectives for improved sentence representation. So during the training of these LLMs or training of embedding models, the word representation plays a very critical role, especially if you're doing translation from one language to another language. So in this research they're presenting Maxima, which is their novel pre trained cross lingual sentence encoder. And the idea is that they are using token level objectives which will give the models better representation, especially if it's in the cross lingual or multilingual setting. So specifically for translations, not only just looking at the structure of the sentences, but by looking at the token level representation you might be able to get better translations. This new encoding technique covers 80 different languages, so it's going to be pretty useful for multilingual applications. And the last one is Self Taught Evaluator, Strong generative reward models with synthetic data. Synthetic data has been a critical component of some of these new foundation models. And the concept of LLM as a judge has also played a pretty critical role, especially for reward models. So they are presenting Self Thought Evaluator, a new method for generating synthetic preference data to train reward models without relying on human annotations. And they're using this model plus the synthetic data train. LLM is just to produce reasoning traces for evaluation and final judgment. They are releasing the model which is trained with direct preference optimization. You can access the model but again just like the other models, if you are trying to access this, you have to fill out this form on hugging face. This can be used for the purposes of LLM as a judge and according to them it outperforms bigger models or using human annotated labels such as GPT4, llama 3.1405 b, instruct and Gemini Pro. So it seems like it's a pretty smart model compared to some of the state of the art models specifically for evaluation purposes. And since it's a smaller model it you can get up to seven to 10 times faster than the default GPT4 evaluator on the Alpaca Evolve leaderboard. Now this was a quick overview of the models that are released today. Interestingly, Meta is leading the way at open sourcing these models. Now, technically, some of the models are not open source. More it's a open weights. But even if you can't use it for commercial purposes, having the availability of the weights and the ability to run these models locally is pretty significant. Especially that not everybody is going to be using them for commercial purposes. I'll be definitely making a video on the Spirit LM model when I have this up and running, so if you're interested in that, make sure to subscribe to the channel. This was a quick video. I hope you found it useful. Thanks for watching and as always, see you in the next one.