Apples New Mutlimodal AI BEATS GPT-4 Vision (New APPLE AI)

So we finally have some news from Apple regarding their machine learning LLMs in terms of what they've finally been developing. So Apple has introduced a multimodal AI system that is pretty impressive because it does actually exceed GPT4's capabilities in some regards. And this might be the scenario that many have been looking at when they say the GPT4 is, is no longer king. So let's take a look at exactly what Apple has introduced and how good this new multimodal AI system really is. So let's take a look at how this system works. It's called Ferret. So this is essentially the Ferret model and it's based by the Apple researchers. These are the ones that created it. And essentially it's mainly a vision model. So first it uses a tool called ClipVitl14 to understand what's in the picture and then turn it into a form the computer can work with. Secondly, it also looks at the words you give it and converts them into a format it can understand. Then it identifies areas in the image and if you talk about a specific part of the picture, like a cat in the bottom left hand corner, the model uses special coordinates to find exactly where, where that is in the image. Of course, we do have processing and shapes features and it's really smart in dealing with different shapes in the picture, not just simple boxes. It looks at many points in the area you're talking about and understands the details and locations of each point. Finally, it brings together this information together to accurately find and describe the specific part of the picture you're talking about. Essentially what we have here is a really impressive advanced image identification model that once when on certain benchmarks compared to GPT4, and I did test it myself to just make sure it actually does exceed GPT4's vision capabilities. So you can see here, first of all, there are some benchmarks that you may want to look at. So you can see on the benchmarks for the Ferret model, we can see that Ferret actually has all of the input types, which are point, box and Freeform. It also has very good output grounding, which essentially just means that it can understand exactly the relationship between certain objects in the image and what they actually do in the real physical world. Then of course we have on data construction and GPT generate and robustness and of course the quantitative evaluation of refer ground with chat. So this is actually very interesting because in this section of the paper they didn't actually Compare it to GPT4 with vision, they compared it to GPT4 ROI but later on in the paper I will show you that compared to GPT4 with vision. So if we take a look at GPT4 ROI, we can see here that it says GPT4 ROI instruction tuning, large language model on region of interest. And essentially what GPT4OI was was a specifically fine tuned version. So in the benchmarks of the PDF, I'm guessing that the researchers likely tested against GPT4ROI instead of GPT4 Vision, because GPT4RO is specifically designed for understanding and interacting with the regions of interest in images, which is a more advanced and specialized task than what GPT4 Vision might be designed for. GPT4 ROI's ability to combine language and detailed image analysis, especially focusing on specific areas within images, makes it a more suitable benchmark for testing the Ferret model's capabilities in fine grained multimodal understanding and interaction. This comparison helps to highlight the advancement and specific strengths of the Ferret model in handling complex vision tasks. And we can also see here the GPT4ROI benchmark, so that when you see what GPT4ROI is and why it was compared against the Ferret model, you can see exactly why it was very effective. And there are some of these things before, like we actually did a video on Visual ChatGPT. But GPT4ROI does have image, it does have a region, and it does have multi region. And even after all of that, the Ferret model actually does surpass it. So here's where we do need to take a look at one of the examples where we do Compare it to GPT4. So this is an example where they say, what is the purpose of the object on the bike? And region zero is the highlighted yellow region that you can see right here. And it says the ground truth. The object is a shock absorber on the bike. Its purpose is to absorb or dampen shock impulses. It does this by converting the kinetic energy of the shock into another form of energy, which is then dissipated in the context of the motorcycle. It is particularly useful in providing comfort and stability for the rider, especially when traveling over uneven or rough terrains. So you can see lava, the model, the Vision model didn't get it right. Cosmos 2, Microsoft's multimodal model actually didn't get it right. Shakira, this another model just completely didn't get it right. And then the Ferret model actually gets it 100% correct. It says the model is a shock absorber, yada yada yada. And, and this shows just how effective it is. Now, like I Said I did actually try to put this image into ChatGPT. And I said, what is the purpose of the highlighted region on the bike? And the highlighted region on the motorcycle is where the exhaust pipes and muffler are located. Which is completely wrong. I don't want to state that. I did actually try this multiple, multiple times. Like I tried this prompt so many different times and, and ChatGPT didn't get it right at all. Now maybe you could prompt this better than I did, but on this zero shot setting, it's simply just not that effective at providing insights onto certain things. Now one thing that they did actually talk about was the further comparisons between this and GPT4. So if we do take a look, you can see that they actually did some of this testing their self. So gpt4vision versus ferret. You can see here that we have Ferrit spirit. And essentially this is part of the section that actually talks about referring and grounding. So for example, you can see right here they say, what is region zero used for? And it says the object is a pipe used for transporting exhaust gases from a milk cycle. That's correct. Then the second region, region one, the object is a shock absorber. That is also correct. Now this is where they tried to GPT4 and they actually did try to prompt it in two ways, exactly like how I did. So they asked it for the red object. It did actually get the exhaust pipe or muffler correct. Then the object, object in the red circle. Actually of course it does say disk breaks. And this is something that I also did encounter. Now what they also tried to do was they tried to get GPT4 vision with the coordinates because it might be a bit more accurate. And they did actually get a more accurate answer on the first part of the question. But the second example, it once again just completely fails. Now I would like to see tons of more different examples because just one region with a motorcycle isn't exactly the best. But what's also fascinating was the grounding. Okay, so we can see here that this is a popular capture that we all know and that we all frequently see when signing up to different websites or signing into different applications. You can see right here that it says we have the traffic lights and it says detect all objects among the traffic light and it actually shows us where the traffic lights are. Then of course, ChatGPT tries to detect the traffic lights and it gets it wrong. You can see the Traffic lights in ChatGPT are highlighted here and there aren't any traffic lights here. Now I Think this is absolutely crazy because in terms of referring, you can see that GPT4 vision falls short in understanding relatively small regions. And similarly for grounding, GPT4 Vision fails to localize relatively small objects in complex scenes and specific regions. But as for grounding, we follow the Yang et al. Prompt and localize the image in using bounding boxes. The image size is width and height. And as we observed, GPT4VS is able to understand the referring to a certain extent, either colored region in the image or with the coordinates in the text. But when compared to Ferret, it does fall short in precise understanding with those really small regions. However, in the paper they actually did say that on the other hand, GPT4 vision is more knowledgeable, is more knowledgeable in common sense. For example, it can further highlight that the exhaust pipe can reduce the noise. And it does talk about the fact that GPT4's enhanced linguistic capabilities are much more advanced. Now, in regard to the grounding that we do see at the bottom here, Ferret does excel at identifying most traffic lights, even in the cluttered scenes, so the paper says. Nevertheless, Ferret shines especially when precise bounding boxes for grounding are needed and catering to those applications that require pinpoint accuracy in smaller regions. And this is precise, and this is precisely where Ferret steps in to fill the gap. So overall, if we compare GPT4 Vision to Apple's new multimodal Ferret model, it's clear that Ferret excels in accurately identifying small and specific regions in images, particularly in complex scenarios. But GPT4 can recognize areas outlined in red or specific in text, but tends to struggle with smaller regions. Whereas GPT4 Vision is knowledgeable and effective in general knowledge, question and answering related to the image regions, Ferrite actually stands out for its precision in pinpointing small areas, filling the crucial gap in a detailed image analysis. Now we can talk about some of the image implications of this, because previously, if this is very effective and very white might be, we might have a situation on our hands where we have vision models that really do help in terms of performing many different tasks that they weren't trained to. For example, there was a paper which actually was talking about how there were early explorations of visual language model on autonomous driving. So essentially this paper actually talked about how you could potentially use GPT4's vision capabilities for essentially just driving on the road. So of course everyone knows that there are different AI systems used in full self driving capabilities. And although we're not there yet, maybe GPT4 could help because it, it's essentially kind of like a mini AGI system that could interpret out of context scenarios. So you can see here that it's able to identify certain things and describe the image and exactly what was going on. And essentially what they did here was they tried to understand the traffic lights. They also tried to essentially say based on the image that you're seeing, what would your next thing be. And sometimes it did get it right. So red highlights the wrong understanding, green highlights the right understanding. And if we do get an image model that is really effective, we could be seeing these kind of models, maybe even more effective than some of the AI systems that we do have in cars and thus giving us the full self driving capabilities. Because we know that just being able to identify scenarios isn't good enough. Because what a lot of these car companies are facing is the fact that not every scenario is going to be the same. A lot of these training examples are in dry and very simple road conditions. Whereas we, when things are out of context, when there is snow, all these judgments and things that you need, kind of like a mini AGI system, which is exactly what Elon Musk said, are things that I guess you could say can't just be done with those AI systems. So this could mean that maybe we're just about to get some kind of huge update from Apple. I'm not entirely sure what they're working on, but this does bring us to the question and more importantly one of the big questions, which is where is Apple anyways? They've got Siri and they've been sitting on it for quite some time and you might be thinking what on earth are they going to release? Are they ever going to release any kind of AI model or any AI system? But I've got to be honest with you guys, you have to understand that Apple are a company that tends to wait. But this is the one time that I think Apple waiting might actually be a horrible situation because it's not like this is a traditional kind of technology. This is the kind of technology that does move very quickly and if you're not caught out you can be left behind. And Apple traditionally doesn't really care about what Samsung do because usually Samsung has the best features first. But with Apple people are loyal. They will wait for the features even if they're three years behind and even if the other side, which is usually just Samsung versus Android versus Apple, people will say that Android had it first. But Apple, the core die hard supporters will really not care and just simply say it doesn't matter. And then of course, it will be intriguing to see what Apple actually does, because as we know, anything is truly possible. Ladies and gentlemen, Apple have finally decided to make their entrance into the generative AI space. Apple recently announced something called Apple GPT. Apple GPT is an artificial intelligence language model rumored to be in development by Apple. It is expected to be similar to OpenAI's GPT3 and aims to enhance Siri's virtual assistant capabilities and other AI powered features in Apple's products. The informal name Apple GPT suggests that it could use a generative pre trained transformer model, the same kind of model that ChatGPT uses now. Apple GPT started as an experiment by a small team of Apple engineers in 2022 and is currently limited to internal use, assisting with prototyping future features. So it's clear that Apple has realized that the markets are moving very, very quickly and they do have an entirely new platform to deploy their generative AI features. From the new Apple Vision Pro to their new iPhones, Apple has a variety of applications that they could use to deploy their new Apple GPT. And as we stated, Siri seems to be getting a major, major upgrade. There are some predicted features that we do want to talk about. The most anticipated features of Apple GPT include better natural language understanding, which essentially means that when we do talk to Siri and when Siri talks back to us, the conversations are going to be a lot better than the monot tone ones that we do currently engage in. This is something that Apple hasn't really improved on since the major release of Siri. Number two is we're also going to get some improved text generation. As you know, sometimes when you're typing on your keyboard you do get a bunch of suggested words. And if the generative in generative pre trained transformer can actually allow us to get improved text generation, writing messages in imessage is going to get a whole lot easier and I'm pretty sure that this Apple GPT is probably going to assist you in many other Apple applications as well, such as notes, iMessage, WhatsApp and of course any word writing software. Number three is of course the enhanced conversational abilities. And this could mean that potentially we might be able to customize our own versions of Siri which could be unique to us. That would be really, really interesting and a unique spin on on what we already have with the generative pre trained transformers. Now these features are expected to improve Siri's contextual understanding, provide more accurate responses, and enable more realistic conversations with users. Now, in Comparison to other AI tools, Apple GPT is quite similar to other AI tools like ChatGPT and Google Bard in terms of performance and functionality. According to some sources. However, it's not publicly available yet and it's only accessible through a web interface for a select group of Apple employees. And according to many different resources such as Bloomberg, Apple is expected to make a major announcement about its AI efforts in 2024. So Apple GPT is a language model rumored to be in development by Apple and it seems that like in 2024 we're going to get a major overhaul. Now, we aren't sure at when in 2024 this groundbreaking announcement is supposed to be, but like many different Apple conventions, it's probably going to be one of Apple's live stream events that they host throughout the year when they're unveiling latest products or just doing a standard keynote. So we now essentially what Apple have done is they've upgraded autocorrect to the point where it actually uses machine learning. So before Apple used to use an archaic old version of machine learning to predict text, but now, as you know, as Google pioneered the way in actually creating the transformer architecture for people to now use the thing that actually makes ChatGPT so effective, which OpenAI actually built their chatbot around, this is what Apple are now essentially using for their autocorrect word prediction. So although this firstly wasn't a major announcement, it just goes to show that of course Apple as a big company as they are, are seriously paying attention to what is going on in the space. I mean, how could you not pay attention to the rapid rise of AI? There was also another small AI announcement in which many people also did miss, which was introducing Apple's new journal feature. So essentially what journal was is it's pretty much a feature that allows you to write down your journals, but it is going to be powered by an on device AI. The word that they actually used was on device machine learning. So essentially your iPhone can create personalized suggestions of moments to inspire your writing. Now they also stated that suggestions will be intelligently curated from information on your iPhone, like your photos, location, music, workouts and more. And then of course you can essentially control what suggestions that they pull from your phone. So essentially what we have here is an AI tool that is going to allow you to write more effectively by pulling from every single piece of data that it has on your phone, such as your photos and many other different sources. Now one thing that I did find very interesting about this talk from Apple was that they did refuse to mention the term artificial intelligence or AI. Now when you look at the transcript right here, you can see that AI isn't mentioned, but machine learning is actually mentioned in seven different times. Then of course we have this tweet from a user named Ethan Mollick and essentially it's a very, very valid point. So in this tweet he basically says that Apple didn't address the dead end that is Siri in the age 20 of, of AI. So if you don't know what Siri is, for those of you who don't use Apple, essentially it's a voice assistant that you can prompt by saying, hey Siri. And then your phone will simply go up and wake up with a woman who essentially asks you what would you like to do? Now it can be a man, it can be a woman. Essentially it's quite like Amazon's Alexa but for iPhone. Now the problem is, is that when you ask Siri for a restaurant prompt, which is exactly what this guy did, this is Siri's response versus what Microsoft's Bing can do with the same exact prompt. Now we do know of course Microsoft's Bing isn't voice activated, but it just goes to show that in the age of AI, why is Apple declining to spread any news or any advancements? Now I do have an answer for that and it's simply autonomous products. Now Apple has been actively acquiring a range of artificial intelligence companies in recent years with the aim of enhancing the AI and machine learning capabilities of its products and services. The list of companies acquired by Apple includes EmMotionT, a startup company that uses AI technology to read people's emotions by analyzing facial expressions. Turing, a small Seattle based startup specializing in machine learning and artificial intelligence. Realface, a cyber technology startup whose facial recognition technology can be used to authenticate users AI Music, a startup that uses AI to generate personalized soundtracks and adaptive music. Wave One, a California based startup that was developing AI algorithms for video compression. And to name some others, they acquired Shizam, Sensomotorik, SilkLabs, Drive AI, Laserlike, Spectral Edge, Voyosis, Xnor AI and many, many more. These acquisitions have allowed Apple to tap into the expertise and technology of these companies to develop advanced AI and machine learning capabilities for a range of applications. For example, the acquisition of Turi in 2020, 2016 gave Apple access to the company's expertise in developing machine learning tools and platforms. While the acquisition of Xnor AI in 2019 provided Apple with low power edge based AI technology for its products by investing in a wide range of AI companies, Apple has been able to stay at the forefront of the AI race and to drive innovation in the technology industry. The company has introduced a range of AI powered features in recent years, such as facial recognition in the iPhone X and Siri's improved natural language processing, and it has continued to invest heavily in AI research and development. Overall, Apple's acquisitions in the AI space demonstrate the company's commitment to staying ahead of the curve in the technology industry. By leveraging the expertise and technology of the companies it has acquired, Apple has been able to enhance the AI and machine learning capabilities of its products and services, driving improvements in user experience, efficiency, and productivity. Apple's extensive research into machine learning is a key part of the company's strategy for staying ahead of the curve in the technology industry. With a dedicated department focused on machine learning, Apple is able to invest heavily in research and development, driving innovation, and pushing the boundaries of what's possible with this technology. One way that Apple is demonstrating its commitment to machine learning is by regularly publishing research papers that highlight the company's innovative work in the field. These papers cover a wide range of topics, from computer vision and natural language processing to autonomous systems and data privacy. One recent example of Apple's innovative work and machine learning is the development of a program called Facelit. This program uses machine learning algorithms to create photorealistic 3D renders of a person's face just using two photos. This technology has significant applications in the field, such as virtual reality gaming and film production, and it demonstrates the potential of machine learning to drive advances in a wide range of industries. Overall, Apple's heavy focus on machine learning is a testament to the company's commitment to staying at the forefront of the technology industry. By investing heavily in research and development and sharing its work with the wider scientific community, and pushing the boundaries of what's possible with machine learning, Apple ensures that it remains a major player in the AI race.