Google DeepMind's New AI Model - TAPIR: Seeing Through the Lens of AI

So Researchers at Google DeepMind have created a new AI model known as Taper. And it's honestly one of the coolest things I've seen in the field of computer vision. Computer vision is a branch of artificial intelligence that deals with understanding and analyzing visual information such as images, videos or live streams. Its models can do amazing things like recognize faces, detect objects, segment scenes, generate captions, and much more. These models can help us derive meaningful insights from different types of media and use them for various applications such as security, entertainment, education, healthcare, and so on. But how do computer vision systems function? Well, they use deep learning techniques to learn from large amounts of data and extract features that are relevant for the task at hand. For example, if you want to recognize a person's face in a photo, you need a model that can learn to identify the key characteristics of a face, such as the shape of the eyes, nose, mouth, etc. Then you need a model that can compare the features of the face in the photo with the features of the faces in your database and find the best match. Sounds simple enough, right? But what if you want to track a specific point on a person's face or any other object in a video sequence? For example, what if you want to track the tip of someone's nose or the center of their pupil as they move around in a video? This is where things get tricky. You see, tracking a point in a video is not as easy as finding a point in a single image. You have to deal with challenges like occlusion, motion, blur, illumination changes, scale variations, and so on. These factors can make it hard for the model to keep track of the point as it moves across different frames. Now this is where taper comes into the picture. Taper stands for tracking any point with per frame initialization and temporal refinement. It's a new model that can effectively track any point on any physical surface throughout a video sequence. And it doesn't matter if the point is on a person's face, a car's wheel, a bird's wing, or anything else, it can handle it all. It was developed by a team of Researchers from Google DeepMind VGG Department of Engineering Science and the University of Oxford. They published their paper on ARXIV on June 14, 2023 and they also open sourced their code and pre trained models on GitHub. You can. You can find the links to their paper and code in the description below. Okay, so how does taper work? Well, it uses a two stage algorithm that consists of a matching stage and a refinement stage. The matching stage is where it analyzes each video frame separately and tries to find a suitable candidate point match for the query point. The query point is the point that you want to track in the video sequence. For example, if you want to track the tip of someone's nose in a video, then that's your query point. To find the candidate point match for the query point in each frame, we it uses a deep neural network that takes as input an image patch around the query point and outputs a feature vector that represents its appearance. Then it compares this feature vector with the feature vectors of all possible points in each frame using cosine similarity and picks the most similar one as the candidate point match. This way, Taper can find the most likely related point for the query point in each frame independently. This makes it robust to occlusion and motion blur, because even if the query point is not visible or clear in some frames, it can still find its best approximation based on its appearance. But finding candidate point matches is not enough to track the query point accurately. You also need to take into account how the query point moves over time and how its appearance changes due to factors like illumination or scale variations. This is where the refinement stage comes in. The refinement stage is where Taper updates both the trajectory and the query features based on local correlations. The trajectory is the path followed by the query point throughout the video sequence, and the query features are the feature vectors that represent its appearance. Now, to update the trajectory and the query features, it uses another deep neural network that takes as input a small image patch around the candidate point match in each frame, and outputs a displacement vector that indicates how much the candidate point match should be shifted to match the query point more precisely. Then it applies this displacement vector to the candidate point match to obtain a refined point match that is closer to the true query point. In simple terms, the system works by examining small parts of an image, figuring out how much to adjust a selected point to match a target point, and then moving the selected point closer to that target. Tapir also updates the query features by averaging the feature vectors of the refined point matches over time. This way, it can adapt to changes in the query point's appearance and maintain a consistent representation of it. By combining these two stages, it can track any point in a video sequence with high accuracy and precision. It can handle videos of various sizes and quality, and it can track multiple points simultaneously. Alright, now let's see how Tapir performs on some benchmarks and demos. The researchers evaluated Tapir using the TapVid benchmark, which which is a standardized evaluation dataset for video tracking tasks. It contains 50 video sequences with different types of objects and scenes and it provides ground truth annotations for 10 points per video. They compared Taper with several baseline methods such as SIFT, Orbi, KLT, Superpoint, and D2.NET. They measured the performance using a metric called average jacquard, which is the average intersection over union between the predicted point locations and and the ground truth point locations. The results showed that Tapir outperformed all the baseline methods by a significant margin on the TapVid benchmark. It achieved an AJ score of 0.64, which is about 20% higher than the second best method, D2.net, which scored 0.44. This means that TAPIR was able to track the points more closely to their true locations than any other method. It also performed well on another benchmark called davis, which is a dataset for video segmentation tasks. It contains 150 video sequences with different types of objects and scenes and it provides ground truth annotations for pixel level segmentation masks. The researchers used taper to track 10 points per video on Davis and computed the AJ score. As before, they found that Tapir achieved an AJ score of 0.59, which is again about 20% higher than the second best method, D2.net, which scored 0.39. This means that it was able to track the points more consistently across different frames than any other method. But benchmarks are not enough to show you how awesome Taper is. You need to see it in action yourself. And luckily the researchers have provided two online Google Colab demos that you can use to run taper on your own videos. The first demo is called Tap Vid Demo and it allows you to upload your own video or choose one from YouTube and then select any point on any object in the first frame that you want to track throughout the video. Then it runs Taper on your video and shows you the results in real time. The second demo is called Webcam Demo and it allows you to use your own webcam as the input source and then select any point on your face or any other object in front of you that you want to track live as you move around. Then it runs tape here on your webcam feed and shows you the results in real time. I've tried both demos myself and I have to say they are insane. You can see how Tapeer can track any point on any object with amazing accuracy and precision, even when there are occlusions, motion blur, illumination changes, scale variations, and so on. It's honestly fascinating. Alright, I hope you enjoyed the video and learning about Tapir. I think this is a huge breakthrough for computer vision and I can't wait to see what kind of applications this model can enable in the future. If you liked this video, please give it a thumbs up and subscribe to my channel for more videos like this one. And don't forget to hit the bell icon so you don't miss any of my future uploads. Thank you so much for watching and I'll see you next time.