Károly Zsolnai-Fehér explains an AI technique that matches points between images by comparing neural network representations instead of pixel colors. It lets artists stitch or blend images with colored masks, and morph across different styles like cartoon to photo or cat to dog with smooth, meaningful transitions.
Dear fellow scholars, this is two minute papers with Consider this. We have a pair of images that are visually quite different but have similar semantic meanings and we wish to map points between them. Now this might sound a bit weird, so bear with me for a moment. For instance, geese and airplanes look quite different, but both have wings and front and back regions. The paw of a lion looks quite different from a cat's foot, but they share the same function and are semantically similar. This is an AI based technique that is able to find these corresponding points between our pair of images. In fact, the point pairs you've seen so far have been found by this AI. The main difference between this and previous non learning based techniques is that instead of pairing up regions based on pixel color similarities, it measures how similar they are in terms of the neural network's internal representation. This makes all the difference so far. This is pretty cool, but is that it? Mapping points? Well, if we can map points effectively, we can map regions as a collection of points. This enables two killer applications. One, this can augment already existing artistic tools so that we can create a hybrid between two images. And the cool thing is that we don't even need to have any drawing skills because we only have to add these colored masks and the algorithm finds and stitches together the corresponding images. And two, it can also perform cross domain image morphing. That's an amazing term, but what does this mean? This means that we have our pair of images from earlier and we are not interested in stitching together a new image from their parts, but we want an animation where the starting point is one image, the ending point is the other, and we get a smooth and meaningful transition between the two. There are some really cool use cases for this. For example, we can start out from a cartoon drawing, set our photo as an endp, and witness this beautiful morphing between the two. Kind of like in style transfer, but we have more fine grained control over the output. Really cool. And note that many images in between are usable as is no artistic skills needed. And of course there is a mandatory animation that makes a cat from a dog. As usual, there are lots of comparisons to other similar techniques in the paper. This tool is going to be invaluable for, I was about to say artists, but this doesn't require any technical expertise, just good taste and a little bit of imagination. What an incredible time to be alive. Thanks for watching and for your generous support and I'll see you next time.
