Start from the center, or identify and fill salient regions first.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Everybody loves style transfer. This is a task typically done with neural networks, where we have two images, one for content and one for style. And the output is the content image reimagined with this new style. The cool thing is that the style can be a different photo, a famous painting, or even wooden patterns. Feast your eyes on these majestic images of this cat reimagined with wooden parquetry with these previous methods. And now look at the result of this new technique that looks way nicer. Everything is in order here, except one thing. And now hold on to your papers, because this is not style transfer. Not at all. This is not a synthetic photo made by a neural network. This. This is a reproduction of this cat image by cutting wood slabs into tiny pieces and putting them together carefully. This is computational parquetry. And here the key requirement is that if we look from afar, it looks like the target image. But if we zoom in, it gets abundantly clear that the puzzle pieces here are indeed made of real wood. And that is an excellent intuition for this work. It is kind of like image stylization, but done in the real world. Now, that is extremely challenging. Why is that? Well, first, there are lots of different kinds of wood types. Second, if this piece was not a physical object, but an image, this job would not be that hard, because we could add to it, clone it, and do all kinds of pixel magic to it. However, these are real, physical pieces of wood, so we can do exactly none of that. The only thing we can do is take away from it. And we have limitations even on that, because we have to design it in a way that a CNC device should be able to cut these pieces. And third, you will see that initially, nothing seems to work well. However, this technique does this with flying colors. So I wonder, how does this really work? First, we can take a photo of the wood panels that we have at our disposal and decide how and where to cut. Give these instructions to the CNC machine to perform the cutting. And now we have to assemble them in a way that it resembles the target image. Well, still, that's easier said than done. For instance, imagine that we have this target image and we have these wood panels. This doesn't look anything like that, so how could we possibly approximate it? If we try to match the colors of the two, we get something that is too much in the middle, and these colors don't resemble any of the original inputs. Not good. Instead, the authors opted to transform both of them to grayscale and match not the colors, but the intensities of the colors instead. This seems a little more usable until we realize that we still don't know what pieces to use and where. Look here on the left. You see how the image is being reproduced with the wood pieces. But we have to mind the fact that as soon as we cut out one piece of wood, it is not available anymore. So it has to be subtracted from our wood panel repository here as our resources are constrained. Depending on what order we put the pieces together, we may get a completely different result. But look, there is still a problem. The left part of the suit gets a lot of detail, while the right part not so much. I cannot judge which solution is better, less or more detail, but it needs to be a little more consistent over the image. Now you see that whatever we do, nothing seems to work well in the general case. Now, we could get a much better solution if we would run the algorithm with every possible starting point in the image and and with every possible ordering of the wood pieces. But that would take longer than our lifetime to finish. So what do we do? Well, the authors have two really cool heuristics to address this problem. First, we can start from the middle. That usually gives us a reasonably good solution since the object of interest is often in the middle of the image and the good pieces are still available for it. Or even better, if that does not work too well, we can look for salient regions. These are the places where there's a lot going on and try to fill them in first. As you see, both of these tricks seem to work quite well most of the time. Finally, something that works. And if you have been holding onto your paper so far, now squeeze that paper. Because this technique not only works but but provides us a great deal of artistic control over the results. Look at that. And that's not all. We can even control the resolution of the output. Or we can create a hand drawn geometry ourselves. I love how the authors took a really challenging problem where nothing really worked well. And still they didn't stop until they absolutely nailed the solution. Congratulations. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lambda GPU Cloud. They recently launched Quadro RTX 6000, RTX 8000 and V100 instances. And hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure. Plus they are the only cloud service with 48 gigabyte RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
