Dr. Zsolnai breaks down Omniphotos, a technique that stitches 360-degree photos from consumer cameras without machine learning. The method uses a selfie stick waved in a rough circle, then estimates camera positions and warps the images together. Code and a working demo are available online.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Virtual reality VR in short, is maturing at a rapid pace and its promise is truly incredible. If one day it comes to fruition, doctors could be trained to perform surgery in a virtual environment. We could train pilots with better flight simulators, teach astronauts to deal with zero gravity simulations, you name it. And you will see that with today's paper we are able to visit nearly any place from afar. Now, to be able to do anything in a virtual world, we have to put on a head mounted display that can tell the orientation of our head and often hands at all times. So what can we do with this? Oh boy, a great deal. For instance, we can type on a virtual keyboard or implement all kinds of virtual user interfaces that we can interact with. We can also organize imaginary boxes. And of course we can't leave out the two minute paper's favorite, going into a physics simulation and playing with it with our own hands. In this previous work, hand hand interactions did not work too well, which was addressed one more paper down the line which absolutely nailed the solution. This follow up work would look at our hands in challenging hand to hand interactions and could deal with deformations, lots of self contact and self occlusion. Take a look at this footage and look. Interestingly, they also recorded the real hand model with gloves on. We might think, what a curious design decision. What could that be for? Well, what you see here is not a pair of gloves. What you see here is the reconstruction of the hand by this follow up paper. Absolute witchcraft. Now as you see, we can manipulate objects or even wash our hands in virtual reality. This is all great when we play in a computer game because the entire world around us was previously modeled so we can look and go anywhere, anytime. But what about operating in the real world? What if we wish to look at a historic landmark from afar? Well, in this case, someone needs to capture a 360 degree photo of it because we can't turn our head around and look behind things. And this is what today's paper will be about. This new paper is called Omniphotos and it helps us produce this 360 view synthesis. And when we put on that head mounted display, we can really get a good feel of a remote place, a group photo or an important family festivity. So clearly the value proposition is excellent. But we have two questions. One, what do we have to do for it? Flailing. Yes, we need to be flailing. You see, we need to attach a consumer360 camera to a selfie Stick and start flailing for about 10 seconds like this. This is a crazy idea because now we created a ton of raw data, roughly what you see here. So this is a deluge of information and the algorithm needs to crystallize all this mess into a proper 360 photograph. What is even more difficult here is that this flailing will almost never create a perfect circle trajectory. So the algorithm first has to estimate the exact camera positions and view directions. And hold on to your papers. Because the entirety of this work is handcrafted, no machine learning is in sight and the result is a quite general technique. Or in other words, it works on a wide variety of real world scenes. You see a good selection of those here. Excellent. Our second question is this is of course not the first method published in this area. So how does it relate to previous techniques? Is it really better? Well, let's see for ourselves. Previous methods either suffered from not allowing too much motion or the other ones that give us more freedom to move around did it by introducing quite a bit of warping into the outputs. And now let's see if the new method improves upon that. Oh yeah, a great deal. Look, we have the advantages of both methods. We can move around freely. And additionally there is much less warping than here. Now of course, not even this new technique is perfect. If you look behind the building, you see that the warping hasn't been completely eliminated, but it is a big step ahead of the previous paper. While we look at some more side by side comparisons, we. One more bonus question. What about memory consumption? Well, it eats over a gigabyte of memory. That is typically not too much of a problem for desktop computers, but we might need a little optimization if we wish to do these computations on a mobile device. And now comes the best part. You can browse these omniphotos online through the link in the video description and even the source code. And. And a Windows based demo is available that works with and without a VR headset. Try it out and let me know in the comments how it went. So with that we can create these beautiful omniphotos cheaply and efficiently and navigate the real world as if it were a computer game. What a time to be alive. What you see here is a report for a previous paper that we covered in this series which was made by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through WNB.com papers or or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
