Neural networks reconstruct 3D CAD models of indoor rooms from a single photo by detecting furniture, matching it to a database of real products, and refining the scene to match the original image. The whole process takes about five minutes, though it struggles with non-rectangular rooms and can misalign objects.
Around five minutes.
Dear fellow scholars, this is two minute papers with CAD stands for Computer Aided Design. Basically a digital 3D model of a scene. Here is an incredibly difficult problem. What if we give the computer a photograph of a living room and the output would be a Digital fully modeled 3D scene? A CAD model? This is a remarkably difficult problem. Just think about it. The algorithm would have to have an understanding of perspective, pers, illumination, occlusions and geometry. And with that in mind, have a look at these incredible results. Now this clearly sounds impossible. Not so long ago we talked about a neural network based technique that tried to achieve something similar, and it was remarkable. But the output was a low resolution voxel array, which is kind of like an approximate model built by children from a few large Lego pieces. The link to this work is available in the video description. But clearly we can do better. So how could this be possible? Well, the most important observation is that if we take a photograph of a room, there is a high chance that the furniture within are not custom built, but mostly commercially available pieces. So who said that we have to build these models from scratch? Let's look into a database that contains the geometry for publicly available furniture pieces and find which ones are seen in the image. So here's what we do. Given a large amount of training samples, neural networks are adept at recognizing objects on a photograph. That would be step number one. After the identification, the algorithm knows where the object is. Now we are interested in what it looks like and how it is aligned. And then we start to look up public furniture databases for objects that are as similar to the ones presented in the photo as possible. Finally, we put everything in its appropriate place and create a new digital image with a light simulation program. This is an iterative algorithm, which means that it starts out with a coarse initial guess that is being refined many, many times until some sort of convergence is reached. Convergence means that no matter how hard we try, only minor improvements can be made to this solution. Then we can stop. And here the dissimilarity between the photograph and the digitally rendered image was subject to minimization. This entire process of creating the 3D geometry of the scene takes around five minutes. And this technique can also estimate the layout of a room from this one photograph. Now, this algorithm is absolutely amazing, but of course the limitations are also to be candidly discussed. While while some failure cases arise from misjudging the alignment of the objects, the technique is generally quite robust. Non cubic room shapes are also likely to introduce issues such as the omission or misplacement of an object. Also, kitchens and bathrooms are not yet supported. Note that this is not the only paper solving this problem. I've made sure to link some more related papers in the video description for your enjoyment. If you have found this interesting, make sure to subscribe and stay tuned for more 2 Minute Papers episodes. Thanks for watching and for your generous support and I'll see you next time.
