Researchers built a system that turns 2D floor plans and photos into 3D virtual apartment tours. It uses the floor plan's geometry, image classifiers to identify room types, and optimization to figure out where each photo belongs on the walls. The team released a dataset of over 200 apartments with 1,000+ annotated photos.
Dear fellow scholars, this is two minute papers Withehir. In this piece of work we are interested in creating a 3D virtual tour for an apartment. However, for this apartment no 3D information is available. Instead, the input for the algorithm is something that we can obtain easily. In this case, a 2D floor plan and a set of images that we shot in the apartment. From this information we would create a 3D floor plan that is not only faithful to the real one in terms of geometry, but the photo with the correct viewpoints are also to be assigned to the correct walls. In order to accomplish this, one has to overcome a series of challenging problems. For instance, we have to estimate the layout of each room and find the location of the camera in each of these images. Also, to obtain high quality solutions, the goal is to extract as much information from the inputs as possible. The authors recognized that the floor plans provide way more information than we take for granted. For instance, beyond showing the geometric relation of the rooms, it can also be used to find out the aspect ratios of the floor for each room. The window to wall ratios can also be approximated and matched between the photos and the floor plan. This additional information is super useful when trying to find out which room is to be assigned to which part of the 3D floor plan. Beyond just looking at the photos, we also have access to a large swath of learning algorithms that can reliably classify whether we are looking at a B bathroom or a living room. There are even more constraints to adhere to in order to aggressively reduce the number of physical configurations. Make sure to have a look at the paper for details. There are lots of cool tricks described there. As always, there's a link to it in the video description. For instance, since the space of possible solutions is still too vast, a branch and bound type algorithm is proposed to further decimate the number of potential solutions to evaluate. And as you can see here, the comparisons against ground truth floor plans reveal that these solutions are indeed quite faithful to reality. The authors also kindly provided a data set with more than 200 full apartments with well over a thousand photos and annotations for future use in follow up research works. Creating such a data set and publishing it is incredibly laborious and could easily be a paper on its own. And here we also get an excellent solution for this problem as well. As in a separate work, the authors also published a different version of this problem formulation that reconstructs the exteriors of buildings in a similar manner. There is so much to explore. The links are available in the video description. Make sure to have a look. In case you're wondering, it's still considered a crime not doing that. I hope you have enjoyed this episode, and I find it so delightful to see this unbelievably rapid growth on the channel. Earlier, I thought that even two would be amazing, but now we have exactly eight times as many subscribers as one year ago. Words fail me to describe the joy of showing these amazing works to such a rapidly growing audience. This is why I always say at the end of every episode, thanks for watching and for your generous support, and I'll see you next time.
