Károly Zsolnai-Fehér shows how video game footage can replace hand-labeled photos for training computer vision. Labeling real images takes 60 minutes each; game footage takes 7 seconds per image and produced 25,000 annotated frames with comparable learning quality.
Dear fellow scholars, this is two minute papers with what are datasets? A dataset is typically a big bunch of data. For instance, a database of written letters, digits, images of human faces, stock market data that scientists can use to test their algorithms on. If two research groups wish to find out whose algorithm performs better at recognizing traffic signs, they run their techniques on one of these datasets and test their methods on equal footings. For instance, the camvid dataset stands for Cambridge Driving Labeled Video Database, and it offers several hundreds of images depicting a variety of driving scenarios. It is meant to be used to test classification techniques. The input is an image, and the question is, for each of the pixels, which one of them belongs to what class? Classes include roads, vegetation, vehicles, pedestrians, buildings, trees, and more. These regions are labeled with all the different colors that you see on these images. To have a usable data set, we have to label tens of thousands of these images. And as you may imagine, creating such labeled images requires a ton of human labor. The first guy has to accurately trace the edges of each of the individual objects seen on every image, and there should be a second guy to cross check and make sure everything is in order. That's quite a chore. And we haven't even talked about all the other problems that arise from processing footage created with handheld cameras. So this takes quite a bit of time and effort with stabilization and calibration as well. So how do we create huge and accurate datasets without investing a remarkable amount of human labor? Well, hear out this incredible idea. What if we could record a video of us wandering about in an open world computer game and annotate those images? And this way we can enjoy several advantages. One, since we have recorded continuous videos, after annotating the very first image, we will have information from the next frames. Therefore, if we do it well, we can propagate the labeling from one image to the next. 1. That's a huge time saver. 2, in a computer game, one can stage and record animations of important but rare situations that would otherwise be extremely difficult to film. Adding rain or day and night cycles to a set of images is also trivial because we simply can query the game engine to do this for us. Three, not only that, but the algorithm also has some knowledge about the rendering process itself. This means that it looks at how the game communicates with the software drivers and the video card tracks when the geometry and textures for a given type of car are being loaded or discarded, and uses this information to further help the label propagation process. And number four, we don't have any of the problems that stem from using handheld cameras. Noise, blurriness, problems with the lens, and so on are all non issues. Using this previous CANVID dataset, The annotation of one image takes about 60 minutes while with this dataset 7 seconds. Thus, the authors have published almost 25,000 high quality images and their annotations to aid computer vision and machine learning research in the future. That's a lot of images. But of course the ultimate question how do we know if these are really high quality training samples? They were only taken from a computer game after all. Well, the results show that using this dataset we can achieve an equivalent quality of learning compared to the CAMVID dataset by using one third as many images. Excellent piece of work. Absolutely loving the idea of using video game footage as a surrogate for real world data. Fantastic. And in the meantime, while we are discussing computer graphics, here's a nice computer graphics challenge grant from Experiment. Basically, if you start a new research project through their crowdfunded system, you may win additional funding that comes straight from them. Free money. If you are interested in doing any kind of research in this area, or if you are a long time practitioner, make sure to have a look. The link is available in the video description box. Thanks for watching and for your generous support and I'll see you next time.
