YouTube explainer (LARRUZ, Aug 10, 2018) describing DeepMind's procedurally generated visual IQ-test dataset and a dedicated model (Wild Relation Network). Standard architectures trained on large amounts of such examples perform poorly (22–42% accuracy), while the Wild Relation Network achieves 62.6% accuracy with visual distractors and 78% without; the dataset was publicly released by the authors, inviting further research.
Dear fellow scholars, this is two minute papers with throughout this series, we have seen many impressive applications of artificial intelligence. These techniques are capable of learning the piano from the masters of the past, beat formidable teams in complex games like Dota 2, perform well in the game Sonic the Hedgehog, or help us revive and impersonate famous actors who are not with us anymore. However, what is often not spoken about is how narrow or how general these AI programs are. A narrow AI means an agent that can perform one task really well, but cannot perform other potentially easier tasks. The holy grail of machine learning research is a general AI that is capable of obtaining new knowledge by itself through abstract reasoning. This is similar to how humans learn and and is a critical step in obtaining a general AI. And to tackle this problem, scientists at DeepMind created a program that is able to generate a large amount of problems that test abstract reasoning capabilities. They are inspired by human IQ tests with all these questions about sizes, colors and progressions. They designed the training process in a way that the algorithm is given training data on the progression of colors, but it is never shown similar progression examples that involve object sizes. The concept is the same, but the visual expression of the progression is different. A human easily understands the difference, but teaching abstract reasoning like this to a computer sounds almost impossible. However, now we have a tool that can create many of these questions and the correct answers to them. And I will note that some of these are not as easy as many people would expect. For instance, a vertical number progression is very easy to spot. But have a good look at these ones not so immediately apparent, right? Going back to being able to generate lots and lots of data, the Black Belt fellow scholars know exactly what this means. This means that we can train a neural network to perform this task. Unfortunately, existing techniques and architectures perform quite poorly. Despite the fact that we have a ton of training data, they could only get 22 to 42% of the answers right. However, these networks are amazing at doing other things like writing novels or image classification. Therefore, this means that their generalization capabilities are not too great when we go outside their core domain. This new technique goes by the name Wild relation network and is trained in a way that encourages reasoning. It is also designed in a way that it not only outputs a guess for the results, but also tries to provide a reason for it, which interestingly further improved the accuracy of the network. And what is this accuracy we are talking about? It finds the correct solution 62.6% of the time, but it gets better because this result was measured in the presence of distractor objects like these annoying lines and circles. This is quite confusing even for humans, so a result above 60% is quite remarkable. And it gets even better because if we don't use these distractions, it is correct 78% of the time. Wow. This is indeed a step towards teaching an AI how to reason. And as the authors made this dataset publicly available for everyone, I expect a reasonable amount of research works appearing in this area in the near future. Who knows, perhaps even in the next few months. Thanks for watching and for your generous support, and I'll see you next time.
