An algorithm called pix2code converts screenshots of user interfaces into working code for iOS, Android, or HTML. A convolutional neural network reads the image while an LSTM generates the code sequence. The creators plan to release the source code and dataset on GitHub.
Dear fellow scholars, this is two minute papers with creating applications for mobile, Android and iOS devices is a laborious endeavor which most of the time includes creating a graphical user interface. These are the shiny front end interfaces that enable the user to interact with the backend of our applications. So what about an algorithm that learns how to create these graphical user interfaces and and automates part of this process? This piece of work takes one single input image that we can trivially obtain by making a screenshot of the user interface, and it almost immediately provides us with the code that is required to recreate it. What an amazing idea. The algorithm supports several different target platforms. For instance, it can give US code for iOS and Android devices. This code we can hand over to a compiler which will create an executable application. This technique also supports HTML as well for creating websites with the desired user interface under the hood, a domain specific language is being learned and using this it is possible to have a concise text representation of a user interface. Note that's by no means the only use of domain specific languages. The image of the graphical user interface is learned by a classical convolutional neural network, and this text representation is learned by a technique machine learning researchers like to call Long Short term memory lstm. In short, this is a neural network variant that is able to learn sequences of data and is typically used for language translation, music composition, or learning all the novels of Shakespeare and writing new ones in his style. If you were wondering why these examples are suspiciously specific, we've had an earlier episode about this. I've put a link to it in the video description. Make sure to have a look. You're going to love it. Also, this year it will have its 20th year anniversary. Live long and prosper little LSTM. Now I already see the forums go up in flames, sweeping generalizations, far reaching statements on front end developers around the world getting fired and all that. I'll start out by saying that I highly doubt that this work would mean the end of front end development jobs in the industry. However, what I do think is that with a few improvements it can quickly prove its worth by augmenting human labor and cutting down the costs of implementing graphical user interfaces in the future. And this is another testament to the variety of tasks modern machine learning algorithms can take care of. The author also has a GitHub repository with a few more clarifications stating that the source code of the project and the dataset will be available soon. Tinkerers Rejoice. Thanks for watching and for your generous support and I'll see you next time, Sam.
