Google’s New AI: DALL-E, But Now In 3D! 🤯

Dear fellow scholars, this is 2 Minute Papers with Dr. Today we are going to see how this new AI is able to take a piece of text from us. Anything we wish, be it a squirrel dressed like the King of England, a car made out of sushi, or a humanoid robot using a laptop. And magically it creates not an image like previous techniques, but get this, a full 3D model of it. Wow. This is absolutely amazing. An AI that can not only create images, but create 3D assets. Yes, indeed. The result is a full 3D model that we can rotate around and even use in our virtual worlds. So let's give it a really hard time and see together what it is capable of. For instance, OpenAI's earlier Dall? E text to Image AI was capable of looking at a bunch of images of koalas, and separately a bunch of images of motorcycles. And it started to understand the concept of both. And it was able to combine the two together into a completely new image. That is a koala riding a motorcycle. So let's see if this new method is also capable of creating new concepts by building on previous knowledge. Well, let's see. Oh yes, here is a tiger wearing sunglasses and a leather jacket. And most importantly, riding a motorcycle. Tigers and motorcycles are well understood concepts. Of course, the Neural Network had plenty of these to look at in his training set. But combining the two concepts together, now that is a hint of creativity. Creativity in a machine. Loving it. What I also loved about this work is that it makes it so easy to iterate on our ideas. For instance, first we can start experimenting with a real squirrel. Or if we did not like it, we can quickly ask for a wooden carving or even a metal sculpture of it. Then we can start dressing it up and make it do anything we want. And sometimes the results are nearly good enough to be used as is even in an animation movie or in virtual worlds. Or even in the worst cases. I think these could be used as a starting point for an artist to continue from. That would save a ton of time and energy in a lot of projects, and that is huge. Just consider all the miraculous things artists are using. The Dall E2 text to image AI and stable diffusion for illustrating novels, texture synthesis, product design, weaving multiple images together to create these crazy movies, you name it. And now I wonder what unexpected uses will arise from this being possible for 3D models. Do you have some ideas? Let me know in the comments below and just imagine what this will be capable of. Just a couple more papers down the line. For instance, the original Dall? EI was capable of this. And then just a year later this became possible. So how does this black magic work? Well, the cool thing is that this is also a diffusion based technique, which means that similarly to the text to image AIs, it starts out from a piece of noise and refines this noise over time to resemble our input text a little more. But this time the diffusion process is running in higher dimensions. Thus the result is not a 2D image, but a full 3D model. So from now on, the limit in creating 3D worlds is not our artistic skill. The limit is only our imagination. What a time to be alive. Weights and Biases provides tools to track.

Your experiments in your deep learning projects. What you see here is their amazing sweeps feature which helps you find and reproduce your best runs. And even better, what made this particular run the best. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.

Thanks for watching and for your generous support and I'll see you next time.