Zuck's Metaverse Revenge

Mark Zuckerberg has had a couple of rough months, but he's planning a comeback. Even though his company's stock is down nearly 50%, he's not worried about it. See, Zuck is in a very unique position. Even though Meta has lost hundreds of billions of dollars in value since the peak of the Metaverse hype wave, they aren't taking their foot off the gas. Zuck is extremely motivated to build the Metaverse, and he knows it's not going to happen overnight. He has a chip on his shoulder right now, and if he can pull it off, he'll make every YouTube video about him struggling look ridiculous in hindsight. So what's his plan? Well, Zuck has a few tricks left up his sleeve, but let's start with artificial intelligence. If you ask a random person what technologies will be most important in the Metaverse, they will almost always say virtual reality. And that's not wrong. VR is going to be important. But a big part of the problem with VR right now is that there just isn't much to do. Once you put on a headset, the experiences aren't very immersive, and the amount of content is pretty meager. But AI can help here. We're still years away from massive virtual worlds being built automatically by AI, but there are dozens of small places where AI can speed up development and improve user experience. One of the big ideas underpinning this whole Metaverse thing is the ability to move digital assets from one virtual world to another. But whenever I mention the idea of bringing a Fortnite skin into a different game like Minecraft, I immediately get tons of comments saying that it's impossible because different games use different file formats and rendering technologies. And that's true. But AI could make it a lot easier to translate assets from one system to another. That technology doesn't exist yet, but AI is already handling other translation tasks incredibly well. So well, in fact, that when Zuck hosted the most recent Meta AI event, translation was one of the main topics. Zuck is planning on spending $180 billion on the metaverse, and a decent chunk of that is going to be used to develop advanced AI to make the Metaverse feel more immersive. We've all seen how important hand presence is in cinematic depictions of VR. The current Oculus touch controllers are pretty good, but you're never really fooled into thinking that your hands are actually there in the virtual world. It still feels like you're holding a controller. But Zuck and his team are working on ways to change that. Deep learning models are already getting pretty good at hand tracking. But the Meta AI team isn't stopping there. They recently developed a system called Reskin, which uses a deformable membrane with embedded magnetic particles to detect movement. As the hand moves, the magnetic signal changes. Then AI is used to interpret that signal and detect where the glove is making contact. None of that matters though, unless you actually have a virtual world that you want to interact with. And that's where Zuck's second big announcement comes in. BuilderBot is a new AI that takes in simple descriptions of environments in plain English and then produces 3D worlds to match that input. Building virtual worlds is getting easier every year, but even a state of the art platform like Unreal Engine 5 still requires a decent amount of technical expertise to get up and running. By developing this Builder Bot AI, the hope is that everyone, regardless of their background, will be able to create exactly what they want in the Metaverse. Bringing other useful tools into the metaverse will be important as well. So the Meta AI team is working on next generation virtual assistants. Current chatbots are pretty bad, but it's because most of them use outdated technology. Machine learning is used to recognize speech, but pretty much everything else has to be coded by hand. Meta's AI team is working on changing this by building a system called Karaoke that processes the entire bot interaction through machine learning. This AI will still use traditional APIs to do specific tasks like find the weather forecast, but essentially your entire interaction will be fed through a deep learning algorithm. By layering in more and more artificial intelligence, these bots should start to become extremely reliable and then become available in virtual worlds through avatars. Training these new models is no easy feat though. Meta has an incredible amount of data, but actually using it all to train a new system requires a massive supercomputer. Zuck has been investing in AI hardware for nearly a decade now, and is on the cusp of rolling out the fast, fastest AI supercomputer in the world. It's called the AI Research Supercluster and it already has over 6,000 top tier Nvidia GPUs installed, with another 10,000 coming online soon. This will allow Meta to train models with more than a trillion parameters on datasets as large as an exabyte, which is the equivalent of 36,000 years of high quality video. That might sound unnecessary, but deep learning thrives on massive datasets. There's a reason why deep learning has become the go to method for building state of the art AI systems. It scales incredibly well as you add more and more data. The results just get more and more reliable, and you don't really need to change that much of the code to get these performance improvements. This computing power allows Meta's AI team to tackle increasingly complex problems. We've had decent translation for years now, but there have always been drawbacks to the automated approach. Google Translate launched in 2006 and used statistical machine translation to convert English to other languages. It wasn't great at grammar, though. Billions of people use Facebook, Instagram and WhatsApp. But because of language barriers, they can't all easily communicate with each other. Zuck wants to change that by improving translation using AI, and his team is working on a unique approach. Instead of taking what someone said, transcribing it to text, translating that text and then generating the resulting speech, his AI team is working on going directly from speech in one language to speech in another. They're effectively cutting out the middleman, which has been a source of pesky errors for years. It's a subtle change, but if it works, audio translation should get a lot better. Now, Zuck is in a strong position to do this precisely because Meta has so much training, data and the supercomputer to process it all. But data and hardware are just one piece of the puzzle here. All those resources are meaningless if you don't have talented people working on these problems. Fortunately for Zuck, he has one of the best researchers in the world leading his AI team. His name is Yann Lecun, and he's one of the fathers of deep learning. He pioneered this approach alongside Geoffrey Hinton and Yoshua Bengio in the early 2000s, and is now recognized as one of the best computer scientists in history. It wasn't always so clear that Lecun's approach would work, though. In traditional computing, a program directs the computer with explicit step by step instructions. And pretty much everyone thought this would always be the right way to do things. This approach was stalling out, though. It was great when tasks were clearly defined, but when you gave a computer an entirely novel situation, it would struggle. Artificial intelligence research was going nowhere fast, and we entered an AI winter. LeCun and his associates kept the candle burning and eventually turned things around. Advances in computer hardware and access to larger data sets finally came together to allow deep learning to take off. And the AI community hasn't looked back. Everything from self driving cars to tagging photos of cats taken with mobile phones using uses deep learning these days. But recently a new challenger has emerged. His name is Gary Marcus, and he believes that all this deep Learning stuff is overhyped. Could Zuckerberg and Lecun be heading toward a dead end? Well, that's what Gary Marcus alleges in a recent article he published. But why does he think that? Well, for one, in 2014 he founded an AI company that was built around a different model of AI development called Symbol manipulation. The basic idea is that instead of giving an AI model a big pile of data and just letting it figure out the important connections, engineers code specific symbols to represent important details about the real world. Sounds good in theory, but things get a lot more complicated in practice. See, while it might be straightforward to hard code symbols for everyday objects like tables and chairs, you'll quickly run into problems as you try to scale up with deep learning. Going from 90% accuracy to 95% accuracy is might require 10 times more data, but it probably won't require 10 times the amount of engineering effort. But getting that same performance improvement using symbol manipulation might actually require 10 times the amount of human effort. And that's just unrealistic. Now, is it possible that the deep learning community is missing something though? We certainly don't have human level AI yet, and we don't even really have fully self driving cars, Even Yann Lecun admits that deep learning isn't perfect.

How is it that we, most teenagers can learn to drive a car in about 20 hours of practice, whereas even with millions of hours of simulated practice, a self driving car can't actually learn to drive itself properly. And so obviously we're missing something. You know, the immediate response you get from many people is, well, you know, humans use their background knowledge to learn faster and they're right. Now how was that background knowledge acquired? And that's the big question.

AI researchers are racing to develop new machine learning models that can include this background knowledge. 1 Once a system has an accurate world model, teaching an AI to handle a new task should be trivial. And symbol manipulation feels like it could be the perfect fit for this. There's only one problem though. Across the board, deep learning is absolutely crushing. Symbolic models Gary Marcus claims that deep learning is hitting a wall. But as recent performance shows, if deep learning is hitting a wall, that wall is getting completely demolished. Even still, both Yann Lecun and Gary Marcus might be biased in their outlook. Both have built entire careers around their respective methodologies. So we need to dig deeper in order to get a straight answer. I called up my friend Charles. He spent years working on machine learning at Google, but more importantly, he's independent now. He's building a new startup that uses machine learning and he doesn't have a boss dictating a particular direction to him. But here's the thing. When it came to start building, he stuck with a deep learning approach. Sure, that's what he's familiar with, but but the bigger reason he chose deep learning is because of how these models scale. His startup is small, so he doesn't have a lot of data yet, but as he grows, he'll be able to feed his model more and more data and instantly get better results. If he was building a symbolic model, every incremental improvement would require painstaking engineering effort. You can see how he's thinking about scale on his YouTube channel, which I'll link to below. Charles Story is interesting not just for the technical reasons, though it also highlights an important trend that brings us back to Talented Engineers are hard to retain Hiring the best engineers is clearly a cornerstone of Zuck's Metaverse strategy. We've seen how big his vision is for this thing, and it's clearly going to take years to come to fruition. Better VR headsets, new immersive applications, and complex AI systems to power it all are going to require top notch talent to develop. And the competition is heating up. One of the classic tools that big tech companies use to convince tech talented individuals to come on board has always been stock options. Joining a company like Facebook in the early days could quickly make you a millionaire if you played your cards right. But with the stock price falling, these options aren't as attractive as they used to be. Now Meta has plenty of cash, so it shouldn't be a problem, right? Well, it's a little more complex than that. See, when high growth tech companies report their financials to Wall street, they typically break out stock based compensation as a separate line line item. And for better or worse, investors have historically focused on operating income figures that exclude the cost of stock options. This can have a significant impact on the true financial performance of a company. Take Shopify for example. When you exclude stock options, they had a margin of 9%. Not bad. But including those stock options drops their margin to just 1%. If investors expect Zuckerberg to start paying employees more in cash, it could hurt how they view Meta's prospects. Ultimately though, compensation is just one piece of the puzzle. Talented machine learning engineers want to go work on hard problems with smart people, and that's a big part of why Meta is hosting these AI focused events. But whether you go work for a big tech company or you build your own startup, you'll eventually need to decide what to do with your money. And that's where channel sponsor Masterworks I.O. comes in. See, it might be decades until we understand how the price of art moves in the metaverse, but we have plenty of data about art in the real world now. Contemporary art prices outpaced the S&P 500 total return by 164% from 1995 to 2021. The problem is that typically you had to be incredibly wealthy to build a real portfolio of fine art. Masterworks solves this problem and allows you to invest in world famous artworks from Banksy, Monet and other iconic artists for just a fraction of what these pieces typically cost. All you need to do is create an account on their website, right, browse their artwork and then start diversifying your portfolio. With one of the most stable assets around, Masterworks is unlocking a new market. If you like a particular artist, you don't have to spend millions of dollars to buy a full piece. You can trade shares with other members just like you would sell a stock on a trading app. I'll leave a link in the description to skip the waitlist. And if you'd like to know more about the history of VR, just go watch this video about the founder of Oculus Palmer. Luckey. Thanks for watching.