Dr. Zsolnai walks through how NVIDIA is turning AI research into products: hybrid AI splits lightweight tasks locally while heavier models run in the cloud, generative characters synthesize speech and lip-sync in real time, RTX Remix upgrades old games with AI visuals, and robots train in simulations using domain randomization before moving to the real world.
Today we are going to see how AI research is changing the landscape of gaming, video conferencing, robotics and so much more. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Now first here we talk about papers. This is not a paper. So you will see results that we cannot evaluate as thoroughly as we normally do with peer reviewed research. However, I still think it is important to talk about because even though this is not a paper, but it is the papers coming to life in the world around us. You see, these technologies appeared or will appear in products that millions of fellow scholars are going to use all around the world. So I would like to show them to you and I will note that we have no business ties with Nvidia. Now here they talked about hybrid AI where your local graphics card will do one part of the computation and a different part will be streamed from the cloud. So why? Well, for instance, noise removal for your video conferencing should run locally on your machine as it requires low latency. It cannot wait for the cloud to do all this. It would take too long. There is not enough time for that. But Photoshop's generative fill can take a little longer in the cloud to ensure it runs on a smarter AI model and give you your text to image results here. Now these are the local and cloud models. But what is this hybrid thing? Well, consider their previous work on creating virtual characters with generative AI. Earlier they showcased a system which can automatically come up with dialogue for a virtual character in a video game. First, the text that the character needs to say will be synthesized by an AI. Then a text to voice AI will make it actually say it in a human voice and the mouth movements will also be synthesized. Look.
Hey Jen, how are you? Unfortunately not so good. How come? I am worried about the crime around here. It's gotten bad lately.
It sounded a little stiff, could have been a bit more natural. But this is only the first version of something that was previously nearly impossible. So here's the breakdown. When you talk to a virtual character, your local voice input is transcribed locally. This then goes into a large language model in the cloud. And when we get the text response from the game character, speech is synthesized in the cloud and then it comes back to your machine where the lip syncing is done quickly and locally. And all this happens without you noticing all this wizardry that is quite impressive. And this is available for developers to use right now. We can also use generative AI to create these virtual characters themselves too. Like the video game characters from this earlier video, they can all be given an identity and a backstory. And like good little actors, they will pretend to be this person. They can even become a fellow scholar. Now at this point my reaction was that I'd love to see that in action in this framework. And luckily that's exactly what happened.
Nova, you ever get that uncanny feeling we're like those AI constructs from the old sci fi flicks? But nah, we're too quirky for that now.
These characters are much more true to life and it gets better. Every time you enter the room. The conversation will be different, all synthesized on the fly if the occasion calls for it. Words are connected to actions as well and you might get a drink for free. Also, more games will come with RTX technology with AI based ray tracing, super sampling, anti aliasing, you name it, all in the name of better visual quality for virtual worlds. These look fantastic. And during March ray tracing is also coming to Diablo 4 and RTX Remix will help you remaster previous games and add ray tracing effects to it and and upscale old and blocky textures. If you ask me, anything that can make the portal game better is good news in my book. And also hold on to your crowbars fellow scholars, and check out this glorious Half Life 2 remix their new text to image AI is also out. This can perform not only the usual image from text prompts, but we can also add these masks to apply the prompt to to only a subset of the image, extend the image and more. Note that these tools also exist elsewhere and that this is a paid service and before you pay, free and open source techniques for instance, stable diffusion also exist. And if you watch Twitch where you can watch a stream of someone playing games, the quality of these streams can be enhanced with RTX cards. Oh yes, cards. A new series of graphics cards, the Super Series, was also announced. I love these animations in which they introduced them. This one looks like it is coming straight from outer space. This has more Cuda cores and more memory for faster generative AI. New RTX laptops are also coming if you can always look up independent benchmarks for the particular application you intend to use them for, because before buying now we already know that AI is infusing these robots in more and more warehouses and factories, but now they are also permeating agriculture, construction, healthcare and retail as well. Due to GPT these robots now understand our commands significantly better. And don't forget a virtual version of these factories can be modeled by a generative AI with which is kinda mind blowing. And now these robots can train in these virtual environments to make sure they are safe before being deployed in the real world. And in a simulation we can also do tons and tons of domain randomization, apply changes to these virtual worlds to make sure these AIs are safe and work properly in any kind of situation. This used to take professional artists and days and days of work. Now it runs in a matter of minutes and just text prompts are needed. And now we can tell them to shake hands, lie down and nod their heads. This will lead to smarter vacuum cleaners and lawn mowing robots and so much more. What a time to be alive. When I was a PhD student my experiments created lots of data for my deep learning projects. But I was not looking for more data, I was looking for insights. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
