Jesse Damiani surveys the VR and AR ecosystem, covering web frameworks like A-Frame and React VR, game engines Unity and Unreal, and specific uses: surgical guidance, pain management through mechanotherapy, art tools like Tilt Brush, and volumetric video. He also discusses haptics, eye tracking, spatial audio, and blockchain for virtual world identity.
Chapters
There's one startup, actually, in Boost right now who are making guided walkthrough VR videos of surgeries, which is kind of hilarious, because we'll look up and see some crazy surgery on their screen happening. But they're trying to develop this massive library so that other people can train using those. But yeah, we'll talk about that later, because it's a really interesting subject. So what kind of coding do you use for language for VR? So that's a longer conversation about WebVR, or WebXR, WebAR, depending. But so that's an outgrowth of JavaScript. And then Facebook has React VR, which is an outgrowth of React. And Firefox has A-Frame. So a lot of web development tools you're used to are sort of growing into 3D web interfaces that start to come together. Wow. But to build experiences, you would use something like a game engine like Unity or Unreal. And you would be, like, in VR building that thing? Typically, no. You'd be on a computer. Because, you know, right now, building in VR just takes a lot of effort. And, you know, some people are doing that, and it's interesting, and we'll see where that goes. That's a big interest of mine, obviously, because I'm really interested in how people bring stories out of their heads into the virtual space. Right on. Cool. So jumping to the next slide. I'm terrible at this. So this is sort of like capping and giving a visual sort of representation of some of the stuff we just talked about. So we talked about advertising. So imagining games where there's a little mini experience that instead of being like, buy this thing, it's like, hey, walk into this and maybe you'll walk away with something really interesting. And it's a cool experience in and of itself. You have art creation tools. like Medium, Tilt Brush, Masterpiece VR that let you actually paint and sculpt in 3D. And now, actually, Masterpiece just announced they're going to have 4-person simultaneous painting and sculpting. So you and 3 of your friends could go into Masterpiece VR and sculpt and paint the same project together. As to the— you were referencing chemistry. There's actually a company out of Russia called MEL Science. And they actually let you— it's not quite what you were describing, but they let you sort of assemble atoms and see when you start to add protons or neutrons or electrons, when it becomes radioactive and things like that, and go into the graphite in a pencil and zoom in and out and see the interaction of those molecules. And then you have out-of-home and location-based, big, massive VR experiences that are in warehouses or in specifically designed sets so that, say you're in the VR experience with your friends and you reach out and the wall's there. You're touching the actual wall that you're seeing in VR. Then you have things like mechanotherapy and rehabilitation. So people that have suffered some kind of trauma or are trying to heal, VR is actually being used to help them. And one thing that I think is really interesting is people that have, like, let's say, chronic pain in their left arm, there's this mirroring method where in VR, they'll just have the person take their right arm, and they'll mirror that in their vision so that their left arm is doing the exact same thing as their right arm. And it trains the passageways in your brain to sort of let go of that pain, because it's seeing the left arm do these activities without feeling the pain. So it's things like this, where it's like, we can actually hack our minds with just images on a screen, essentially. And then you have things like the Wave VR, which are putting on the equivalent of like a rave in VR. So people from all over the world are going into this rave with various artists. And that's both— that can be both live or just totally remote in VR. There could be a real-time rave going on in the world. That you could then hop into in the wave and engage with people that are there. The medical stuff we talked about, I mean, here's an example just of augmented reality sort of guiding a surgeon as they move through the process of conducting surgery. You have things that help with— somebody had brought up safety. Something like a firefighter helmet that actually can give you infrared overlays so that you can see through smoke and densely clogged visual environments. Yeah. And then you have things like esports and Vsports, where that's ranging from something like the VirZOOM, which is actually a VR experience hooked up to a bike, so you're biking, but in the VR experience, you're a cowboy chasing after the bad guys or whatever, so you're getting your exercise through a game, all the way to just traditional, what we think of as esports, but moved through VR, so you're actually kind of embodying. And then also, we haven't touched on it as much, 360 video is a really important aspect of VR as well, because it allows us to be really present in a scene that actually happened. And this technology is definitely the least interactive of any immersive technology right now, but over time, especially as VR video becomes more volumetric and we can start moving through space, like if there were a camera here and we could all come back to this moment, there's a lot of power to that. So this gets to the discussion about corollary technologies. And it's really important to sort of see this larger trend happening and understand that VR and AR are kind of like the body. We're being given these new senses to interact with a digital world that's existed for 30, 40 years. And we're being brought in in all these interesting ways. So volumetric video, obviously, which would be taking a 3D sort of scan or view of an environment. Haptics, so you have the suit, you have controllers. These right here, are actually force feedback controllers. So if I were to reach out and grab something, they hold my fingers back when I touch it so it would simulate the real object. Then you have things like eye tracking, which watch where you're watching, but you can also control an experience using your eyeballs. So you can look at things and blink and lift them up and things like that. So we can start— and that's fun in VR. But when we take that into AR, imagining when we have these glasses in the real world and we can all be manipulating and sending messages and doing all these things, with each other and with digital objects that we're sharing, we start to see that the shape of the world looks very dramatically different. And it's not just like, oh, Pokémon Go, I'm gonna throw— I mean, no hate on Pokémon Go, but it's not just this one thing, this one-off kind of, okay, we did our augmented reality thing, now on to something else. And then you have things like computer vision, which are taking all this data and understanding how to recognize objects in the world and geocoding them so that we can say, well, That's this store, and here are all the reviews other people have said, and it's just right, a little feed in your eyes. Spatial audio, so we can understand sound the way we feel in the real world, and our music can be made so that we really feel the full depth of the music. IoT, so Internet of Things devices, so the obvious example would be like Google Home or Alexa. These are smart objects that we're interacting with that are internet connected, so again, if we have a low form factor augmented reality set on our heads, It might know when we're coming in the house, and it can sense we're stressed because of our pupil dilation, so it drops the temperature a little bit because it wants to cool us down. And it's connected to our smart fridge, which knows that we have, you know, 3 things that are about to go bad, so it gives us a recipe for the food to make. And we start to sort of connect all these little pieces. And then obviously we all know about AI. Blockchain is something I'm personally interested. Less people see the sort of interconnection of VR and AR, but For me, blockchain is a nice infrastructure to help with things like identity, obviously monetization, and other sort of, of the less sexy aspects of having virtual worlds. So,
