The man who invented tomorrow’s tech — yesterday | Richard Marks x Jason Jerald

This is Impactful Realities. This podcast is for those at the front line of innovation who thrive in chaos and engineer breakthroughs. We're talking high-pressure industries, high-stakes decisions. We go deep with leaders designing AI-driven virtual and augmented realities and building cutting-edge solutions for the world's most dangerous, demanding, and high-stakes environments. If you're ready to uncover what it really takes to create new realities that defy limits and reshape industries, Then listen carefully as if the future depends on it, because it does. Here is your host, Dr. Jason Jerald.

Today we have an amazing guest, Richard Marks, who has done work for decades in the XR space. Originally, he came from MIT and Stanford where he did his PhD in robotics, but then he quickly moved on to Sony where he worked with various computer vision technologies and created technologies such as the Sony EyeToy, as well as the PlayStation Move, and later had a major contribution in PlayStation VR. He's also worked with Google for several years and is currently professor at the University of North Carolina Chapel Hill School of Data Science and Society. And Richard, I'd love to start off by you sort of explaining your very long journey And sort of what you learned through those years and how you ended up at UNC.

Sure. Yeah. So you covered it a little bit. I worked as, I studied underwater robotics at Stanford, and then I started at a startup that was doing computer vision, and I did computer vision for the underwater robot. And then I, uh, uh, the startup actually got acquired and then the company that acquired us decided they didn't want us. So they kind of laid us off and then I had to figure out a new career path. So I worked as a consultant for a little while and then I saw the PlayStation 2 announced and I thought it'd be really interesting to plug a camera into the PlayStation 2 and see what could be done with that. So I joined the PlayStation R&D group and I was there for 19 years.

Wow.

Working on a lot of different things, including, as you mentioned, PlayStation VR, PlayStation Move. So yeah, a lot of interesting things came out of that work, and it was a really great experience for me. I had a lot of freedom to explore the different things I wanted to explore. And I know maybe we won't talk as much about games in this, but one of the things that I think is really important about fields like that is you can try new things without nearly so much concern about like, there's no plane is going to crash if your video game, you know, don't try something new. So you can really be more experimental, which I think is almost the obligation of that space. So you can really explore. And then I left there for some, maybe we can get into that a little bit. I left PlayStation under really good terms, but I wanted to try something new. And I went to Google and joined the group ATAP, which is an applied research group.

Yeah.

and Google and did some things about understanding spatial context. So yeah, that's kind of the summary, but maybe we can talk.

Yeah.

The story about why I left is kind of interesting, why I left, like the thing that kind of finally pushed me over the edge to try something new. I've always like thought about going to other companies, but I really liked PlayStation so much that I didn't. But, uh, every, every, every Christmas I would get like a new kind of technology And try to learn about it. So like one year I got the Roomba and like all different kinds of vacuuming robots, try to understand how they work. And one year I got all the smart speakers and I compared them, Alexa and, you know, Google Home devices. And it's, then I really liked the Google Home one. And this was while I was at PlayStation and I bought one for my father for Christmas and he said he wanted to return it. And I asked him why, and he said it didn't work. And I said, well, why? He's like, I asked it a question and it didn't answer it. I said, well, what question did you ask? And he said, I asked, when is golf? And I'm like, well, that's a pretty vague question. And he's like, what do you mean that? I mean, but I can see from like an end user's point of view, like, like maybe he was playing golf that day and he had a tee time and he thought it should know his tee time, or maybe, you know, The Masters was on that day and he thought it should know to tell him the television time when it was going to be on. But without enough context, there's just no way for the system to know. And so it really got me thinking, the hard part about doing these kinds of things is really having enough context for the system. And in particular, the things I focus on are spatial context. There's some cues that we use with each other all the time that computers just don't see or have, or they haven't until recently. Now, Now it's a big focus, but for so long nobody really was getting that kind of information into a computer. So that's what I worked on at Google, actually.

Yeah. And I remember we first met, you may not remember this, but this was probably 20 years ago at SIGGRAPH where you were showing an early version. I think it was pre-iToy. And yeah, I get that why you loved it there because you were kind of exploring and showing really cool stuff, but maybe you were still in that exploration stage of getting that context of the world around you by tracking, you know, brightly colored, you know, sword objects and such. I think they were, if I remember, foam at the time. But oh, I should also mention that, yeah, so Richard's lab's called the Magic Lab. Like, how cool is that? You know, that does imply that he had a pretty awesome job that was probably tough to leave and gave you that ability to to create. But yeah, that also allowed you to look at computer vision in different ways that I'm sure many others weren't. I know academics have been looking at that for decades, but actually applying it versus just research paper. And then eventually that evolving into the PlayStation Move by tracking the glowing orbs and such. And I know it's integrated with gyros and such, at least that's my understanding. But yeah, shipping around 20 million units, right? So that is not just an academic thing of this is how it should work. Yeah.

So one of the things, like my focus was definitely entertainment, but that ends up actually sometimes having impact in a way that is a little bit indirect, but because we make things at such mass scales for entertainment, sometimes the cost of things goes down dramatically. And my background, as you mentioned, I was like doing computer vision in an academic setting originally for my PhD. And then when I went into industry, I wanted to make a really good camera so I could do the kinds of things like— and so we designed a camera for the PlayStation 2 and then really the PlayStation 3 camera in particular, the PlayStation Eye. We had made a really good lens. We made the video uncompressed so it wouldn't have any artifacts. We designed it so you could— we could control the Bayer pattern conversion. We could do a lot of this really low-level stuff. And it turned out that that camera got used by many, many computer vision academic labs throughout the world, actually. Often I'd go into some academic lab and see that camera being used by researchers there because it's just a really good low-cost camera. And some companies used it in their environment. When I got to Google, they had some kind of prototype, or there was a camera that I designed there in their prototype. So things like that. And the Kinect for sure has impacted people.

Yeah.

You know, the 3D camera that Microsoft created really got useful, used by a lot of other people, not just in the entertainment space.

I know PlayStation VR had shipped tons of units as well. I suspect most, many, if not all of our listeners are familiar with PlayStation VR. I know I certainly am. We had actually created a game that was just a blast specifically for PlayStation VR that we launched. And so thank you, Richard, for creating that. That was an amazing device. And my understanding is that was the most shipped device. More of those units were sold for a period of time than any other VR device.

Until the Quest. Yeah, it was for a while there. And I think, you know, we kind of like felt like with VR, we like, we wanted all ships to rise. We competed against them in some ways, but in other ways we were a console. They're Different units. So like, I don't think we were really competing, but yeah, they fenced us up though.

Friendly competition for sure. And so yeah, I'd love to hear more great stories about that.

We'll maybe next time we'll talk about some old stories about those days when we were all in the trenches getting VR off the ground.

Well, yeah, I'm sure when, you know, Meta's pumping $10 billion a year, although I know, Luke, Sony's budgets is not bad either, but as Pretty tough to compete with when you're talking about those sides of budgets. And so, but yeah, both PlayStation as well as the Meta hardware is just amazing. And at the consumer level, I mean, you talk about impact of whether you're doing research or you're creating games, a few hundred dollars for a headset, you know, people complain about that now. That is crazy low price. So back when, you know, Richard and I back in the, you know, '90s and 2000s, these headsets would be $30,000, $40,000. So. I'm sorry if you're complaining about $300, but then you have, you know, your $1,000 iPhone, you have nothing to complain about.

Actually, maybe a quick story. So we actually were at the Game Developer Conference and we unveiled the PlayStation VR. It was called Morpheus back then, and we were showing some demos, and actually Mark Zuckerberg wanted to come and see it. And so he came in after the show closed and we gave him some private demos and things. And You know, he was interested. And then a couple of weeks later, I was on a plane and I went and actually visited Joe Laviola at Central Florida, who we've taught a class together with at SIGGRAPH.

Yeah.

You and I, Jason. And so on the flight back, I'm on the plane and I'm looking at my phone and it says Facebook just bought Oculus for $2 billion. I'm like, what just happened? And so, yeah, that was a huge kind of like moment. And I realized we just showed Mark Zuckerberg our stuff, and then right after that, he bought Oculus.

I thought you, when you told me this story once, I thought you or someone in your lab, as soon as he left, one of you said, wow, I think we just sold Oculus to Facebook.

Well, we definitely like pieced that together later. I don't know if we knew that right then, but yeah.

And so what caused you to decide to go over to Google with ATAP, their Advanced Technology and Projects group?

Well, so like I mentioned, I was really interested in the idea of context and Google reached out and was interested in exploring some new things. So I, and I kind of described some of the things I'd be interested in working on and they said, sure, come do that. And one of the things was this idea of of understanding spatial context around you. And with the PlayStation Move, we had done a lot of stuff. I have one here. So, and it can— I'd made this like internal kind of like hackathon demo where we use the PlayStation Move. We could like point it at like IoT devices and turn them on and off using the PlayStation Move because it knows where it's pointing, where it is. And then it can push a button and send a signal to—

Yeah.

something. And so I thought this would be kind of a cool thing. And when I got to Google, I started working on the idea of having this kind of pointing context added to what you're already doing when you talk to a computer. So we have this voice input that was pretty mature, but we wanted— I wanted to add this ability to point and say, like, ask something about that particular thing, or, you know, group these things together and do something. So I started working on that and we basically, we took a, actually we took a Vive controller, VR controller, and a phone and like duct taped them together. And so the phone had a camera and could look out at the world. It had a microphone so you could talk and do all the kind of Google stuff that we could do already. And then the extra piece was this spatial input that was coming from the Vive controller initially. Eventually we moved on and took all the kind of information from the phone itself and did some—

Yeah.

kind of XR or some like AR kind of, I guess, ARKit. No, ARKit is Apple. ARCore is what it's called at Google. So the idea, we did it all inside of us, kind of like the phone. And that whole idea was like we'd make a smart remote for your home where you could point it at things in your home and it would know what you're pointing at and it would control them.

So it sounds like there's a lot of opportunity to explore, but also thinking not just explore for that sake, but how it might be used in the real world, which certainly would have impact.

Yeah, it was definitely an applied research group. And the particular like kind of producty thing I was thinking about was like a smart remote control. But the whole idea of having just spatial context in general, I think was interesting, especially in the future when we expect maybe there will be smart glasses that can see what's happening around you. And so if I point or if I'm looking and turn my head and look at something, it'll know now that that's the context it should use when it's answering a question. And we're seeing a lot of that happen in the AI space now with the kind of multimodal AI that's happening.

Yeah, it seems context is sort of everything, especially when it comes to augmented reality. Virtual reality as well, but not as much as augmented reality. It's, you know, you're not just in this void.

Yeah, the great thing about virtual reality is you have all the context really, right? You're simulating the whole environment. You have the trackings. capability. So you have a lot of the, you know, like pretty much all the context. So you can experiment, like when you have this context, what new things can you do? And you can do a lot of that work in VR, and then you can take that work and try to carry it over into like AR and the real-world space.

And so, yeah, that I think transitioned nicely into what I wanted to talk to you about later, but let's talk about it now, is I know a lot of us have been very frustrated without being able to get access to the camera feeds and the consumer head-mounted displays. In the past, we were able to do that, right? Because they're sort of, they're, you know, at the industrial level that you can do whatever you want. But to me, that is like perhaps the most important part of augmented reality is to be able to, the computer to know what the human is seeing, right? And to be able to analyze that data with, you know, computer vision and AI to get that context around you. And I am so relieved because we've been wanting this so long, is these consumer devices, my understanding, is starting to open up and Meta is just starting to do that with their headsets so we can get access to that video feed so we can better analyze the context around the user and analyze it in real time of what are you looking at and such. Could you maybe discuss, especially with your background in computer vision, of what that means as far as impact with these headsets of where, how just opening that up just changes things so much, how that's a game changer.

Sure. I really miss the days of being inside the company because when you're inside the company, you have access to everything and having that live video and anything you wanted really. You could add whatever sensors you wanted and tie them in and have, you know, the hardware engineering group was right down the hall. You know, that helps a lot. So I miss that those days. And it has been frustrating. A lot of my students have been interested in like doing some of the sensing kind of work and analyzing using video to do stuff. We have to just add an external camera of our own, and then we have synchronization and calibration issues and all those extra issues that— so it is a challenge, I think, for those companies to give that data out. And really, they can make their own products without it, but for third parties and researchers, they're kind of left in the cold. So it is great to hear that's happening. Same thing with eye tracking data. I'm really into eye tracking, and eye tracking Nicely on the Quest, you can get access to some eye-tracking data, but on a lot of the other platforms, you don't have so much access to the eye-tracking information. And it makes it a lot harder as a third party or a researcher to do the kinds of things you're interested in.

Could you give some examples of what you'll want to do with these, or maybe what you've done when you did have access to those things?

Yeah. So, I mean, so there is, there is a lot of kind of special effect things you can do where you blend the video and the graphics together in interesting ways and make things. And if you don't have access to the raw video, you can only do these limited amount of graphics. Also understanding what's around you. Right now we're kind of at the mercy of whatever the platform provides for understanding. And if you want to do anything new, you don't really have access to that. So in the future, I definitely think that that pathway of the video into AI and having the video and the audio go into a multimodal kind of generative AI analysis can give a lot of capabilities. And so I do think these kind of spatial contexts are going to come. There's a lot of work in spatial intelligence right now in generative AI. There's companies just formed recently and Google's been talking a lot about that as well. understanding both graphical, like real-world space, but also photographs and other things that you can feed in. Yeah.

And how does AI fit with XR, right? Lots of different ways it can fit, but especially when we start getting access to that actual computer vision of what the user is seeing, then there's just so much. I mean, AI is going to be so useful for that.

Yeah, I think somebody recently was trying on the Google new XR glasses and they thought that the killer app for XR was AI because it like could see the world around them and they could ask it questions and it would know what they're talking about and those kinds of things. So tying them together, I think, is a really strong feature. Actually, there's a conference that I was just on the committee for called AI, sorry, sorry, VR x AI. So kind of VR crossed with AI. All the letters.

Yeah, that's hilarious.

But yeah, that combination is powerful.

And so yeah, as far as spatial computing goes, how do you think about or define spatial computing? Because I know you're going to be teaching a course on that coming up here real soon. So Yeah, what are your thoughts of what spatial computing means?

Yeah, so the first semester I got to UNC, I taught a class called Introduction to VR and 3D Graphics, but I'm going to change that class up a little bit and call it, I think I'm going to call it XR and Spatial Computing. And so we will learn a little bit about graphics, but a lot more about the input side of things and understanding the world around you and the And being able to combine the context of the real world with graphical rendering. So spatial computing to me is really like, already we have to do it for just to even implement a VR headset. Like it has to know the motion of your head. It has to know how to render things properly. So that part I already taught in the VR class, but then combining that in like mixed reality way where the geometry of the world and the rendering of the scene together. I think there's lots of cool things that people are starting to do. And one of my personal things, a little bit my research area, is thinking about having a spatial intelligence, like some kind of AI, but have it embodied in the real world so it feels like it's physically present. And like you can see, Maybe you heard this analogy before, like one of the problems with self-driving cars is that there's no driver there for you to get any feedback from. So you look and you don't know if it sees you or not, the car, like you don't know if it's going to run you over because there's no eye contact happening. And so same thing with AI, we don't really know what the AI is ever thinking until it like spits out something. So having some kind of constant presence there, it's there and you can see kind of what, like what it's pointed at, kind of Gives you some cues back. I'm really interested in that in a research space.

That's super interesting. Maybe the most compelling use case for AI-generated characters or virtual characters will be driving, so you can have that nonverbal communication. I guess in some cases, maybe you'll start yelling at each other. There might even be verbal communication.

People have talked about putting like a screen with eyes so it shows like the part.

Yeah, yeah, yeah, for sure. Kind of like the Vision Pro.

Yeah.

So yes, since we're on that topic of what you'll be teaching and such, we haven't talked about sort of what the School of Data Science and Society is about at UNC. And so I guess a couple parts of that. One is why the transition, because that sounds like a pretty amazing place to be at Google in that group, but why you transitioned to academics and what sort of the mission and the focus of the Data Science and Society School at UNC is about?

So both of my parents were teachers and I had always intended to go and become a teacher someday. When I was about, I don't know, 16 years old, I made this life plan of like, I'll be in school 25 years, then I'll go into industry 25 years, then I'll go teach for 25 years. And I'm kind of in that third phase now. And I I'm a little late. I didn't quite get there when I was expecting to, but maybe partly I waited until COVID was over to start teaching. That didn't seem like an ideal time to start a teaching career. But yeah, so I knew a lot of the people in the computer science department at UNC. I had worked with some people who had graduated from UNC. I knew you. I knew Henrik Fuchs, who was a professor here, and a lot of people from UNC have a presence at SIGGRAPH, which is a conference I'd gone to so many times. So they're very friendly and engaging. So I just was the top school for me to consider. And so I reached out to the computer science department right away and they told me that, you know, there's also this new school of data science forming. And since I was coming from Google, they really thought I should also talk to the school of data science. The school of data science had a lot of, uh, it was just, it's like more kind of like a startup environment. It's like I said, a new school, so kind of getting to help define what it is. And so I still work a lot with the people in computer science, but I found I really was drawn to this idea of the society part of the School of Data Science and Society. And so not just doing these things, but thinking about the implications of them. So it's not just the tech, but also how it— and so I'm teaching a class next year in the spring of next year on systems. And systems have a lot of technical information, but systems are also very much about how they matter to society. So you can have computer systems, but you can also have political systems and economic systems and physical systems, biological systems. So there's a lot of, all this same kind of technology and understanding can be transitioned to other Yeah, we can learn those lessons from a complete different field.

And I know, yeah, the computer science department, that is a big focus there, is working with other departments, whether it's, you know, medical or the School of Data Science and Society. And yeah, I remember seeing you at that UNC alumni event, and hopefully I didn't offend you. I think it kind of came out wrong because I was like, what? I was like, wow, like I was not expecting you to be there. So I was like, what are you doing here? So I hope that didn't come across. I didn't offend you of like— You don't belong here. That was pretty awesome to see you there.

Yeah. Yeah, there's so many people that actually are tied to UNC in this field, so I think it's a good place to be.

Yeah, for sure. And especially now that you're there.

I got to meet Professor Brooks early on when I visited a few times, and I actually put the PlayStation VR on his head and we gave him a demo of the PlayStation VR. So I think You know, he's one of the people that made VR even happen at all. So it's pretty cool. Yeah.

And he's so friendly. Although being, yeah, when he's redlining your drafts on your papers and publications, he's very direct and yeah, he will definitely question you, but yeah, super friendly. So I'm curious as far as being data science, what types of data you look at that relates to society at large, right? Or sort of maybe what are some examples you or your colleagues are looking at that relates to society versus something narrow and niche?

So the School of Data Science is really like a hub for data science which is being done in all the departments and across the whole campus already, especially in the medical field. And so we're physically co-located close to the medical groups, close to the medical school, close to the school of nursing. School of nursing's using actually VR and X for a lot of their training and for the undergraduate program. But anyway, yeah, so the data science, we have people who actually have a joint appointments in the School of Data Science and then another department as well. And so they can take a lot of the data science work and transition it into practice in some other field. And like I mentioned, medicine and nursing already, but also in political science and sociology, journalism, all these fields, we have professors that are kind of joint between these different departments in our—

Yeah.

And then of course, computer science, statistics, mathematics, those are also, we have connections with those too. So I think that the society part, there's different ways you can think about society and we're still, you know, we're new, we're still defining ourselves. And so it's very easy to think about society from like a political science or sociology or medicine kind of standpoint. But even in the framework of AI, you can think about like, How is AI going to impact society? That's a question that a lot of people are wrestling with, and it's a perfect thing for, I think, the School of Data Science and Society to think about as well. So the computer science people that are in our school can think about that kind of impact on society.

Yeah, and we certainly— one thing I know we previously discussed was creating visualizations to make your YouTube videos or whatever more compelling. We haven't done too much of that here on the podcast with us just getting started, but I'm sure we'll be adding some of those things. But I guess you also have to be careful of, you know, you have something so cool in the background, you stop listening to, you know, the intent of the podcast. And so maybe there's a balance of that. But I know you had mentioned that you'd seen some pretty cool sort of engaging visualizations for educational content on YouTube and such. And so what are your thoughts on that as sort of what's already been done, is being done out there and where that might further be embedded within XR?

Yeah, I think this whole space of how we interact with AI systems to generate content is a really interesting one, the user interface for it. When I was in Google, there was a group that was creating 3D models using generative AI. So you'd kind of say like, I want, let's say like a, a tree that looks like a wolf, and it would generate like 16 things, and then you'd pick which one you like the best, and then it would generate 16 more. Or maybe you'd pick the 3 you like the best and it'd do some variations or combinations. And so it's like constantly generating things and you're selecting and kind of steering it. And I think that's an interesting UI model there. And that was primarily to generate content for like video games is what they were doing, the 3D models for those. But for the YouTube thing you mentioned, like some of my favorite YouTube videos, they actually use some software that they've made to create this. And then actually generative AI can author in the same software that they've done that. So now it's very much like you can use the AI to help generate this content instead of creating it from scratch all the time. So yeah, the visualizations of just for education are getting so much better every day, I feel like.

Yeah. Yeah, it's only getting better, even though AI is certainly not perfect in all ways. Yeah, it's not going to get worse. Although I guess you could argue maybe in some ways, right, we do need to be careful because if it's more clickbait or whatever it might be, but in general, the technology obviously is becoming better and better.

Yeah. And a very recent thing that people talk a lot about is called vibe coding, which is like basically you're programming and it's like pair programming with an AI. It's constantly helping and finishing things and doing things. And I've been doing some programming recently in Unity in particular. And it's like, it really knows the Unity framework quite well and it can just extend. Like I write very little code anymore and it's just constantly filling in the code for me. And sometimes it's wrong, but I know what I want because I've done a lot of coding in the past. So I have a really good kind of relationship with the AI to fill in the blanks for me.

Yeah, you guys are good friends. So I actually had an interesting conversation this morning that a lot of the research publications, and it sort of makes sense from the academic side, right? Of, you know, if you're writing your essay to be graded, you probably, you know, not fair if ChatGPT writes it. But also, there's just so much power there, right? If you're learning how to use AI with your own knowledge and with your own expertise. And my concern of academics, if they're, you know, downplaying that and there's the bias against it specifically in publications, you're not allowed to use that. And if you do, you need to disclose it. And then some, you know, reviewers might be, you know, biased against that. My understanding, there is a lot of bias against that in the research academic community. And so my concern there is that's going to put the academic community further behind. Versus in industry, you do whatever's the, you know, best way of getting the job done. Who cares if, you know, you're smart or not, or how much effort you put into it? So I'm curious on your side, when you're teaching these courses, you know, what is that balance? Because you don't want, you know, just ChatGPT, the student to, you know, do everything for the student, but you also want to encourage them to use these tools. And so what are your thoughts of balancing all that out in an academic setting?

Yeah, it's a, it's a challenge and it's evolving so fast that it's, it's actually quite difficult to have like an answer that will still be the right same answer 6 months from now. So I think that one of the things that the academic community, like you mentioned, the goal is to make sure that everything that's being written is being written with intention, not just generated and no one has read it. So you don't want it to be that it's saying something that isn't what you intend to say, but sometimes that's not implemented as well as it could be. And then we're restricting, like you said, maybe too much we're restricting, but—

Absolutely.

For students, it's definitely a challenge because you have this situation where it already knows. It's kind of like having maybe like a really overly helpful parent nearby. If the parent does all the work for you, you don't really learn, but it's really great that the parent's there and can help you when you need help. We just need to make sure the balance of how it helps is the right way, not just doing it. Like if you just said, here's my homework, mom, please do it. I'll get it tomorrow morning from you. That's not what we want.

Right.

But to say you shouldn't have access to a really involved parent is not what we want either. So it's something that, that's my analogy I have, I guess.

I love that analogy. Yeah. Wouldn't it be nice if we could say, hey, mom, dad, you know, my dad's a professor of computer science. Can you do this for me? But yeah, you're not necessarily learning. Yeah.

And I did have 2 teachers as parents, right? So I actually had this exact situation all the time. I had people who knew the answers all the time nearby, but they didn't just tell me the answers.

Oh, I was, yeah, you got my, uh, I would've been inspired if, uh, they'd done all the work for you. Then you can focus on the important things, right?

What, so what, I mean, I think a common strategy is they'd ask questions. They would ask me questions back, right? I'd ask them a question and they'd ask a question back. And by them asking me questions, I figure out. And so maybe that's a strategy we can use.

Yeah, man. I don't know if there's any AI tools that behave in that way. Maybe that would be a really good AI education.

I think that's the strategy some of the AI tutors are starting to take, to adopt is that like, but sometimes like you can game those things too. And I used to do this to my parents, I'm sure, is like you can figure out how to ask them a question and then when they ask you a question back, they give you enough information in their question to Yeah.

But you still have to think about it and use your brain and actually create those strategies that could be very useful.

Yeah. The other thing I think is a challenge is that maybe a pretty famous thing is, I think it's Malcolm Gladwell had the 10,000 hours is what it takes to be master of something. And so that's a lot of time that it really takes to become a master at something. And I've definitely spent 10,000 hours programming. in my life. So now when I use AI to like vibe code, I feel like, you know, I have that 10,000 hours behind me, but somebody learning now doesn't have 10,000 hours invested. And so it's a different equation, different balance, and I'm trying to understand how that's going to evolve in time is again, maybe a societal impact we need to understand.

Yeah. How can we make learning more efficient and yeah, not just get the answer. And so, Yeah, that's something I kind of talk about in my talks, right, is failure. And that's a great thing about computer science is you can fail, you know, thousands. I mean, you can fail hundreds of times in a day just by, you know, how many errors you get from your compiler, right? But maybe there's a way to even accelerate that, those failures and learnings even more quickly with AI being able to give even more useful feedback more efficiently.

Yeah.

Okay, so I also wanted to discuss, I know one of your interests very much is digital twins, and this isn't necessarily, you know, a brand new topic, even though maybe what we call them is brand new. But I know, you know, Disney and other companies have been using those, you know, digital twins to evaluate their rides and such. And yeah, maybe your thoughts on those, sort of the history of digital twins and how that's evolving and how we might use those things nowadays.

Yeah, going back to my background and my PhD work was like robotics. And so a lot of times in robotics, you don't want to necessarily have to hook up the real robot and do everything. So you do stuff in simulation. And so sometimes you do stuff in simulation, but then you hook up the real sensors. So digital twins is very much this kind of idea where you have a simulation, but you have real data coming in from sensors and you can analyze it. And maybe with a digital twin, you have a model and you're Checking if the data is consistent with the model you have, or you're seeing if there's any discrepancies, or like maybe the model is just previous data. You don't necessarily have a physical model, but you just have a lot of previous data and you're seeing if there's anomalies happening. And then visualizing all that in a way that is useful to people instead of just a bunch of numbers on the screen. So actually XR is a really interesting way to visualize some of this data. So you can imagine walking up to something and looking at the factory floor and seeing the data visualized next to the thing that actually you care about. That's kind of like that whole space I think is interesting combining again, like graphics and kind of understanding how to render things in a right space with live data coming in from sensors.

Yeah. Integration of not just the pre-created model, but what's actually happening in real time that's modifying and updating that model. In my mind, what's really fascinating is that whole XR continuum of maybe you use that twin to train with virtual reality, right? Then you could train with augmented reality on the real device. And then maybe there's sort of on-demand learning as well, that there's nothing needed if you know what you're doing, but then you have some questions, you can easily bring up the manuals or pointers of— what to troubleshoot further. And all of those rely on the digital twin sort of integration.

Yeah, I mentioned when I was at Google, we took the Vive controller and taped on a phone to it. But initially, we did just in VR, we took the Vive controller and used it. And so, you put on the VR headset, but it looked like you were holding a remote control in VR. And then, we simply We'd experiment with what kind of experience we wanted in VR, and then we'd transition that back into the real-world example. So we could use it as like kind of a prototyping framework, VR, but also then we can, you know, it's the same data we were getting from the tracking. So there's like kind of a blend of what is real and what is not real as it starts to cross over.

Yeah. And you can bring in some data that's real and some that's simulated, and you can kind of mix them together in interesting Yeah, that prototyping of, you know, I don't think a lot of people think about that, but VR is just such a great prototyping and research tool that, yeah, it can take time, you know, create these environments and such, but not that much time now with, you know, Unity that you can get something up and going pretty quickly. But it's just so much more efficient to prototype in VR before you build those physical mockups. And then you can also collect data on it. Now, I don't think it's, you know, going to completely replace everything, but certainly a big part of that pipeline when you're creating products and such.

Yeah, and I'm interested in making that prototyping experience even faster. So the authoring for 3D content still, at least until recently, has been very much like on a 2D screen and you're doing stuff, but in VR, you actually could author potentially much better. I think Horizon from Meta is starting to be more in that space where you're really authoring things while you're in VR, but that's pretty new and recent. And then if you imagine combining that with AI, you have this spatial input, you're doing stuff, and you say, I want this over there, and it creates it, the 3D content dynamically right away for you. I think you could make a really fast, rapid prototype kind of situation.

Yeah, yeah.

You know, I just— the short story is I just ordered a couch and it was much smaller than I expected, and I should have looked at the dimensions, right? I mean, it was my fault, right? And I was actually using some apps to look at it in 2D, but then I kind of gave up because, you know, it wasn't the quality I was looking for. But man, if I would have put it in AR, right, I wouldn't have made that mistake of, now it's too small, but it's not worth returning the couch sort of thing, so I'm going to have to live with it. So, you know, Stupid simple example of how that would be useful.

Yeah, we had an arbor built in our backyard in our old house, and when they were telling us what it would be like, I was trying to visualize it, and my wife's really good at visualizing stuff, and I'm not so good actually at it. So I was just thinking, man, if I could just put on an XR headset and stand out there and see the arbor right there integrated, that should not be hard. But there aren't— that hasn't made its transition into those kind of products yet in those spaces.

So one of the most sort of, I guess, impactful in some ways application of virtual reality was at the UNC Computer Science, the Sitterson Hall. So the older computer science building, that lobby, they put it in VR. This was back in like the 1980s, right? So it was, you know, pretty primitive VR, but it was enough to understand the spatial relationships of that lobby. And they realized only by seeing it in VR because it looked fine on the architectural plans, but that lobby was way too narrow when you have the, you know, looking downstairs, then, you know, the main reception desk. And so they actually changed the plans to widen that by, you know, 5 or so feet. And now when people are walking by, you know, you're not hitting shoulders with someone walking the other way. And so, yeah, as far back as the 1980s, it's even with something primitive can give a lot of insight.

That's a good story.

Okay. So yeah, lastly, what are your thoughts on the future? And I guess we talked about this a little bit, but expanding upon that because that's pretty important. I mean, I'm a futurist and the future is what really gets me excited and how it's going to impact the world we live in. Even if the end result, like these prototyping things, for example, the end result might have nothing to do nothing to do with these XR technologies. It was just a tool to get to that point that changes the real world. So what are your thoughts on not just XR, but the integration of AI with XR and what might that ultimate impact be?

Yeah, I think that all these things that we have done in XR, they really— well, maybe I'll go give you a little another story from early days when I was at Sony. We were doing stuff where we had the camera and we're trying to track people's bodies. And games have like 3D engines they're doing stuff with. And we started working with Nike and Nike was interested in people moving their body in new ways and they're interested in the idea of digital sport. Again, this is like a long time, 2002. So long, long time ago. And One of the things that they realized is that they could actually simulate a lot of their clothing things in like a simulation environment. And they've never, had never done that before. And so they, we kind of introduced them to game engines and game engines are really powerful simulators really. And because they do have this kind of physics and spatial understanding. And so same thing is being done for like self-driving cars. They train them in simulation and the simulation. is basically a game engine that's been transitioned into something else. And so NVIDIA is very much like in both of those worlds, right? So NVIDIA is actually in all these worlds, AI and then 3D graphics. And that came from the games world mostly is where NVIDIA's bread and butter was for a long time. So all this stuff doesn't seem like it's directly related, but it really is and connects in a lot of ways. So I think that People sometimes segregate out like this kind of graphics as being not relevant to their industry, but I think that's shortsighted. It really is just a simulation tool in a lot of ways.

Yeah, it's very much all coming together versus being this disparate set of tools or set of ways of thinking.

And I think right now we've seen this huge stride of breakthrough with generative AI in text input and text. But that's funny because that's like the narrowest bandwidth kind of thing of like communication. And so like, again, we are seeing it expand now, but in the early days of video games, it was like text adventures is what you had and that's it. And then it became more and more graphical and more and more 3D and things. And I think we're seeing the same thing starting to happen with AI is like, We had only this kind of text capability and language, but now we're expanding into audio and you can talk to it and then expand into video and it can see and things. And so having this kind of natural understanding of how these things work in spatial ways, I think people who work on that, I think have a good career path for a few years.

Absolutely. As we mentioned earlier in the podcast, Context is everything, whether the, you know, your father's golf example or, you know, just reading body language, right? Or we talked about on a previous podcast of getting EEG, you know, information, right, of the human body and such. And it's like things become so much more interesting and applicable because, yeah, if you only have that single modality of, you know, the transcript, then it can be misinterpreted.

And one thing we haven't talked about very much, which comes up a lot when you talk about VR, is presence. And the idea of presence to me also is, I guess I mentioned it a little bit, having an embodied AI makes it feel present with you. Like, I don't really feel like ChatGPT is present. Like, it's kind of like this thing that I send something off to and something comes back and it doesn't really feel like the same kind of presence. And even like a dog, which has very little intelligence feels very present. When there's a loud noise, like the dog reacts and ChatGPT doesn't react to anything like that, right? So like those kinds of things, I really think if you want to have an intelligence that you interact with in a more deep way, it needs to have some kind of feeling of being present with you.

So I know you're not focused on robotics anymore, but to me, that is where the next big change is going to be, right? Of mixing AI with the physical world more. So more robotics and having these intelligence out there in the world. And, you know, people are working towards this, but there's so much more out there of, you know, maybe these augmented reality becomes more physical realities of these, you know, physical intelligent matter moving throughout Yeah, I think robotics, definitely robotics has a long runway.

I'm actually interested in a shorter-term view of that. So like, again, like a quick way to simulate robotics is with like VR and XR. You can have basically a robot there being rendered, but not actually have to build that whole thing and figure out—

Yeah, much easier.

And also like, and then sometimes I don't want to wear the headset. So I've also been interested in doing things on like a a holographic display or something like something from the Looking Glass company where it just sits there on my desktop, but it feels like it's physically there. So like if there was a rendered AI, it could turn its head and look and I would see kind of its spatial representation that would make sense to me. So that's another space that I think is exciting and interesting.

Yeah, real-world presence. Okay, well, we've discussed a lot of interesting topics and I know we could go on for hours, maybe days nonstop. But I also want to respect your time, Richard, and I'm positive our audience is going to appreciate this as well.

Great. Well, thank you for inviting me. I haven't gotten to see you as much as I hoped to see you when I moved to North Carolina, but—

Yeah, we're almost neighbors now, so we'll have to get together again sometime.

Yeah. And it was great of you to come and teach a lecture in my class last year.

That Oh, absolutely. I'd love to again at some point.

Yeah, definitely. And actually this coming semester, maybe.

Okay. All right. Okay. We will call that a wrap.

Thanks.

The Impactful Reality Podcast requires more than a conversation. It takes action, research, and collective wisdom. If it made you think, we'd love to hear your insights in the comments or contact us directly to shape the future together. Don't just subscribe. Pass the torch. Put it forward. Send it to the next one in your circle who manifests things that matter. The builder who dreams big and innovates towards more impactful realities.