Scott Erickson on Microsoft HoloLens | #ProductSF

At Product SF, Scott Erickson takes us behind the scenes with Microsoft HoloLens. He demos the many applications that their untethered mixed reality technology can have for different industries. This presentation was recorded at Product sf, an event hosted by Greylock Partners that brings together founders, PMs and product leaders to talk about the challenges of building new innovative products that change lives and create habits. For more podcasts, please visit news.greylock.com I'm.

Excited to welcome Scott Erickson up to the stage. He's the general manager of HoloLens at Microsoft, which is what they call the world's first fully untethered holographic computer. Scott has over 15 years of consumer brand marketing and design experience. He's been at Microsoft for a long time, building products like Xbox and Zune. Let's welcome Scott Erickson.

It's always great to start with being introduced, having made Zune. That's always a good one, especially when you're in the Valley. It's great to see everyone. Thank you very much for having us here today. And I see some familiar faces, some folks who've worked on the project as well. We do have demos running up until 7 o', clock, so if you haven't had a chance to see HoloLens yet, my colleagues upstairs will happily take you through it. The team that I work on at Microsoft is responsible for creating new products. When I say new products, I don't mean bigger screen laptops or faster processors or better pixels or anything like that. We're charged with finding the future of computing and figuring out how do we evolve personal computing in ways that we've never even been able to do before. When we do this, we look at how are you going to do it? Technology really exists to make us all better, to make us communicate better, to be entertained differently, to be educated in different ways. But to get a true advancement in technology, it's not just about better battery life or bigger screens. Those are important steps. But it's really about changing the paradigm and changing the way in which you think about personal computing. So how do you do this when you think about things that might be science fiction on one hand and then you want to turn it into reality? Well, I think about the phone. This is something I think a lot about. We kind of know generally. We all went to fourth grade, we kind of learned about Alexander Graham Bell, and there was something in the mid-1800s where he and his buddy Watson created this thing that we now know as the telephone. But obviously that wasn't the beginning of the phone. Something had to have happened because something obviously has happened after. And I thought, well, what happened? How did he get to that point where he was able to create the telephone? About 200 years before, almost exactly 200 years, there was a guy called Robert Hooke. He was an engineer and a scientist out of the uk and he created something that looks very similar to what's on the screen here. It was dubbed by the common people as the lover's telephone or the lover's phone, which was basically a string with a couple tin cans on it. And that's how, hopefully at greater distances than these two people that think they need to use this technology and would communicate. So for 200 years, this evolved feature improvements kept coming. Different materials were used for the strings and cups changed. And at one point there was a big product release where there was a six mile long string. This was revolutionary. Today we stand in line at the Apple store to get our new phones and you can imagine going to the product launch for a six mile long string. Well, from 1860s, a lot of things happened very quickly. New metals were introduced, you know, improvements with electrical circuits. We started having operators, we started having transcontinental and transatlantic calls. And then something happened in the mid-1970s with a mobile phone. 1973 was the first consumer mobile phone call. The government and the army had been playing around with mobile phone calls. But it was really a shift in paradigm that almost overnight the business models changed. We had different types of regulation and we obviously had different types of technology that could allow us to do different things. Things. Now, this picture is not the exact phone. It's very similar to this in 1973. But the paradigm shift of having people lining up at payphones into then using mobile technology was a good example. It changed everything. And now, only about 50 years later, 45 years later, we're looking at billions and billions of handsets sold every year. And in many countries, in certain demographics, it's really the only computer that people use. And the cell phones that you all have in your pocket right now are, are more powerful in computing processing than what put the first man on the moon that NASA was using as their main computers? So we look at this as an evolutionary move forward and we ask ourselves, can you really change personal computing? The idea of personal computing evolving without changing the idea of a computer. So is it a better laptop? Is it faster speeds? We don't think so. We started looking at some ideas. Well, what do we actually care about? What are we trying to solve? Natural interaction, the thing that I'm doing right now, I'm speaking to you, I'm moving around and I'm gesturing. These are all things that humans have done forever. Computers can't do those. They're starting to be able to do it. But computers are ironically, finally becoming smart enough that they can understand us instead of the other way around. So that we have to learn how to type or to click a mouse, or to interact with a computer in a way that it understands. And then we said, well, how do we enhance the world around us? So why do I have to look at a screen? Why am I blocked by this thing here or this thing that's in my pocket? What if I could just augment the world around me by using information that way? And finally, how do these holographic experiences or this idea of data or visualized information, how can that change the world that I'm living in? And let me do experiences that have never been possible before. And we created what we call, as it was announced, the world's first fully untethered holographic computer. So it's a wearable computer that you have on your head. There's multiple laptops worth of computing power in this one device at less than half the weight of one computer. So there's a tremendous amount of computing processing going on in there. And it has to be wearable, has to be lightweight, it has to be heat controlled. And there's a few things that we learn along the way. I'll talk about some of them, but I'm going to roll a video for a few minutes that shows some of our designers and engineers and some of the behind the scenes looks at what they had to deal with when we were creating the product. We're gonna roll that right now.

When you think about how you experience technology today, it's like behind this glass screen and you're kind of stuck. It feels cold. We've unlocked the screen.

Really what we're trying to do is break down the walls between technology and people.

This is the next generation of computing. This is the next PC.

It's basically a see through display that you wear.

And it lets you see holograms in your world, in space, all around you.

A hologram is made out of light, which ultimately allowed us to pour digital assets over the real world so that humans can essentially be entertained and be empowered to do things in brand new ways.

The ability to draw with your hand or to take an asset, you know, an image and move it, or scale it is a very natural way of interacting with it.

Here's what you can do with holographic video. Here's what you can do with using your world as a game level. Here's what you can do with taking the power of 3D modeling and creation, and guess what? Pulling it out of a screen into real 3D and then sharing what you've created with others.

That was science fiction, and now we're bringing it into science fact.

They've been working on this for, you know, years, secretly. You need to have a bunch of obsessive people who see this as a meaningful part of their lives.

This is by far, far, far the hardest, most complex, most challenging, and most intriguing thing I've ever done in my life.

I mean, we are literally smacking up.

Against some of the laws of physics on what we can and can't do with light.

For all this complexity, it has to be incredibly easy, incredibly familiar, incredibly universal. How you use this new palette.

We just recently started engaging with third parties to build applications for the device. And the things they're doing, just, they're humbling.

When you've got a thousand different people from all different backgrounds with all different types of experience, you're going to see that the stuff that we've built at the beginning is just a tiny scratching of the surface of what this technology can do and what it can mean for for people.

When the other perspectives come to it and all the creatives out there, it's going to blow our minds.

This is a new medium for artistic expression and a new medium for creation that we've just never had before.

This platform is going to truly blend.

Your digital life with your physical life, and it is so thrilling. This is truly about seeing the world in a whole different way.

So when you're creating new products, the first thing that mostly your executive team will want to know is what category does this fit into? We as humans always want to categorize something, and in this space, the most common thing that people talk about is virtual reality. It's something that we've actually had around since the 1800s. We've all played virtual reality games with things like View Masters. So we understand there's a physical reality, the world that we're in right now, and there's a virtual reality. But by definition, virtual reality is more about creating a fake or digital world that obscures or obstructs the world around you. And it's great for certain experiences. But that's not quite what HoloLens was about, because we have see through lenses and we want to augment your world. Augmented reality on the other hand, is something that we could call it. But augmented reality has a connotation in our industry that it's a basic fixed piece of information that is locked to you and follows you around. HoloLens is contextual to the environment, so it understands that this is the edge of the stage, or there's chairs or there's people. And it can change the information based on what I'm looking at. So we said that's not quite right for those two things. And we think there's something in between this that we call mixed reality and the ability to blend your physical and your virtual worlds. It takes elements of virtual reality, elements of augmented reality, but really is its unique thing in between. And how do we do that from a product perspective and really understand what mixed reality could be? The first thing is it's untethered. There are no wires that come off of the device. It doesn't need any markers in the room. It has onboard cameras. It looks around and knows what you're looking at. And you don't need to tether it to a phone or to a PC. So everything is fully self contained right on your head. When you start to strip away what's underneath it, the first thing that we had to do is create a bunch of sensors. There are six cameras in the device and many other sensors that understands where you are in the room, it understands your location, it understands the context of what you're looking at. And that's what allows us to put holograms in the world in a relevant way. I mentioned transparency. This is a key element of HoloLens, the ability to see through. If I was wearing it right now, or for those of you who've demoed it, the ability that I could see you sitting here in the front row, but I could augment this experience by putting different types of digital elements around you. This also preserves your peripheral vision so that when you're wearing the device, you can be walking around and you're not constantly worrying about tripping. There's other types of products on the market that don't allow for this and they have more of a seated experience, which might be more immersive, but this allows for more mobility. We had to create a different type of processing unit. There's a lot of information that's constantly going and being processed by this. So it runs all the applications natively on the device, but we also have to take all the information from the, the sensors and make sense of it. So if you look at this as an evolution of Computing. You can compare this to the CPU to a move to the gpu. This is now a move to what we call the HPU or the holographic processing unit. So the ability for the silicon and all these processors to work together to take all this information of what you're seeing, plus run the application and do it in a way that has never existed before. And we've heard a lot about sound. Sound is incredibly important. We don't just see, see in 3D, we hear in 3D. So in the real world, if something is going on behind me, my head and my brains and my ears know that something is happening back there. We have 3D positional audio on the device that allows us to project Sound as a 3D matrix so that you can hear a hologram that's over here. This opens up a lot of different possibilities for things like training or for education or for safety. Now, when we talk about the input of the device, we refer to it as ggv. Gaze, gesture, and voice. Three things that we all do all the time. Your eyes are your cursor. So as you look at something, just like in the real world, if I want to focus on something that's in front of me, I look at it and then I look over here. That works as your cursor gesturing. The ability to move around like I'm gesturing with my arms. Or if we want to make some sort of selection, you can do that with the device. The device is smart enough to understand what you're doing and of course, voice one of the most natural inputs that we all have. We can speak to the device to have it do what we want it to do. So as you're developing products for this in applications, we're seeing these three main inputs or modalities are allowing people to be free. They're not looking at controllers, they're not looking at keyboards or mice. Those are all options if you want to use those. But these allow for the computer to understand us as people. Now, we talked about some of the things that we had envisioned in some of the applications that we've built, but that was really just the beginning. We make the technology, but the technology becomes good or successful or fails based on what the community and the ecosystem of people using it do. And we've seen some amazing work from people like NASA to Case Western Reserve University, to individual developers, to Volvo, Volkswagen, Audi. Lots of amazing applications that people are doing right now. The device has only been out since March, and we've already seen tremendous interest and just some very, very cool applications that our partners are doing. So I want to show a very short video of just some of the work that has been created with HoloLens that our partners have. SA it's so that was Scott Kelly on the International Space Station. They made the first intergalactic Skype call using HoloLens and Skype. We just heard back from our partners at Case Western Resources. They have medical students get one cadaver lab. Mostly in medical school, they did their cadaver lab and then they did a digital cadaver lab. And they tested the students between each for comprehension. And the results following the digital cadaver lab were really high. And they were surprised because the students had the context of the physical. But then they could explore alternative types of anatomy by seeing virtual cadavers. And their comprehension of the body went way up. We're seeing some really interesting things. We just announced yesterday that we're shipping in six more countries, and we're already seeing in Australia and Germany and France some amazing developers coming out with some great stuff. So this is really just the beginning. If we're at the Watson Graham Bell age of our technology, I think we've just started to scratch the surface. The next products will be even more amazing. The things that competitors or partners make will be even more amazing. But what we really look at is this is a paradigm shift of how we use technology. The technology is again, finally powerful enough that it can understand who we are as humans and how we communicate and how we interact. It can then work for us versus the other way around. And whether we call that mixed reality or augmented reality or virtual reality, it is all completely a new reality that allows us to work differently with machines and allows us to accomplish the things that we have originally set out to do. So with that, I think we have plenty of time for questions, and happy to take any.

You talked a little bit about how technology gets adopted through various channels. Historically through the military, then perhaps through enterprise, and then consumer, especially with unwieldy technologies are very big. Is the military using this?

I feel like that was a question that had a different angle. Military is actually one, the first virtual reality. If you go back and look at some of them, the Sword of Damocles is kind of a famous example where there was this massive room and this person had to be put into this cage, basically, and they had these things hanging from the ceiling. And that was the first version of virtual reality. And it has a lot to do with just being open to new technologies and also big budgets. And so you do see military adopting NASA was one of our first customers. They've been working with us. They actually get Mars rover information every 24 hours, comes back to Earth and they take it and they digitize it. And their scientists around the country and around the world can work together and look at the same thing at the same time. They've never been able to do that before. They just took all of that and consumerized it and opened up the space center in Orlando. And now consumers and customers can go through and see, see basically via Buzz Aldrin, what you can see on Mars. And so there is. That's sort of a micro example of going from a government or a military type application for commercial use into the consumer side. You saw a few others in the videos. Lowe's home improvement stores, they're really using it. It's a cost saving measure for them to build out kitchens and bathrooms in all of their stores. Every time something changes, it's incredibly expensive. And so if they have sort of a baseline kitchen or a bathroom, and then they can put a digital overlay on top of that, and you could change the cabinets and the sinks and all that, it saves them a lot of time. And their designers can work very, very differently with their customers. That stuff's happening right now.

We were talking earlier about the fact that to form a habit should experience something more than once a week. Do you see a future where regular consumers would be using the same kind of technology at home or during their day to day life more than once a week? Because a lot of the examples that you're giving, which are very interesting, seem to be talking about a one off, like I'm remodeling my home, for instance.

Right. Remember the days when we used to say, why would I want a car phone? Why would I want a phone in my car? I have a phone in my house. I'm too young. I remember, maybe I'm dating myself. But I thought the greatest phone technology improvement we had in my house growing up was when, when my parents allowed us to get the longer phone cord that my sister and I could take into the other room. There's a time when all technology, we look at that and go, why would we need that? Why do I need a cell phone that I see. Most people in this room have been checking at some point because it's how we get our information. It's the technology that we work with. Phone is a really easy example. Laptop's another one. We used to work in a society that would do everything on paper and flip charts. And now if you Come to a meeting like this and, and someone doesn't have PowerPoint, you think, what's wrong? So do I see a world where something like a Hololens or a head mounted computer or something that is like, that becomes more mainstream? Absolutely. It needs to get smaller, it needs to get lighter, it needs to have more applications. All of the typical things that you would need for any new technology. But I absolutely do see a world for it.

So I'm actually a bit curious about how you guys decided to embark upon this in the first place. Like, it's just sort of the things like this may cost us a couple hundred million dollars. We have no idea. Like it might take us seven years and it's not going to be commercially viable. Fifteen, like, how do you pitch that inside of your company?

You sound like my cfo. The mission of Microsoft is, you know, for all times, has been about empowering people in new ways through technology and making technology, be it hardware or software, that allows for that to happen. And we speak very, very honestly and quite extensively about what is the future of personal computing. And again, it could be better laptops. We make those too. We get better battery life, bigger screens. That stuff's all really important. And that's right for what the paradigm of working is right now. But we knew there was more. And it's not about what is the technology that we can just go build because we can go build it. It's what are we trying to solve for the customer and to allow them to do something with technology. And we said, well, the answer here is let's get rid of the screen. Let's not have something that has to be a two dimensional thing. We all work in 3D, we all see in 3D. Why are we not using 3D as a means to communicate and collaborate, be entertained? So that was what drove us in terms of how do we do it. It's a lot of persistence and no, not the whole time. People did not want to keep this project going. It was in development for years and years and years before it was ever made made public. And a lot of it came from learning that we had with Xbox and Kinect. So we basically had a depth sensor and a depth camera in your living room or family room as you're playing a game. And a lot of that technology, if we hadn't done Kinect, we never could have done hololens. This is kind of think of this as like Kinect on steroids on your head with a big computer attached to it. Not our marketing message but it's, it's close. So we had learning from connect, we had learning from other parts of the business, and we also had just a passion and a belief that we could really change how people worked with computers. Well, thank you very much.