Palmer Luckey on the Pentagon’s future of mixed reality

Welcome to MIT Technology Review. Narrated My name is Matt Honan. I'm our Editor in Chief. Every week we'll bring you a fascinating new in depth story from the leading edge of science and technology, covering topics like AI, biotech, climate, energy, robotics and more. Here's this week's story. I hope you enjoy it.

My name is James o' Donnell and I'm an AI reporter at MIT Technology Review. The story we're bringing you today is about one fight to make augmented reality headsets useful, not for immersive entertainment, but for warfare specifically. It's about a program the US army has been working on for years without much success, to make AR headsets for soldiers. To get a sense of the story, I spoke with one of the most influential people in AI. Palmer Luckey built the first Oculus headsets, sold the technology to Facebook for $2 billion and and then founded his own defense company, Anduril, after he was ousted from Facebook. Lucky is now being brought into the Army's headset project and he's optimistic that it can work. If it does, it will bring AI onto the battlefield in ways we haven't seen before. Enjoy the story.

Narrated by Noah Listen to more of.

The best articles from the world's biggest.

Publishers on the NOAA app or@newsoveraudio.com Palmer Luckey has in some ways come full circle. His first experience with virtual reality headsets was as a teenage lab technician at a defense research center in Southern California, studying their potential to curb PTSD symptoms in veterans. He then built Oculus, sold it to Facebook for $2 billion, left Facebook after a highly public ousting, and founded Anduril, which focuses on drones, cruise missiles and other AI enhanced technologies for the US Department of Defense. The company is now valued at $14 billion. Now Lucky is redirecting his energy again to headsets for the military. In September 2024, Anduril announced it would partner with Microsoft on the US Army's Integrated Visual Augmentation system, arguably the military's largest effort to develop a headset for use on the battlefield. Lucky says the IVAS project is his top priority. At Anduril. There is going to be a heads up display on every soldier within a pretty short period of time, he told MIT Technology Review in an interview in October 2024 on his work with the IVAS goggles. The stuff that we're building is going to be a big part of that. Though few would bet against Lucky's expertise in the realm of mixed reality, few observers share his optimism for the IVAS program. They view it thus far as an avalanche of failures. IVAS was first approved in 2018 as an effort to build state of the art mixed reality headsets for soldiers. In March 2021, Microsoft was awarded nearly $22 billion over 10 years to lead the project, but it quickly became mired in delays. Just a year later, a Pentagon audit criticized the program for not properly testing the goggles, saying its choices could result in wasting up to $21.88 billion in taxpayers funds to field a system that soldiers may not want to use or use as intended. The first two variants of the goggles, which the army purchased 10,000 units, gave soldiers nausea, neck pain and eye strain, according to internal documents obtained by Bloomberg. Such reports have left IVAS on a short leash with members of the Senate Armed Services Committee, which helps determine how much money should be spent on the program. In a subcommittee meeting in May 2024, Senator Tom Cotton, an Arkansas Republican and ranking member, expressed frustration at IVIs slow pace and high costs, and in July 2024, the committee suggested a $200 million cut to the program. Meanwhile, Microsoft has for years been cutting investments into its HoloLens headset, the hardware in which the IVAS program is based, for lack of adoption. In June 2024, Microsoft announced layoffs to its HoloLens teams, suggesting the project is now focused solely on serving the Department of Defense. The company received a serious blow in August 2024 when reports revealed that the army is considering reopening bidding for the contract to oust Microsoft entirely. This is the catastrophe that Lucky's stepped into. Anduril's contribution to the project will be Lattice, an AI powered system that connects everything from drones to radar jammers to surveil detect objects and aid in decision making. Lattice is increasingly becoming Anduril's flagship offering. It's a tool that allows soldiers to receive instantaneous information not only from Anduril's hardware, but also from radars, vehicles, sensors and other equipment not made by Anduril. Now it will be built into the IVAS goggles. It's not quite a hive mind, but it's certainly a hive eye is how Lucky described it to me. Boosted by Lattice, the IVAS program aims to produce a headset that can help soldiers rapidly identify potential threats and take decisive action on the battlefield, according to the Army. If designed well, the device will automatically sort through countless pieces of information drone locations, vehicles, intelligence and flag the most important ones to the wearer in real time. Lucky defends the IBAS program's bumps in the road is exactly what one should expect when developing mixed reality for defense. None of these problems are anything that you would consider insurmountable, he says. It's just a matter of if it's going to be this year or a few years from now. He adds that delaying a product is far better than releasing an inferior product, quoting Shigeru Miyamoto, the game director of a delayed game is delayed only once, but a bad game is bad forever. He's increasingly convinced that the military, not consumers, will be the most important testing ground for mixed reality hardware. You're going to see an AR headset on every soldier long before you see it on every civilian, he says. In the consumer world, any headset company is competing with the ubiquity and ease of the smartphone. But he sees entirely different trade offs in defense. The gains are so different when we talk about life or death scenarios. You don't have to worry about things like, oh, this is kind of dorky looking, or oh, you know, this is slightly heavier than I would prefer, he says, because the alternatives of, you know, getting killed or failing your mission are a lot less desirable. Those in charge of the IBAS program remain steadfast in the expectation that it will pay off with huge gains for those on the battlefield. If it works, james Rainey, commanding general of the Army Futures commander, told the Armed Services Committee in May 2024, it's a legitimate 10x upgrade to our most important formations. That's a big if, and one that currently depends on Microsoft's ability to deliver. Lucky didn't get specific when I asked if Anduril was positioning itself to bid to become IRAS's primary contractor should the opportunity arise. If that happens, US troops may willingly or not become the most important test subjects for augmented and virtual reality technology as it is developed in the coming decades. The commercial sector doesn't have thousands of individuals within a single institution who can test hardware in physically and mentally demanding situations and provide their feedback on how to improve it. That's one of the ways selling to the defense sector is very different from selling to consumers, lucky says. You don't actually have to convince every single soldier that they personally want to use it. You need to convince the people in charge of him, his commanding officer, and the people in charge of him that this is a thing that is worth wearing. The iterations that eventually come from IVAs, if it keeps its funding, could signal what's coming next for the commercial market. When I asked Lucky if there were lessons from Oculus, he had to unlearn when working with the Department of Defense, he said. There's one worrying about budgets. I prided myself for years. You know, I'm the guy who's figured out how to make VR accessible to the masses by being absolutely brutal at every part of the design process, trying to get costs down. That isn't what the DoD wants, he says. They don't want the cheapest headset in a vacuum. They want to save money. And generally spending a bit more money on a headset that is more durable or that has better vision and therefore allows you to complete a mission faster is definitely worth the extra few hundred dollars. I asked if he's impressed by the progress that's been made during his eight year hiatus from mixed reality. Since he left Facebook in 2017, Apple, Magic Leap, Meta, Snap and a cascade of startups have been racing to move the technology from the fringe to the mainstream. Everything in mixed reality is about trade offs, he says. Would you like more computing power or a lighter and more comfortable headset? With more time at Meta, I would have made different trade offs in a way that I think would have led to greater adoption, he says. But of course, everyone thinks that while he's impressed with the gains having been on the inside, I also feel like things could be moving faster. Years after leaving, Lucky remains noticeably annoyed by one specific decision he thinks Meta got wrong not offloading the battery. Dwelling on technical details is unsurprising from someone who spent his formative years living in a trailer in his parents driveway, posting in obscure forums and obsessing over goggle prototypes. He pontificated on the benefits of packing the heavy batteries and chips in removable pucks that the user could put in a pocket rather than in the headset itself. Doing so makes the headset lighter and more comfortable, he says. He was pushing Facebook to go that route before he was ousted, but when he left, it abandoned the idea. Apple chose to have an external battery for its Vision Pro, which Lucky praised. Anyway, he told me. I'm still sore about it eight years later. Speaking of soreness, Lucky's most public professional wound his ouster from Facebook in 2017, was partially healed in September 2024. The story involving countless Twitter threads, the doxxing retractions and corrections to news articles, suppressed statements, and a significant segment in Blake Harris 2020 book the History of the Future is difficult to boil down, but here's the short version. A donation by Lucky to a pro Trump group called Nimble America in late 2016 led to turmoil within Facebook after it was reported by the Daily Beast. That turmoil grew, especially after Ars Technica wrote that his donation was funding racist memes. The founders of Nimble America were involved in the subreddit the Donald, but the organization itself was focused on creating pro Trump billboards. Lucky left in March 2017, but Meta has never disclosed why. In April 2024, Oculus former CTO John Carmack posted on X that he regretted not supporting Lucky more Meta's CTO Andrew Bosworth, argued with Carmack, largely siding with Meta. In response, Lucky said, you publicly told everyone my departure had nothing to do with politics, which is absolutely insane and obviously contradicted by reams of internal communications. The two argued in the X argument, Bosworth cautioned that there are limits on what can be said here, to which Lucky responded, I am down to throw it all out there. We can make everything public and let people judge for themselves. Just say the word. Six months later, Bosworth apologised to Lucky for the comments. Lucky responded, writing that although he is infamously good at holding grudges, neither Bosworth nor current leadership at Meta was involved in the incident. By now, Lucky has spent years mulling over how much of his remaining anger is irrational or misplaced. But one thing is clear. He has a grudge left, but it's against people behind the scenes PR agents, lawyers, reporters who, from his perspective, created a situation that forced him to accept and react to an account he found totally flawed. He's angry about the steps Facebook took to keep him from communicating his side. Lucky has said he wrote versions of a statement at the time, but that Facebook threatened further escalation if he posted it. What am I actually angry at? I'm angry that my life went in that direction, absolutely, he says. I have a lot more anger for the people who lied in a way that ruined my entire life and that saw my own company ripped out from under me, that I'd spent my entire adult life building. He says I've got plenty of anger left. But it's not at Meta, the corporate entity. It's not at Zuck, it's not at Boz. Those are not the people who wronged me. While various subcommittees within the Senate and House deliberate how many millions to spend on IVAs each year, what is not in question is the Pentagon is investing to prepare for a potential conflict in the Pacific between China and Taiwan, The Pentagon requested nearly $10 billion for the Pacific Deterrence Initiative in its latest budget. The prospect of such a conflict is something Lucky considers often. He told the authors of Unit X how the Pentagon and Silicon Valley are transforming the future of war that Anduril's entire internal roadmap has been organized around the question, how do you deter China? Not just in Taiwan, but Taiwan and beyond. At this point, nothing about IVAS is geared specifically toward use in the South Pacific, as opposed to Ukraine or anywhere else. The design is in early stages. According to transcripts of a Senate Armed Services subcommittee meeting in May 2024, the military was scheduled to receive the third iteration of IBAS goggles earlier this summer. If they're on schedule, they're currently in testing. That version is likely to change dramatically before it approaches Lucky's vision for the future of mixed reality warfare, in which you have a little bit of an AI guardian angel on your shoulder helping you out and doing all the stuff that is easy to miss in the midst of battle. But will soldiers ever trust such a guardian angel? If the goggles of the future rely on AI powered software like Lattice to identify threats, say an enemy drone ahead or an autonomous vehicle racing toward you, Anduril is making the promise that it can sort through the false positives, recognize threats with impeccable accuracy, and surface critical information when it counts most. Lucky says the real test is how the technology compares with the current abilities of humans. In a lot of cases, it's already better, he says, referring to Lattice as measured by Anduril's internal tests. It has not released these, and they have not been assessed by any independent external experts. People are fallible in ways that machines aren't necessarily, he adds. Still, Lucky admits he does worry about the threats Lattice will miss. One of the things that really worries me is there's going to be people who die because Lattice misunderstood something or missed a threat to a soldier that it should have seen, he says. At the same time, I can recognise that it's still doing far better than people are doing today. When Lattice makes a significant mistake, it's unlikely the public will know. Asked about the balance between transparency and national security in disclosing these errors, Lucky said that Anduril's customer, the Pentagon, will receive complete information about what went wrong. That's in line with the Pentagon's policies on responsible AI adoption, which require that AI driven systems be developed with methodologies, data sources, design procedures and documentation that are transparent to and auditable by their relevant defense personnel. However, the policies promise nothing about disclosure to the public, a fact that's led some progressive think tanks like the Brennan Centre for Justice to call on federal agencies to modernize public transparency efforts for the age of AI. It's easy to say, well, shouldn't you be honest about this failure of your system to detect something? Lucky says regarding Anduril's obligations, well, what if the failure was because the Chinese figured out a hole in the system and leveraged that to speed past our defenses of some military base? I'd say there's not very much public good served in saying, attention everyone, there is a way to get past all of the security on every US military base around the world. I would say that transparency would be the worst thing you could do. You're listening to MIT Technology Review, where James o' Donnell writes Palmer Luckey on the Pentagon's future of mixed reality. This article was published on 28th October 2024 and was read by Martin Buchanan for Noah.