Why Light Field Cameras are Hard

Earlier this week, I received a comment on my Magic Leap video asking me to do a deep dive into Lytro. I was only vaguely familiar with the product, so I did some research and put together some thoughts on the company and why it never received mass adoption. I think it's a really interesting case study in the race between hardware and software solutions to common problems. Let's jump right into it. Lytro designed and sold what are called light field cameras, meaning that they capture not only the intensity of light in a scene, but also the direction that the light rays are traveling. By capturing this extra data, the focus of the resulting image can be changed after the fact. Everyone has tried to capture an important moment, only to later realize that the subject was blurry. With a light field camera, that problem promises to be a thing of the past. The inner workings of a light field camera are extremely complex, so I'll let Larry Greenmeier from Scientific American explain how it works.

A light field camera lets you fix the focus of an image even after it's been taken. So how does it work? The image sensor in a normal digital camera adds up all of the rays of light around whatever you're photographing and counts them as a single amount of light. It's helpful to use music as an analogy. Taking a conventional digital photo is like recording all the instruments in a band all at once. A light field camera, on the other hand, would be like recording individual instruments on separate audio tracks. That's because a light field camera uses a special sensor to record the color and intensity of light traveling in every direction, through every point in space. Breaking down a song into individual tracks lets you fine tune the sound. In the same way, breaking down a photo into individual rays of light lets you improve the clarity of a picture even after it's been taken. Let's use the Lytro light field camera as an example of how this works. A litro camera has a lens, a light field sensor, and a microprocessor. This setup isn't too different from a normal digital camera. What is different about the Lytro is that its sensor is covered by a matrix of microlenses. These provide an enormous amount of information about direction, color, and intensity of light. Then the Lytro's microprocessor figures out what an image should look like when it's in focus.

Although the technology is advanced, you can immediately see the benefits of this type of camera. So why aren't we all using light field cameras now? Well, the company was founded in 2006 and it's important to remember how different the world of photography was back then. The iPhone hadn't been announced yet and CNET had just crowned The Sony Ericsson K800i the best camera phone of the year, and it only had a 3.2 megapixel camera. At the higher end, Canon had just launched the 5D Mark I, which was a breakthrough in digital photography, but it only shot 12 megapixel images and it cost over $3,000. And that's before you bought a lens. Additionally, Instagram wouldn't launch until four years later, so photo sharing was still in its infancy. It took six years for the company to launch its first product to market, and by the time the consumers got their hands on the first Lytro camera, technology had advanced significantly. The boxy design was awkward for most users and the post processing workflow was extremely tedious. So since the photos couldn't be edited in Photoshop or Lightroom, the ability to change focus after shooting was definitely cool. But reviews were mixed and the product never took off. To Lytro's credit, the team was able to release a second camera in just two years. The new camera produced more detailed images and featured a more ergonomic design. Unfortunately for Lytro, though, reviews were again mixed. Editing photos from the camera still required the use of Lytro's special app, and it never gained adoption from mainstream photographers. It's important to remember that focus is just one factor that makes a photo look professional. The lighting, composition and colors all play huge roles. And while Lytro made focusing easier, it made tweaking, lighting, composition and color balance in post production extremely difficult. And it's not like the rest of the photography industry had been asleep while Lytro was working on their cameras. Canon launched Dual Pixel Autofocus in 2013, which made nailing perfect focus on high end and prosumer cameras extremely easy and reliable. Additionally, Apple introduced portrait mode in 2016 with the iPhone 7. By using two separate lenses on the new iPhone, Apple's image processing pipeline would be able to produce photos that look like they were taken with a telephoto lens. And this was just the beginning of computational photography. In the last four years, camera sensors on smartphones have essentially plateaued. Resolution is a crude measure of sensor quality, but the last five generations of iPhones have all been 12 megapixels. All the innovation is happening on the software side now. Features like night mode, Smart HDR, and portrait mode are all driven primarily by advances in computational photography. And with the ubiquity of modern social media what's the point of a great photo if you can't share it with the world in a single click? Faced with all of these headwinds, Lytro attempted to shift their focus to VR video capture, but again, their solution wasn't applicable to a wide range of users. It's really unfortunate because light field technology has incredible applications for VR. And just look at this amazing demo from Google earlier this year.

Our pipeline produces volumetric free viewpoint video that can be explored with six degrees of freedom within a spherical 70 centimeter diameter viewing volume. Our capture rig consists of 46 time synchronized action sports cameras mounted on a 92 centimeter diameter plastic hemisphere. It is inexpensive and relatively easy to fabricate. The cameras are roughly 18 centimeters apart and each has a 120 degree by 90 degree field of view. Their overlapping fields of view support reconstruction of objects close to the dome surface. In these scenes we show near field objects as close as half a meter from the cameras.

While that demo is cool, it really shows how far this technology is from general consumer adoption. That Google rig is far too large for any reasonable consumer to use and the adoption of VR head mounted displays is still low. Additionally, that Google demo relies on an incredible amount of software to process the video feeds and interpolate different views to construct the video volume. The hardware is actually pretty simple. The story of Lytro really illustrates how hard it is to bring a new hardware device to market. Even a huge success like GoPro is still worth less than a billion dollars, while Instagram was acquired for a billion dollars only two years after launch. There's a saying in Silicon Valley that hardware is hard and stories like Lytro really enforce that. But what do you think? Did you ever use a Lytro camera? Leave me a comment below and consider subscribing if you enjoyed this video. Thanks for watching.