Photonic computing for AI acceleration

If you look at technologies for powering AI right now, they are all based on transistors. There's never been a computer that's not based on transistors. Like this is the way that the world does computation. But we've run into this kind of fundamental challenge around shrinking how much energy the chips are using and you can't really do that going forward. So if you look at the Department of Energy's estimate for energy consumption, 2030, 10% of the entire planet's energy consumption be on compute and interconnect. So if we look at 2040, you're talking about most like the overwhelming majority of the power being used on this. You always have to think in business about the use case that could motivate using that much of the planet's power because that would cost a lot of money. You will start to see that progress in AI will slow.

Big thanks to our partners Linode, Fastly and launchdarkly.

We love Lino.

They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly.

Learn more at vastly.com and get your.

Feature flags powered by LaunchDarkly.

Get a demo at launchdarkly.com. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're at PracticalAI FM.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

Doing very well, Daniel. Excited about today's show. I think we have some pretty good stuff coming up here. I'm just kind of rearing to go.

We've talked about a lot of computers on this show and accelerators and cards and GPUs and GPUs and all sorts of views. But we have yet to talk about photonic computers. And that's going to be the topic of today's conversation. Today we've got with us Nick Harris, who is CEO of Light Matter, and we're going to hear a lot more about that and photonic computers today. So welcome. Nick. It's great to have you here.

Thanks for having me, Daniel and Chris, thanks and nice to meet you.

So a lot of people might have not heard of photonic computers or even know where they sit in the sort of landscape or ecosystem of accelerators and types of processors and all of this stuff. So maybe just give a little bit of a background on how you came to encounter this topic and then also like maybe some of the motivations behind this whole topic of photonic computing.

Yeah, so I'll start out with just talking about the AI field as it's growing. So in around 2012, we had Alexnet come out. Alexnet was a neural net that was able to do image recognition tasks that were really incredible. And there was no other technique that could really keep up with it. And it made a great business case for companies like Microsoft, Google, Facebook, Amazon and so on to start building neural networks to do special kinds of problems, things that were very hard to write code for, but that neural networks could solve. And over the past 10 years now, we've had a really fast rate of progress in the amount of compute that's going into these state of the art neural networks. It's motivated a lot of companies to start building accelerator chips to try to speed up these computations and make it so that we can make bigger and bigger neural nets. And some people's objective on this bigger and bigger scale is really to try to build something that's on the scale of a human brain. So that's kind of the trajectory where things are going. There's huge capital expenditures that go into building these kinds of AI supercomputers that can do incredible tasks. And what we're doing at Light Matter is we're trying to take a crack at that market with a unique twist. We're using photonics to do computation. I can tell you how I got to that place.

Please do.

Yeah. I was an engineer at Micron working on transistor device physics. So basically I was getting a lot of exposure to the fab processes and some of the challenges that companies, not just Micron, are having and shrinking transistors and getting more performance out of them year over year. So I decided to go to graduate school, ended up at MIT and studying quantum computing. It's a bit of a leap. What light matter is doing isn't quantum computing. But I did spend those years building photonic quantum computers. And that happened right at the advent of that Alexnet neural network kind of coming out. And what we realized in Dirk England's group at MIT was that you could actually use the same kind of processor we were doing for quantum computing to do computing on traditional neural networks using just lasers and the benefits would be pretty massive.

Could you tell us, like, what that implies? What does that mean, how to use lasers for computing? Can you kind of bridge that gap between those two?

Yeah, there's a little bit of history on this. So we're not the first people who have looked at photonic computing. People realized first, I think, in the 80s, that you might be able to do neural network evaluation with photonics. And that field kind of like petered out for a while. But we came at it from a totally different approach. So we were looking at quantum computers, and we had a different motivation and different set of funding pitches for getting money to build these machines. We really took integrated photonics. That's a field that largely the way that you experience integrated photonics is through silicon photonics transceivers. So these are devices in data centers that send your communications, your data between different server racks. Maybe they're hosting a game you're playing, maybe they're mediating your imessages, something like that. But we use that core technology to start doing computing. Now. How do you do it with lasers? In the quantum computing case, we weren't using lasers, we were using single photons. But that's a story for another day. Lasers are devices that generate light. You give them an electrical signal and they're able to start generating light. And they're used in all sorts of traditional communications. The Internet that we're talking over right now is over fiber optics. And the communication is sent using lasers to do that. So we're able to leverage that same infrastructure, lasers and silicon photonics, to actually do core computations that happen in deep learning. And these computations are, I think, tractable really for anybody to understand. Deep learning as a field is really just relying on multiplication and addition to do evaluations. So the mathematics behind deep learning are relatively straightforward. Understanding how to come up with these things isn't straightforward. But when you actually run these programs, it's a lot of multiplication and addition. And indeed, you can use lasers and silicon photonics to do multiplies and addition. And I'm happy to go into how that works a bit later on.

Yeah, I'm curious. Probably the main thing people think about Maybe is like GPUs with computation on deep learning. And you're exactly right. I was just pulling up while you were talking, the recent work from Microsoft and Nvidia, I think, on the Megatron model. So they just have a figure where it's like. I think most people maybe in our audience have heard of Bert, you know, heard that in the context of AI, which had 340 million parameters in the model GPT3, which I think, Chris, you just had a conversation with someone about on the last show. Yeah, 175 billion parameters. And then this Megatron, which is the new natural language generation model, which is talked about just recently. 530 billion parameters. So just commenting and following up on your conversation about these large models is pretty insane. I don't know. Like, I think generally people have in their mind racks and racks of GPUs when they're thinking of this. So like with a photonic computer, and this is maybe just like a very simple question, but to have something in people's mind where like, so I've been near some like quantum computing labs where like they've got dilution fridges and it looks like this like whole like vacuum system and all that.

You.

And then I think about like laser labs with all these like, you know, lasers and you've got these lenses and stuff. And then that seems very different to me than like, oh, I'm going into the data center. I see these racks of computers just as a very simple question. Like if you were to go to like look at a set of, you know, photonic computers, what does that look like form factor wise and like connection wise maybe as comparison to some of these other things?

Well, I would say the biggest deployments of AI are in hyper, hyperscaler environments and in cloud environments. And to play in those spaces you need to avoid doing anything weird. So a big dilution refrigerator or a vacuum, that's going to be a problem. And so if you look at photonic computers, like the things that we're developing, they just look like a normal silicon chip. They do have optical fibers that come out. You do need to get light into the processor so that it can do the calculations. But you know, it's really just a standard computer chip. It looks like a card sitting in a server. But there's a little addition of having a laser in there. Just maybe on the larger picture here there are so many different ways that people are looking at powering AI computation because the market opportunity is so big and we're all trying to power these 500 billion weight neural networks and it's going to take a lot of computing power. And that power is the principal problem. Computer chips are getting way too hot. That's really one of the fundamental things that we're interested in trying to help with.

When you said it was kind of one of the principal things is that kind of the primary value proposition. When you're saying go with photonics laser driven, it doesn't heat up. You don't have the massive problem that you have to deal with in a data center where you have a bunch of that is that the primary thing, are there other performance characteristics or non performance characteristics that play into the field in general in terms of the field itself having value proposition?

Yeah. So I can say some really general things about computers. If you have one processor and you run a program on that processor, you will get one processor worth of performance. If you take two processors and try to scale it up, it'll be just less than two processors worth of performance. When you get to a thousand nodes, you're going to be looking at something that is kind of doing the amount of computation that half the number of units will be doing. Yeah. So what I'm getting at with that is as you try to power bigger and bigger neural networks, you need to scale up. And the reason that you need to scale up is that the individual computer chips that we build Today, Intel's chips, AMD's chips, Nvidia's chips, they all consume a lot of power and you kind of have to spread them out and they're getting to be really big. And so this heat problem, in addition to what's known as Omdahl's law scaling, where every time you add a unit of compute, you do not get a unit of performance out, those two things kind of work together. So really power efficiency is tied to, to compute scaling, it's tied to compute per chip. There's really a maximum amount of heat that you can pull out of a processor. So it's just this whole story that's built around. You can't be trying to dissipate kilowatts of power in a single chip. You're not going to be able to cool it and you're certainly not going to be able to continue a roadmap where you're pumping more power through chips. So it's performance, energy efficiency and all that stuff is fundamentally linked and they sort of hamper your ability to scale out.

And I know that we've talked on the show a couple times about the environmental impacts of what we're doing with these sort of large models. And I think, you know, there was that one study, I forget how old it is now. It's actually probably a couple years old now. Where it was talking about like one of these large language models is like running five cars into the ground for like their whole life period to train like once. And you know, we're training them multiple times. So as you're talking about power, a lot of what I'm thinking is related to that sort of sustainability side of things. So like in terms of that power requirement side of things, I mean, lasers still require power, I'm assuming. So I don't really have a sense of like, what sort of scales are we looking at versus on like a sort of GPU accelerated AI versus like photonic based.

Yeah, maybe we can start with like a bit of the bigger picture around the energy scaling problem. So sounds good. When you double the number of transistors on a computer chip, that's Moore's Law. It should happen every 18 months. Maybe we're a little bit behind schedule on that, but Moore's Law is mostly okay. Every time you double the number of transistors on the chip, they're going to need to use less energy in order for that chip to not get really hot. And since around 2005, when we've shrunk the transistor, the amount of energy that they use isn't shrinking commensurately. And so the chips are getting hotter and hotter and hotter. It sort of pushed us to a spot where you've heard a lot about system on chip and Apple's recent announcement of the M1 chips. You can see it. These are system on chip platforms. They have lots of different functionality, but it turns out that if they turned all of those functionalities on at the same time, you would really hit the thermal limits for the system. So this is a result of the fact that the energy scaling in transistors hasn't continued with their shrinking. That's called Dennard scaling. So that's kind of what's toast right now. So when you look at solutions from Nvidia, for example, they're really pushing the limits of what's possible to cool. I can tell you. We have a packaging team at Light Matter. Their job is to make sure that you can pull the heat out of your computer chips. And we're all very impressed with how much heat they're able to get out of the A100 processor. It's something like 450 watts. There's a new chip from Intel, Ponte Vecchio, and that chip is 600 watts. Really cool technology. It's awesome. But this power thing, it's a real challenge. It turns out that once you get to those kind of numbers, Ponte Vecchio is water cooled. So you really have to go from a heat sink and a fan, which is what you'll find in your computer at home. Unless you're an enthusiast and you love water cooling. You have to move towards water cooling. And after that, you're seeing advertisements from Azure, Microsoft Azure, where they're doing immersion cooling. So they take computer chips and put them underneath apparently edible oil. Don't ask me how I know it's edible.

From the diner.

Yeah. I don't know if it's fry grease, but it's apparently edible. I don't know why you would ever eat it. Maybe it's like a reusable type material.

I don't know why they would advertise it as edible. It's an odd characteristic to note.

It shows safety, I guess, if you're like, if you can eat it, it's probably pretty safe. It sounds like a long chemical word, so edibility is probably a good sign. So the basic point here is that if you look at technologies for powering AI right now, they're all based on transistors. There's never been a computer that's not based on transistors. This is the way that the world does computation. But we've run into this kind of fundamental challenge around shrinking how much energy the chips are using. And you can't really do that going forward. So if you look at the Department of Energy's estimate for energy consumption, 2030, so in about eight years, 10% of the entire planet's energy consumption will be on compute and interconnect. So if you know about compound annual growth rates, if we look at 2040, you're talking about most like the overwhelming majority of the power being used on this. You always have to think in business about the use case that could motivate using that much of the planet's power, because that would cost a lot of money. And I think what happens is that you will start to see that progress in AI will slow because of the heat problem, because it puts a lot of financial pressure on data centers and the people who scale these things. To be clear, those neural networks that Daniel was talking about, those are trained by the biggest companies in the world, and they're hundreds of millions of dollars for the supercomputer. And running those models costs like $10 million. These are massive scale problems. And I would go out on a limb and say it's already probably an uncomfortable amount of money for those companies that they're spending on these things. So that's the state of things today. Everything's based on transistors. And you've really got Denard scaling, which, by the way, it's underpinned by quantum mechanics. It's not something that we can fix. TSMC doesn't have a solution for how they're going to get rid of this little pesky energy scaling problem. I wish they did, but that's kind of where photonic computing comes in. So we don't really have to worry about the quantum tunneling effects because we're not using transistors. It's a totally different type of device and so that part of the scaling doesn't matter so much. So hopefully that starts to give you a picture of like we just sort of said transistors. That's how people do computation for deep learning. We think we can do it with optics. We think we'll use a lot less energy and we'll get rid of that whole energy scaling problem. Just walk around it because we're using different physics.

Foreign.

Is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the Changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog. It's better.

So Nick, you just started getting to where I really was interested in discussing next, which is around like okay, we think we can do sort of these computations related to deep learning and AI with photonics. I'm curious, like I guess a first question is where we add in that process. Is it like this is being done and has been being done and you know, I suspect you've, you've done certain things like how far are we to like doing training and our inference in a reasonable way with photonics?

All right, so there's been a history here. I've been building these systems as part of my graduate work and now the company at Lightmatter for about a decade and over that time we've demonstrated a number of applications running language processing on these chips. That was done at MIT along with a bunch of other cool applications of the processor technology. At Lightmatter we announced the Mars chip and that was@hotchips 2020. Hot Chips is a conference on computer architecture and that chip is capable of running state of the art neural networks. And what we're up to right now at Light Matter is we are gearing up to start delivering our processors to customers. Very big companies that are interested in energy intensive AI and trying to power their roadmap. They really care a lot about how do you get to the trillion Parameter, neural network and beyond. So we're very far along in this journey, like quite confident that you're going to get to see it. And you know, we've built lots of prototypes along the way, peer reviewed stuff in academia, it's been published in Nature, Nature Photonics, Nature Physics, all the natures. And so it's real stuff and we're going to be selling it.

Yeah. And before we go into like the actual like what you'll be selling and some of like how it's related to AI and such, I'm curious, like that's been a long road, I'm sure. Like it hasn't been all smooth sailing. Have there been points along that path where you're like, oh, this is like I don't think we're going to make it or like, you know, what have been major bumps in the road or like major achievements along the way that have happened.

Entrepreneurial war stories here.

Yeah, building companies is a lot of work. Hiring people, hiring really good people takes a lot of time. I have a lot of entrepreneurial war stories. In terms of challenges along the way, I'd say that the first couple years of the company we were spending a lot of time trying to figure out exactly what the photonic compute architecture would look like. We knew it was silicon photonics. We had GlobalFoundries as our partner fabric, but we had to figure out exactly what we were going to build. And there are a lot of ways to build these processors. Like make no mistake, we've patented about 70 ways to do that. But there's only one that really works well from what we've seen. So narrowing that down was a lot of work. Building teams is super hard and especially building teams where you've invented a new field. Silicon photonics is a technology that's been deployed for about 10 years now. So data centers have been using it. Long haul communications, like across the country type communications have been using silicon photonics. But people were not trained in using silicon photonics for building computers because it didn't really exist. So that's some of the big stuff. The other pieces are, you know, you have to build a supply chain for these things and it's, you know, it's not straightforward. There are a lot of people who touch the hardware from start to finish and some of the steps are with companies that are, you know, smaller and you have to kind of work with them to try to get things productized and stable and, and all that sort of thing. You can think about lasers for example.

So can you describe a Little bit. For those of us who are trying to kind of wrap our, our minds around what this kind of chip is like, obviously you, you can't describe in detail the, the internals nor would you want to, but kind of give me a sense of what a photonic chip is like. What are some of the considerations that go into it. For those of us who are brand new to this field and just trying to. I know at the end of the day you're producing a board that's going to go into my computer, but what's this, what's this little photonic magical thing that's inside that, you know, how does that work? What's different about it from the way I might already be thinking about these other architecture chips that we've talked about.

So I can say something sort of boring, which is have you heard of the Google tpu? So gv, formerly Google Ventures is an investor and our technology looks a lot like a tpu.

Okay.

Yeah.

So it is a matrix processor with a lot of other things wrapped around it. It's not just doing matrix processing. There's a lot of things wrapped around it, emphasize on that. But it's basically doing linear algebra. So we have multiple processor cores. Our product's actually a quad core computer, just like an intel like quad core chip. And each one of those cores is doing linear algebra. If you were to look at the chip and just follow the light, what you would see is values from an image. Like let's say we have a pixel, it's red, green or blue and there's an intensity for each pixel from 0 to 1. What you'd see is a brightness of light corresponding to that intensity number. If we're doing image processing, kind of like being distributed around the chip and then you'd see these photonic units in a two dimensional array receiving the signals and then doing multiplication and addition. That's basically it. So it would look to you a lot like a TPU, this like box, like a 2D array of multiply accumulate units. Except you would see light being distributed to all of the individual components.

You talked a lot about heat and its sort of motivation in this. I imagine that there's a number of, you're thinking about the heat problem, but also with a, with this sort of chip I imagine there's like a lot of sort of like interference type things and we're thinking about light. Is there different considerations? Like maybe you have some benefits in the heat side of things, but are there other sort of like shielding or interference or other sort of Challenges related to that in terms of building the physical unit that you have to take into consideration that you wouldn't have to take into consideration with another type of computer.

So, yes and no, we're using light, and so light signals can't interfere with each other. So that's kind of an interesting property. However, whenever you build a metal wire, it ends up being kind of an antenna. And so in chip design, there are these things called antenna rules. And digital chips have to worry about these, too. And so, you know, we have to observe these things so that we don't get unwanted signals from radio hosts or something getting into the chip and messing up the optical computation through the wires. But otherwise, I think that it really just looks like a mixed signal chip. So there's analog circuits, there's digital circuits, and then the photonic circuits, which for all practical purposes are just analog type circuits. So you have to be careful about coupling unwanted signals through antennas that you didn't intend to be there. But otherwise, no, not really.

So I could still have all my RGB lighting in my computer case and, like, have flashing cool stuff.

Yeah, if that's the question. Absolutely. It turns out that it's very hard. We have these things, and I should have mentioned this, we have these things called wave guides. They're about 300 nanometers wide and about 200 nanometers tall. And they're optical wires that are on our chip. So they're really tiny, and we have hundreds of thousands of them on the chip. It's very hard to get light into those. And so if you shine the flashlight on our waveguides, you get, like, nothing into the waveguide. You have to really want to get the light in there for that to work out, which is fortunate and unfortunate.

I'm kind of curious, is there, if you're looking at the wire world that we've been in for so long and the physics, you know, as it gets smaller and smaller and they're running into all sorts of challenges, heat and otherwise. How does that translate? Do you, you know, you just talked about how small that optical wire is, if you will, and the fact that because of that, you don't get the light into it. What are some of the kind of. In the science of this? What are some of the limitations as you are constantly trying to get to smaller and smaller architectures to scale out? What are the things you have to think about?

Yeah, it's an interesting question. So at first glance, you would be like, all right, you've got a new kind of computer. We're going to want to shrink it every 18 months and have a photonic Moore's law or something, some kind of scaling law. You could go down that path. And the way you would do it is you'd start out, we work at 1550 nanometers wavelength. You could go down to 13, 10 nanometers, and then you could go down to 900 nanometers, by the way. Your eye will just start to pick this up and it will look like red and so on down. And then maybe you get to ultraviolet, which is what gives you skin cancer. Okay. And then we keep going. We get to X rays, and the whole time these optical components are shrinking. They're going from 300 by 200 nanometers, eventually at the X ray kind of scale, to, you know, a few nanometers kind of dimensions. You could do that. It's possible. But it turns out that the light sources that you need to follow that path would be pretty tricky. And when you build optical wires, they care a lot about the quality that you build those wires with. The light wants those wires to be very smooth. So if you're not, you know, really, really careful about the quality of those wires as you shrink it, you're going to have a hard time. And luckily, it turns out we don't need to shrink them at all.

Well, you just took the next question right out of my mouth on that one, so keep going, please.

Yeah. So it turns out that in a traditional computer, if I said that your CPU is clocked at 3 gigahertz, you'd be like, yeah, that totally makes sense. If I say 20 gigahertz, you're like, I've never heard of that. And it probably will never be a thing, because I remember, like, when I was in undergrad, like 2005, I had a 3 GHz processor then, if I recall correctly. So, like, that frequency hasn't been scaling with optics. The frequency you can operate at is very high. So we work at about 1550 nanometers. That corresponds to a frequency of 193 terahertz. That's a lot of bandwidth.

Yeah.

Turns out that you will never practically get anywhere near that kind of bandwidth because you have to talk to the thing, you know, like, ultimately these things have to talk to electronic computers because that's sort of how the world works. And so you get limited. Maybe you can do 20 gigahertz, 50 gigahertz, something like this. But the point is that we're able to turn up the clock frequency to obviate the need for shrinking these things because we're going to give you more performance per unit area through the clock frequency. And there's one other thing. So, yeah, we can go fast in clock. We can do something else special, which is we can use multiple colors at exactly the same time. So if you remember, we have this 2D array that's doing matrix computation. Each one of the elements in the array can process different colors of light at the same time. So each, each element could do three colors at the same time, 16 colors at the same time, something like this. And what that means is that in the same area, that same array size, two colors of light, you've just doubled the compute in the unit area. Three colors of light, three times the compute per unit area. And you see, like, there's, there's really no need in principle for this sort of shrinkage of the photonic devices.

Well, as Chris knows, I'm always interested in the very practicalities of things. And I'm imagining, okay, we have this, this processor and it works and can do some computations. You have it integrated into some system. But ultimately, like, I have TensorFlow and I know how to run TensorFlow on a regular computer or on a GPU because there's underlying libraries which eventually get mapped into some type of machine code that, you know, is mapped onto, like, this processor. So maybe just describe, like, okay, you have this photonic computer. How do you even start to think about that process of integrating software into the system and, like, making sure you support, like, tooling that people want to use and that sort of thing?

Yeah, that's a huge amount of work. So building the photonic computer is really hard in terms of the physics and engineering, but building a software stack that integrates with PyTorch and TensorFlow is a ton of work. But it's something that we've bit off and we call it Idiom. So Idiom is lightmatter's software development kit. What we allow you to do is take neural networks that you've built in PyTorch or TensorFlow, import our libraries, and we have a compiler that can take the emissions From PyTorch and TensorFlow and build machine code that runs on Invis. Envise is our photonic computer. So we have our own instruction set architecture, and our compiler emits that. ISA is what people call it instruction set architecture. It's a ton of work. We have a pretty big software team here. I expect by the end of next year that we'll have significantly more software engineers than hardware Engineers, even though we're building these crazy chips, and generally that's the trend certainly in the machine learning space, but in computer chips in general, because people, they don't really care about computer hardware. They just care that it's fast and it keeps getting better and that it doesn't have errors and it's not annoying to use. So we're very focused on just delivering the same experience that you're used to with PyTorch and TensorFlow, and you can just use our alien technology and not worry about what's under the lid. But it's a very big effort.

So if you're thinking about that, just to clarify for a moment, if we're going to PyTorch or TensorFlow and we have your chip in the system and we have idiom there, it's basically taking that, converting it into something that works on your hardware in that context. But from our standpoint as data scientists or software developers or deep learning engineers, our workflow is more or less the same, if I'm understanding you correctly. Is that, is that accurate?

That's right. So right now we're targeting inference. So the way you'd imagine this working is you've got a neural network that you've trained in PyTorch or TensorFlow, and then we have a bridge to Onnx and then Onnx to our compiler and you're good to go.

That makes sense.

Yeah.

And so the chip can do training, but we're not focused on it right now. They're very, very different markets with very different things that you optimize for. Happy to talk about that. But inference and training are really drastically different beasts.

Yeah. Maybe you could touch on that a little bit, because I think maybe the, at least for people getting into deep learning, a lot of the tutorials and whatever they see is all about training. Right. So it becomes this perception that like training is the major problem to Sol, as a data scientist or an AI researcher at your day job, could you maybe like on some of your work with lightmatter, but also just working with clients and understanding their needs, like maybe speak into that in terms of why inferencing is like such a big market, but also really important for practical use cases of AI?

Yeah. So training is R and D mode. You're building a tool that takes inputs and then tells you something about those inputs. If that's all we ever did in machine learning, no companies would, would fund this stuff because the ultimate goal of building those models is that you want to use them for something and if you're Google or Facebook, these neural networks get deployed at massive scale. And their job is to take a user query, run a neural network on the query, give you results. And sometimes there are lots of neural networks in that chain, and that is where the overwhelming majority of the energy footprint of AI will be over time. Training is a really big deal. You gave the example, I think MIT Tech review did that analysis that you were quoting where it was like five cars over their whole lifetime is how much energy that the training was using. And I think the equivalence was in, like, carbon emissions. So, yeah, inference is deployment, which is where most of the scale is. And training is just doing the R and D that lets you do that thing that makes you money, because training stuff doesn't make you money as you go.

I actually want to go back for one second and touch on something to combine two things that you said. So as we're doing this and we get to, you know, as practitioners, keep the same workflow more or less that we're doing, and we are still able to use the tools that we're already using to be productive. We've now been kind of supercharged by having these photonic processors in their. I may not be using the right terminology on that, but. And we've talked about scaling without having that limitation from the thermal limitation that we've talked about does. To go back for just a second of curiosity, is your limiting factor then the receptor of the light inside there being able to detect the number of different light wave frequencies, the number of different colors that are available, so that in theory, if so long as. As you can build better and better receivers for more and more colors in there they can distinguish, then you're essentially unbounded by that constraint until you hit whatever that practical limit may eventually be.

Yeah, that's a great question. So what are the ultimate bounds on using color and frequency? Yeah, detectors are pretty broadband, so they can detect a large number of different colors at the same time. No problem there. But there's a device called a multiplexer or a demultiplexer. If you can think of the Pink Floyd album cover, you have a prism. Yeah. And there's white light that comes in and a rainbow that comes out the other side. That's identically what I'm talking about. So that specific kind of device that takes in a stream of multiple colors and spreads it out into the individual constituent colors, that device is what will limit you from getting to like 64 colors or 128 colors. Eventually it gets really Hard to accurately separate those colors. And then when you're at this point, scale out becomes really important. By the way, that's a long ways away in terms of the roadmap for photonic computing. When we're at 64 colors, the amount of compute on the chip is something like hundreds of PETA ops. It's crazy. So like decadal time scales. But then when you get there, what's really important is that you can efficiently take lots of cores, lots of processors cores, and connect them to each other. Earlier I was talking about Omdahl's law and the fact that you add another unit of compute and unfortunately because of communications, you don't get another unit of throughput. We've invented an interconnect technology that, surprise, surprise, uses light. It's a wafer scale computer chip, 8 inch by 8 inch. So it's the size of your laptop screen practically. And it allows you to let your processors talk optically and they can dynamically configure how they're connected. You remember those old switch rooms where people are making phone calls and they need to connect to some other. So imagine you have someone doing that, but at the microsecond timescale, they're able to unplug the optical waveguides built into this computer chip and plug them into different locations at crazy high speeds. And that allows you to scale it. So whenever you run out of colors, scale out is going to be really important. And we've invented a technology for that and it's called passage.

That's really cool. I love that analogy. And I love the fact that you're also thinking about like scale out, not in terms of like always making things smaller and pushing those boundaries, but doing it in other creative ways as well. So if I am, if I'm understanding right, you kind of have the chip. So in terms of like what you're bringing to market and what you're doing, you have the sort of chip or processor, you have this passage technology and then the idiom software integration. I'm wondering if you could just give a few or highlight a few sort of use cases or tests that you've done with these technologies in terms of maybe models that we might be aware of about inference time or what sort of differences are you seeing and how is that impacting actual inferencing?

Yeah, so comparing our technology to competitors. So on the idiom side, it's a software stack and the goal is really just to, to make sure that it doesn't slow down the hardware compared to what it's theoretically capable of. So we'll leave that on the table. On the Envise side, on our website you can see a comparison between Envise server and Nvidia's DGX A100 server. We use a very small fraction of the power, so that chip's quoted at about 80 watts compared to Nvidia's 450 watts and pretty significant boost on Resnet 50Bert and then I believe we also looked at DLRM. I don't have the numbers off the top of my head, but in many cases it's multiple times faster. But what's extremely important from a scale out perspective is if you look at the power and just like if you think about it this way, we can up the performance. We're not anywhere near that power envelope, so we can keep going up there. There's a point though, and this is why we invented Passage, where making Invise faster is great, but it's just sitting there waiting for work. It's sort of like this ultra capable person and you like hand them a paper once a day and they're just poured out of their mind. So we really had to invent Passage as a way to keep Invise busy. So that should give you an idea.

It does. I know this is day to day stuff for you, but this is quite remarkable for those of us not in the field. Where do you see this going? I mean, you know, is this what everything inevitably has to go to? Do you think that the field will stay with lots of different technologies that are competing? What does the future look like to you? How should the rest of us be kind of thinking about the years forward in our career versus where we're at now, how much change is coming?

So what we're doing is targeted at AI for general purpose computing where you're running Windows and you're playing a video game. Probably photonic computing will not be able to contribute meaningfully. There are some really hard challenges in running general programs. So to be Turing complete you need to have these sort of nonlinear operations. If you're a programmer, you can think about branching like this ability to have conditionality if and then doing these nonlinear conditional type behaviors with optics is super hard. And so you mentioned a future where potentially there are lots of competing technologies. I think that's exactly what will happen happen. We've had this incredible platform with transistors and Moore's Law scaling that's been so general purpose. We use it for everything. What you're going to see going forward is you'll have photonic computers for AI, and our plan is to help them dominate that field because we think that they're very well suited to the mathematics that underlies deep learning. And I think you're going to see quantum computers. I love quantum computing. I did my PhD in quantum computing, but I'm not as bullish on the right now timeline. I think they've got quite a while left, but I'm rooting for everybody working on it. And in general, I just think it's the case that we've reached the limit of transistors being able to kind of do everything good enough. And you're going to see a bunch of different types of technology platforms all competing. You'll see analog electronics at the edge. You'll see digital electronics for running your games. You'll see photonic compute units for doing your deep learning. Like my goal is to have all of Google run on Light Matters processors. So when you say, hey, Google, that goes through envias, that's where I want to get to. So you can kind of see this is how I see it. These technologies are all just suited to different kinds of problems.

We're really looking forward to seeing how you bring this technology to market. It's incredibly impressive. Sort of blown away by the amount of work that's gone into this and just the amazing thought process that goes along with it. We'll include links that we've talked about in our show notes for everyone listening out there. Definitely check out Light Matter, see what they're doing. There's links that we'll post there with some of these benchmarks and other things. But Nick, it's been a real pleasure to talk to you. It's just amazing stuff that you're doing and really looking forward to maybe talking next year and seeing how far you got and how many invised processors are running in Google.

Awesome. Well, thanks for having me, guys.

That's our show. Thanks for listening. For more like this, check out our Master feed. It is all changelog podcasts in one easy to consume place. Let your podcast app snag everything we produce and then pick and choose which ones to listen to. Subscribe today@changelog.com master or just search for changelog master in your podcast app of choice. You'll find it. Special thanks to Breakmaster Cylinder for providing our music and to our longtime sponsors Fastly LaunchDarkly and Linode. That's all for this week. We'll talk to you again next time.

Sam.