Nvidia's Acquisition of Mellanox, Plan for the Future

Hi, this is the Baseless Speculation Podcast with Spartacus and Dylan, and today we're going to talk a little bit about Nvidia. One of the interesting things that has happened to Nvidia recently is they've announced that they've acquired a networking firm called Mellanox for almost $7 billion. Given what Mellanox does Some have been a little bit uncertain as to what this new acquisition will do for graphics-focused NVIDIA. However, we think we have a pretty good idea on how they'll use it. Dylan, do you want to talk a little bit about that?

Yeah. So if we look back into the past, we look at what kinds of acquisitions NVIDIA has done in the past. They've acquired 3 companies of noteworthiness. One was PortalPlayer. And so PortalPlayer used to make application processors for iPods, and NVIDIA acquired them, and then Apple switched to Samsung, you know, a little bit afterwards. So not too successful, but it made sense where they were trying to go. The next thing they acquired was Agea, the PhysX company, and that was in response to Intel acquiring Havok. And NVIDIA's whole thing there was they wanted to have GPU physics engines. And while PhysX was successful, GPU PhysX was not. The third company that Nvidia acquired was Icara. Icara, I'm not sure. But they produced modems. And that made perfect sense along with Portal Player in terms of their whole Tegra SoC phone ambitions. And they continued the development for a while, but eventually they laid off everyone. in that department because it just wasn't successful.

You mentioned all those acquisitions, uh, all the major acquisitions of NVIDIA in a pretty brief amount of time. You can really summarize those because a lot of the links to what NVIDIA was doing or trying to do at that time were pretty clear. Um, so one of the things that we wanted to talk about was how the usefulness of Mellanox is going to be a little bit more involved. There's a little bit more to, to unpack. And so Dylan, could you go through and just start by rolling through what, what does Mellanox do, and then maybe move into how NVIDIA will actually be using them?

Yeah, so Mellanox makes networking solutions. So they're really popularly known for InfiniBand, which is networking between nodes of servers. The other thing they do is make Ethernet NICs, network integrated controllers, the high-end ones. And so their roadmap for those is quite interesting though. They currently just make normal NICs, but soon they're gonna start making smart NICs, if you will. And you didn't see me doing the quotation marks, but the reason they call them smart NICs is because they're going to begin integrating ARM CPUs onto them. And the whole purpose of that is the intelligent routing of data. One of the things they've talked about a lot is the NVMe of, which is basically fabric storage. You have a bunch of NVMe drives connected to the network and the SmartNIC helps request that data from that storage, that pool of NVMe that's connected to the storage in a distributed way. That's similar to what I think NVIDIA wants to do with Mellanox. They want to create the GPU of basically a fabric GPU accelerator network. So recently the OCP created the OAM. So that's 2 acronyms, but the important one is OAM, which is Open Accelerator Module. And so if you've seen the Volta chips out there, they have this the sort of, you know, they have the PCI form factor, then they have this other form factor. The Open Accelerator Module is that other mezzanine-style form factor. With the Open Accelerator Module, which is supported by Baidu, Microsoft, and most importantly Facebook, the biggest contributor to this module, you can do 2 main things. You can scale and you can decrease latency. And this is done because the Open Accelerator Module connects into the network, the fabric, better than past designs of accelerator cards. In the past, accelerators have had to go through the host, aka the server CPU, to get to the network integrated controller, which then attaches you to the fabric. What the Open Accelerator Module allows you to do is it takes the accelerator and it actually connects the accelerator to the host, to the other accelerators, and then also the network. So currently NVIDIA has the NVSwitch, which connects the accelerator to other accelerators. What we think Mellanox is going to allow them to do is do that third piece, the expansion piece, connect the accelerator to the network. And this, like I said, lets you do the 2 main things, scalability and latency. With scalability, NVIDIA can take their huge 8, 16 GPU designs, their systems, and scale them even higher. Instead of having 8 GPUs connected or 16 GPUs and a couple CPUs or 1 CPU, you can now do hundreds of GPUs connected together in this fabric. And Mellanox's technology of SmartNICs is Vital for routing data around between these various GPUs. The other thing is latency. So scalability dealt with training. With latency, you're talking about inferencing. With inferencing, latency is critical. You have to have lower latency. And that's why most latency is still done on CPUs or FPGAs. NVIDIA wants GPUs to take that crown as well, like they have the crown with training. And the The way they wanna do that, I think, is by bypassing the CPU. So instead of having the data come in from the network integrated controller, you know, from the fabric, go to the CPU, and then go to the PCIe device or the mezzanine accelerator, it can instead go from the network integrated controller straight to the accelerator. And then the accelerator does the processing and sends it back out to the network.

And then another key thing to mention on all of this is that While there's a lot of neat things that happen once you kind of bypass the host, the other thing that's important to NVIDIA is that host generally is a CPU, you know, made by Intel and, you know, some other companies, but not NVIDIA. NVIDIA doesn't really have a CPU that can operate in this environment in the same way that, you know, Intel or some other others do. So by solving this problem in this manner, it kind of overcomes a limitation that NVIDIA has. So that's, that's another really interesting thing to mention with how this whole Mellanox arrangement could work. Now we talked a lot about Mellanox, and that's touching on that network— networking aspect. We can't help but talk just a little bit about what NVIDIA does on the actual accelerators, so the actual processing of data. And so the rumors are suggesting that that's going to be pushed forward with a next-gen Volta successor called Ampere. So Volta was announced basically 2 years ago at around GTC 2017. And so we're approaching GTC 2019 and it's about time to see Ampere at some point here. We anticipate that the first thing that we're gonna see with respect to Ampere is gonna be that GA100 type of part. So it's gonna be— maybe we might call that the big Ampere. It might be somewhere around 500, 600 millimeters squared, pretty, pretty big for a 7-nanometer part. Based on NVIDIA's guidance, you know, we expect probably more than 32 gigs of HBM2 and, you know, quite a bit of processing power with, you know, next-gen NVLink, all that jazz. It's going to be pretty interesting. One of the really neat things that was noted by NVIDIA was that this guy's actually going to be using HBM2. Based on most recent guidance. And if you end up doing the math on HBM2, you only get so many different options if you kind of hold NVIDIA to somewhere around, you know, 4 or 6 HBM2 stacks, somewhere in there, and look at the currently available, you know, roughly 2 to 2.4 gigabit per second data rates. And it's going to be interesting. It's going to be somewhere around, you know, 1 terabyte per second up to not quite 2 terabytes per second. There's a little bit of variability in there. We're not quite sure what exactly that's going to be looking like, but it's certainly getting close to the point where we could almost see them moving to more than 4 stacks of HBM2. Depending on how bandwidth hungry this new processor ends up being. And speaking of new processors, one of the things that we're hearing some rumors about are an entirely new GPU just for inferencing. And so we're hearing that this might get attached to kind of a GA101 type GPU rather than a GA100. So the 101 code name really hasn't been used effectively at all, or at least in recent history, by NVIDIA. So this would be a new GPU in a different space. So if it looks like other modern inferencing options, then it would be relatively low power. So, you know, it could be somewhere around 75 watts if they're really pushing for a high-density option like a lot of the other things from, you know, Microsoft and Google and other inferencing FPGAs and all that good stuff. You know, right now What Nvidia is doing is they've got their 104-type part, like the TU104 is in the Tesla T4. You know, they take a GPU that really lives in that 200-odd watt space and they clock it down so it can live in that 75-watt space, and suddenly it gets to to be a lot more efficient and that improves density quite a bit. I'm sure NVIDIA probably doesn't like that. That's not quite the area that they want to be doing. It's a big GPU to be clocking down that much. So we're really interested to see what this kind of thing could look like. It's probably going to be cutting out A lot of the FP64 hardware, probably some of the FP32 hardware as well, if it's really going hardcore on just inferencing. So it's going to be a lot of those Tensor Units, and we're sure it's going to almost certainly be ready to harness a lot of the networking options that Dylan was mentioning. It's going to be pretty fascinating. That's for sure. And then again, we, we just love GPUs over here, and so we can't really stop this podcast without talking about one more thing. So Dylan, do you want to talk a little bit about Orin?

Yeah, so this definitely kind of doesn't fall into the whole Mellanox acquisition, but NVIDIA is almost sure to announce Orin, Orin being the successor to Xavier. And Orin should be about 5 times the performance of Xavier based on some slides and roadmaps that NVIDIA has given out in the past. Orin being 5x of Xavier is kind of hard to imagine if they keep it in the same power envelope. And it's very likely that they'll actually increase the power envelope. Xavier was, you know, 30 watts, and that was a huge increase from prior. And prior Tegra processors, you know, they slowly just evolved from being phone to only tablet to now only 30 watts. And I think they're going to continue this trend and Orin will likely be 100 to 150 watts. The way I arrive at that is, say, it's pretty crude. You know, they're taking Xavier, which is 12 nanometer, 16 nanometer, whatever you want to call it. It's okay. You take their shrink. Okay, that's 2x efficiency. So now you're at 15 watts for Xavier performance. You give NVIDIA credit for sizable architecture change too, but you're still nowhere near where you need to be to get that 5x performance. So the only way is to scale it up even further. And since NVIDIA thinks that the best way to do these is to connect 2 of them, it's probably connected via that NVLink 3 that we're going to see on these other Ampere GPUs. If you look at a block diagram of Xavier, it's pretty clear that this thing is going to be even more full of application-specific integrated circuits. Right now, half the die is application-specific integrated circuits. We'll see what Ampere with Xavier brings, I mean, with Orin brings. On the CPU side of things, I'm expecting them to retain that 10-wide on the VLIW CPU side, but I expect the ARM decoder to go wider, probably 3, maybe 4 wide in terms of ARM decoding. And that's about all I have.

Thanks, Dylan. As you can see, we're pretty excited for GTC, and we look forward to seeing what happens. Thanks, guys.