Exploring NVIDIA's Ampere & the A100 GPU

So now we have this new Nvidia DGX A100.

Maybe you'll get one. I don't know if I'm going to get one, but we'll see. We'll see. Yeah, yeah.

Sadly, I'm not in charge of procurement and I am certainly not in charge of procuring one for my own personal use. That we'll see.

I'm guessing that my nonprofit's not going to get one, but if you happen to be getting your DGX A100 and you'd like me to run my training on it, I would be more than happy to do some benchmarking for you.

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at changelog because of rollbar. Check them out@rollbar.com and we're hosted on linode cloud servers. Head to linode.com changelog linode makes cloud computing simple, affordable and accessible. Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support and scale you need to take your ideas to the next level. We trust Lino because they keep it fast and simple. Check them out@lino.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're at PracticalAI FM. Okay, take take it away guys.

Welcome to another Fully Connected episode where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up your machine learning game. So welcome to the Practical AI Podcast. My name is Chris Benson, I'm Principal AI Strategist at Lockheed Martin and with me, as always, is Daniel Whitenack, Data Scientist at SIL International. How's it going Daniel?

It's going great. It's a beautiful day outside. Hopefully I can take a walk after this. And I've been staring at my screen most of the day since like 7am so I'm ready for a walk around the block or something.

That sounds like a good idea. I am bleary eyed from screen time as well. So get outside and enjoy it. Especially now that at least where I'm at in Atlanta, the worst of the pollen seems to be past, so that's good. No more yellow cars from pine pollen everywhere.

Yeah, that's rough. We need, like, an AI model that, like, takes in pictures of cars in people's driveways and, like, tells you whether it's safe to go outside yet because of the pollen levels or something.

There you go.

I'm sure there's easier ways to do that, but.

That's right. So we'll have to prep people next year. So we'll just ask everybody in the audience next year to send us your images of your cars covered in pine pollen with a date. A date time attached to it. And we will have a great project to go on this.

Yeah, exactly.

So. And before we get going, I know that you've been doing some training classes. How have those been going?

They went. They went really good. It was interesting teaching, by the way.

Not. Not taking.

Yeah. So normally when I do, like, AI trainings in industry or, like, at conferences or something, obviously there's, you know, normally, like, a whiteboard there. There's a lot of kind of changes as the class goes on because there's a lot of easy kind of back and forth. So it was interesting to figure out the virtual dynamic with everybody being at home. I think actually, it ended up having some benefits because it sort of forced me. Like, normally I kind of write things at the whiteboard. I'm able to, like, make changes as I go. But to be able to. Or to have to sit down and work things out in a strict set of slides that I'm showing made me really think about, like, what is the proper flow to explain this idea and, like, show certain things. So in that respect, actually, it was. It was a learning experience for me, I'll bet, because it helped clarify some of that logic and even in my own mind. So. But it went really well, and the students had some great questions and went, you know, all the way from what is AI? To convolutional layers and recurrent layers and training. Cool things. And so, yeah, it was a good time.

You know, I've always found that no matter what the topic you teach, it forces you to assess everything that you think about, because you got to explain it to other people and answer all that. And you all. Every time I've done that, I've learned so much more about whatever it was I was going to teach. So.

For sure. Yeah. And people ask questions that just never have come into your mind, not because they're bad questions. It's just a different point of view. So it forces you to, like, backtrack yourself and Kind of look at things from different directions.

Yeah, well that sounds interesting. I guess you probably looked out, you got through your training classes right before Nvidia this year at their GPU Technology conference, GTC made all their new hardware announcements and the things that go with that. So.

And you know, it's interesting, I actually was spending the evening yesterday with my brother in laws who are living with us. So we aren't like social gathering yet, but my brother in laws are living with us right now while they're back from college. And so they don't work, they're not like in computer science or anything like that. But they are pretty heavy gamers and they were both like we started talking about some of this stuff and they had even seen the keynote from the Nvidia conference even before I started talking about. Even though it was mostly AI related. It was like a, you know, already a sort of general meme that like the Nvidia CEO was like presenting all of this cool GPU stuff from his kitchen, which was like, you know, you could see the spatulas in the background and some very interesting like fresco above his oven and such. So it was really interesting in that sense that there was like, like people, even people that aren't in this space, like it made some impact on their life, which was really interesting.

You know, that's a great point. It's worth talking about. I mean, you know, we talk about Nvidia and Google and other major players in this space often because you can't really talk about AI in a lot of cases without talking about the biggest influencers. In Nvidia's case, they were this gaming company and GPUs were originally to promote graphics and you know, computer gamers, you know, were constantly using it. And now we're doing this in the AI space, you know. So any thoughts on how or insight into how that evolution came about and why we're using GPUs for all this stuff?

Yeah, and actually also demand from the bitcoin mining, that's true space as well. So it's very interesting that like if you look at Nvidia's rise over time, of course they exist for quite some time, but it's almost like so they existed for a reason and they were really good at this reason. And then all of a sudden like the things that they were good at became like the most important things in the world. That's how kind of how it seems. And then they just like they were already there. So it's like they just exploded. So yeah, you're right. So if you think about like video gaming and that sort of thing, or like things you would want to do in video processing or graphics, for example, like you might want to apply a filter to some image or frame of a video, right? Like to darken it or to apply a gradient of color or you know, something like this. And so you're essentially applying some operation to the pixels of an image which are set up in a matrix and have like even some depth because there's a color dimension, right? So you have like this matrix of numbers and then you apply some operation on the elements of this matrix or really this volume, this input volume. In AI, of course, with convolutional layers, like you are doing almost that exact same thing because you're applying like a series of weights and bias and you know, functions like activation functions to individual elements of a matrix or an input volume. But even in recurrent layers or fully connected neural networks and that sort of thing, the types of networks that might be relevant to other things like text or just general classification problem, even those, take some input vector or matrix and just apply a series of weights to those. Apply functions like activation functions like tangent and sigmoid and all of these in an element wise way. And so you're really doing the sort of matrix operations that graphics cards were always good at. And so it turns out that it's really good to use those sorts of graphics cards for those sorts of operations which are done in, specifically in AI training. Of course we're going to talk maybe about inference today too. But I think it came about because these are the sorts of things that happen iteratively thousands and thousands and millions of times when you do training for an AI model.

You know, that has to be the most accessible explanation to that evolution that I think I've heard anyone say. I think, I think you did it better than Nvidia actually says it. So that was well done.

Well, they can pay me if they like. I mean, or they could send me a graphics card that would probably actually be better. Nvidia, if you're listening, hint, hint. You know, Titan rtx, I won't even take the newest one. Don't even give me like a, a 100.

Give them the newest one.

I don't. Just small potatoes. Yeah, the, that $5,000 or what? I don't even know how much it is. Titan rtx.

Well, now that we have been pleading.

For free stuff.

Let'S move on to some of the things that they announced, which many organizations around the world are going to, to evaluate and figure out how they're going to incorporate, buy into and basically utilize this new hardware and the supporting software capabilities that go with it.

Yeah, definitely.

So, you know, I guess one of the things to talk about here is even before we get to announcements are the types of GPUs that are currently available in what forms, you know, what kind of off brand GPUs are out there. Because Nvidia isn't the only player in the space. Any insight into that?

Yeah, I mean it's probably worth distinguishing a few things here I guess. First is like accelerators that are out there and types of GPUs that are out there and also like access patterns to those, whether that be like locally or in the cloud or whatever. At least from my perspective. And I am, I'm by no means an expert on this. On the like graphics card front, actually probably my, my brother in law could do a better job. But there has been a progression and most of the time you'll see like graphics cards referred to by some series of numbers and acronyms. So like recent ones have been like something like 1080 RTX or Titan RTX or something. So those are the graphics processing unit that you would like buy and then you have to plug it into like some computer, right? So some people say like okay, I'm going to, I'm going to develop AI models and so I'm going to buy a, like a computer, like a tower, a desktop computer and then I'm going to buy a graphics GPU, like one of these RTX GPUs or something and I'm going to put it in like my PCI slot and my motherboard. And then when I do AI training then I'm going to like offload the training, some of those training operations to the graphics card or GPU that's input to my computer. So that's sort of one of the first ways you might think about doing this is like I'm going to do AI development so I can buy a computer and then I'm going to buy a graphics card and just put it in there. And a lot of those of course come from Nvidia, they make a lot of those cards. But there's off brand ones that kind of are similar to the models that Nvidia has. And then there's also like other brands that have their own style of graphics card and that sort of thing. Have you ever built or thought about building this sort of like workstation for your home to like sit by your desk or Something I think I'm way.

Too lazy to do that at this point. I'd much rather go to a cloud provider if I'm at home and. And use what they've built. I've noticed that most of the people that had workstations specifically for their AI workflows seem to have moved off those in recent years, either in the cloud or if they're big enough, into more of a data center or at least a, you know, like a workstation level, you know, where they're buying a workstation versus buying individual GPUs. It was one of the things I was thinking as you were just talking about that was we had one of our early episodes, which was episode 15, called Artificial Intelligence at Nvidia. We had Nvidia's chief scientist, Bill Dalley on the show and he absolutely schooled us. Yeah, definitely in the hardware arc. Do you remember that? He really schooled us in the hardware.

Much deeper than we'll go on this episode. So definitely take a look at that.

Yeah. Yes. If you're wanting. And we asked him to. Against other architectures, you know, and he went there and described it. So if that aspect of it, not just the Nvidia architectures, but how they compare to other things, I would encourage listeners to listen to that episode and he will absolutely school you in the fundamentals there.

Yeah, for sure. Because there's not only. So like I describe like the graphics card or gpu, which is what a lot of times what people think of when they think of GPU or. Or accelerator in the AI world, they think of one of these GPUs or a series of them. But there's other options too. So there's like the TPU or Tensor Processing Unit from. Is it Tensor processing unit or TensorFlow process? I don't know if they put the brand in there. I think it's Tensor Processing Unit. I believe it is the TPU from that Google developed, which is another type of accelerator. But there's even other architectures out there other than cpu, gpu, tpu, there's FPGA and other things. And yeah, there's a lot of options out there. And like you said, there's options also. So I kind of described like, okay, if you're developing AI, you could just like create. Just buy one of these computers to have at your desk. But there's also just like how other forms of compute have been commoditized via the cloud. There's easy access to cloud resources for GPUs, too. And in all the Clouds and in even special built GPU like cloud services like Paper Space and others. I know when I was looking around a while back for a project, I don't know if it's still the case, but I was trying to find like what is the cheapest way to use a GPU in the cloud and I ended up going with Paperspace. I don't know if it's the cheapest anymore. I do use like Google Colab, as I've mentioned a lot of times on the, on the podcast. And of course you can have access to a free GPU there. There's trade offs because it's in a notebook and that sort of thing. But anyway there's a lot of ways to access them which aren't buying a computer and setting it on your desk.

That's true.

That's definitely nice.

So why don't we dive into some of the announcements that Nvidia made at GTC as we're recording this. I think it was about roughly a week ago that they made the announcement and it'll be another week as it rolls out. But I'll start us off with they started and I'm going to probably butcher the pronunciation. The Nvidia ampere architecture. Did I get that right? I've read it, but I haven't watched the video to see ampere. There you go. How he's pronouncing it, I think in.

Reference to like amp in electronics, I think.

There you go. I didn't make the connection there. Okay.

So I don't know, at least that's how I was saying it.

Yeah, I know that essentially this is what they've used to replace kind of the existing architecture and expand it. They're really focusing on, I think a more realistic in the sense of kind of I say cloud, but when I say cloud, I don't necessarily strictly mean cloud providers. I mean if you were putting together a data center with a whole bunch of GPUs or GPU servers in it, you know, they're really focusing on not only the performance sides but the usability. As I was reading through it.

Yeah. And I think that what I was gathering also in talking with some other people about this is. So the generation before this latest one was focused more on the ray tracing elements, which is the RTX in a lot of these cards, which to be honest, I'm. I'm not a big expert on ray tracing nor am I. That has implications, of course in graphics and that sort of thing. But it wasn't like a huge advance in Terms of like the size and capabilities of the graphics processing unit itself, it was more of this kind of generation of additional ray tracing capabilities. Whereas this next architecture which they're releasing, which they're calling the A100 or the Ampere architecture, which includes this A100 card or GPU, that this, this is a fairly significant jump in the like size and capabilities of the graphics processing unit itself. I think part of that has to do with I guess the way that they've laid out the transistors and all of that on the substrate, that it's much more dense in my understanding. Yeah.

Am I recalling it was something like 20 times performance improvement over the V100?

Well, yeah, so it's 20 times greater flops, which is like a measure of. Actually you probably are better versed in the acronyms, but this is like a common way to like measure the performance of like computers like supercomputers and that sort of thing. So 20x greater flops for AI. Although they do give some benchmarks, which is pretty nice. Just for reference and what I was looking at, they give some benchmarks for training BERT large scale language models, which we have an episode on BERT as well, if you'd like to learn more about that.

Yes, we do. We've mentioned it in several. Actually.

Yeah, it's good that we ended up having that conversation. But the BERT models are these very large language related models, NLP models that have just tons of BERT parameters. And actually these large language models have even billions of parameters. Now I forget how many BERT has, but they give some benchmarks both for the training and inference on speedups on training Bert. So on Bert itself they're saying that above the V100, so the V100, if you go to like Google Cloud or if you go to Paperspace or one of these platforms, at least right now I think the best GPU that you can get access to is called a V100, which is a previous generation and it's pretty wicked fast. I mean I've used this in a couple of projects and it's quite astoundingly faster than the sort of entry level gpu.

Yes. And it's the basis for the DGX line of servers as well.

I think so.

Or was prior to this release.

Prior to this.

Yeah.

And they're saying that there's a speed up between three and six times in the training for the BERT large scale training. And the difference between the 3-6x has to do with the precision of the floating point numbers that you're using in the model. So I'm stepping way beyond my bounds into like computer science land where I don't deserve to step. But, but in the, in the models obviously you have all these weights and parameters and the matrices that you're transforming in these models and computers work with numbers and those numbers have to be represented. Right. In some form, in some precision. You can't, like if you have, if you're representing PI, you're not going to represent like all digits of PI in PI. You're going to have to cut it off somewhere, right?

Yeah.

This is having to do with that precision of the numbers and you can actually, if you reduce the precision of how you represent numbers, you can sometimes speed up your, your performance. And so that's what they're talking about there with that difference to 3 to 3x to 3 to 6x.

Yeah. And I think I'm looking at their inference and I know they're, they're saying it's a seven times speed up on inference. So it's substantial in that case.

So they say like this card, this a 100 accelerator. They bring up this idea of what do they call it? The, I don't know if they're, if you say it mig. I'm thinking of like the fighter jet, but the multi instance gpu, which is a really intriguing idea.

Do you work for us now in that way?

Yeah. They're saying it's multi instance gpu, which in my understanding are they saying like you can basically treat the GPU as seven GPUs, is that what they're saying?

So I was wondering that myself and so a big topic that I spend my time at work is around multi tenancy in your workflows and you know, the accessibility of compute in those. And I was taking it in that way, but I'm not sure because they're a little bit ambiguous in the, in the way they use some of the terms. Another one that I noticed is they, they talked about the need for no code changes. I'm, and I'm assuming that's CUDA code changes in this case, but they weren't always as specific as they might have been in terms of their explanations here.

I was wondering about that as well. Of course there certainly are ways to make changes like this transparent, but there's a change somewhere. Right. It's just maybe at the abstraction level you're working with in TensorFlow or something, you don't have to make a change in TensorFlow, but in the underlying libraries somewhere. It seems like they're there's some type of change.

Yeah, it talks about on the multi instance gpu, as I'm looking through that, it's talking about seven different isolated GPU instances running different applications simultaneously.

Yeah. So it seems like when they say 7x speed up for Bert large inference and they have under there in parentheses 7 mig or 7 multi instance GPU.

They'Re using them all.

What I'm assuming that is meaning is they basically are running seven inferences in parallel on the seven gpu, which seems to be the same performance that they're indicating for as the v100. So for inference wise, it seems like, you know, the change is that you're able to run things in this parallel way, whereas like on a V100 or something, maybe you couldn't do that. And so there wasn't that, that speed up. I'm making some assumptions here.

That's true. I know for a fact that there are folks at Nvidia that listen to the podcast. So hopefully if we are getting this wrong, they can.

Yeah, clear us up.

Yeah, they can clarify for us and we'll come back at a later time on a later episode and say we were wrong. We're happy to do that. So we're making the best of it.

We were wrong. And if you send us a GPU then we'll prove that we were wrong on our own local system.

You're back to begging.

It seems pretty cool. I mean, I like the idea that if you've gone from a stage of training to inference, basically, whereas before maybe you had this full like powerful GPU that you were basically running inference on, but not like soaking up all of the goodness of the GPU and the computer here they're basically saying, okay, well you can sort of parallelize the inference over that and still utilize this whole compute capability, but now you just have this ability to split it up in nice ways. So I definitely think that's pretty cool.

And yeah, it's interesting with the parallelization of this, there was an image that I saw Nvidia had put out where they were kind of comparing the old architecture with the new A100 architecture. And they basically had, you know, one little server for the new. That was the equivalent. They were showing, you know, rows of racks of servers in terms of its productivity. But it was definitely an impact. It was something that me and some folks I work with were passing around. And so yeah, gotta, gotta keep up with times, I guess if you're gonna keep driving forward on compute anything else on the architecture at large before we talk about DGXs or dive into the processors themselves.

I think the one thing that you mentioned about like the speed up without code change, I think they do introduce this new idea where as people before had talked about floating point 16 and 32 numbers.

Yeah.

Where again these are having to do with the sort of precision with which you're representing numbers. They introduced this new idea of like TensorFlow32.

I saw that.

Which apparently with float 32, obviously if you have more digits you can represent more numbers. Right. There's like this kind of range but you know, it's not as fast as using floating point 16 in some cases. So what they're saying is they're trying to balance the two, I think in that they have a wider range of numbers they can represent in this representation, but with lower precision such that they can, you know, speed up training. So, you know, again, hopefully I've represented that well in terms of how they're thinking about it. There's an image of this on a blog post that we'll link in our show notes if you want to kind of understand how the Floating Point 1632 and TensorFlow32 compare. But this is definitely a new representation on this chip that I don't think has happened on any other, other architecture yet. So that might be worth pointing out.

Yeah, totally. Another thing that we probably should mention from the architecture is that they've gone to the new third generation for NV link and NV switch.

Oh yeah.

And that manages the network scaling of how you're moving, you know, your data around through the chips and stuff. And I think that it's something like a 10 times bandwidth if I recall in terms of what it can do compared to. Or it may have been 10 times more than PCIe generation 4. I think that was what it was that I was recalling reading I'm going.

To get the number wrong. But they said there was like so many terabytes per some insanely small time. So it was like a bunch of data you could transfer back and forth very, very quickly via these links.

Absolutely.

So the NV link that has to do with communication of data between GPUs, is that the idea? That's what I've always assumed. I don't have the opportunity too often to run on like run my training on like 32 GPUs. So this is where I'm kind of getting to the edge of my understanding. But I did watch a YouTube video and I think that's what they implied.

Is that like staying at a Holiday.

Express Yeah, I've stayed at a Holiday Inn Express. Exactly.

There you go.

Yeah. So I've watched the YouTube video. My understanding was, like, because people also build these bitcoin mining rigs, right. And they have all these GPUs on top and they're running all the time. And the way they do that is they basically connect a bunch of them to PCI slots on a motherboard. And to do that, they have these little adapters called risers that, like, come out of the motherboard, but apparently those are very slow in terms of communication between the GPUs.

Yeah.

And PCI is slow in that way. And so at least that's what they're implying, that like, NVLink and some of these other things from Nvidia help facilitate that communication of data. And like, you're saying it helps scale out to, like. Now if you have 32 GPUs in your data center, you know, and you're trying to run some computation across them, you're going to need to have very quick communication for, like, scientific applications or AI applications that are not just bitcoin mining, which is just running operations, there's actually communication that's needed.

Yeah. If I recall correctly, and it's been a while since I've delved into this, back when they originally released the architecture, I believe that NVLink connects GPU to GPU. It gives you that interconnect between the two and then that, you know, essentially that mesh is something that NV switch then connects at a higher level, combining the different nvlinks too.

I see. So NV links GPU to GPU and NV switch, we'll call that now. But if listeners, if you know we're wrong, let us know and we'll put a note in the show. Notes or something that's good to make that connection.

Okay.

Yeah. And of course, these, because they're connectable and scalable in this way, it seems like this is their new way of replacing what they did have in the DGXs, which. The DGXs were the sort of boxes that they put in data centers, GPU data centers, to like, scale up. Like a AI supercomputer of some type.

Correct. Or a cluster of them.

Okay.

Which is becoming more and more common. In the earlier days, you know, people would get, like when the original DGX1 came out and it had eight GPUs in it, and people would get that, and that. That in itself people were calling a supercomputer. And, you know, we talked like that such a long time ago. It's only been a couple of years, but then they moved to DGX2 and then that was 16 and then they've actually scaled back and in just a moment, let's talk about that.

The changelog is deep discussions in and around the world of software and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

At the time, drones were like predators and global hawks and military industrial. And they were classified and super, you know, $10 billion things. And we had just built a drone with Lego pieces around the dining room table programmed by a nine year old and it's like, okay, that should not be possible. You know, when a nine year old can do something that is classified, that literally export as munition with Lego with toy pieces. Something important in this world has changed.

Leaders like Devin Zugel from GitHub in.

The like 10 to 15 year range or 20 year range. What I would really like is for if you have like three 12 year olds hanging out and one of them's like, I want to be a firefighter. Another one's like, I want to be a lawyer. I want one of them to say, I want to be an open source developer.

And innovators like Amel Hussein.

I've yet to kind of see applications at scale that don't use multiple languages, that don't have just arcane stories behind why this weirdo thing exists. You know, like, all right, when you open this file, you're gonna have to turn around three times and tap your nose once. Like it's just the most hilarious stories, you know. But applications are living, breathing, they have craft. That's not normal. So I want to normalize weirdness because that's just how applications evolve over time.

Welcome to the Changelog. Please listen to an episode from our catalog that interests you and subscribe today. We'd love to have you with us.

So now we have this new Nvidia DGX A100, which they've kind of broken the paradigm of their labeling. So they went from DGX1 originally to DGX2 and now they've gone to DGX A100.

Maybe you'll get one. I don't know if I'm going to get one, but we'll see. We'll see. Yeah, yeah.

Sadly, I'm not in charge of, of procurement and I am certainly not in charge of procuring one for my own personal use. That we'll see.

I'm guessing that my Nonprofit's not going to get one. But if you, you happen to be getting your DGX A100 and you'd like me to run my training on it, I would be more than happy to do some benchmarking for you.

Gotcha. I'll talk to my boss's boss's boss's boss.

Exactly.

See what's possible. Yeah, I'm there for you, my friend.

I'm really into the mooching today.

Yeah, totally. I got it.

I'm good.

We're there to support you. I'm there for you. But yeah, I mean with this new architecture it's much more performant, but they've actually cut the number of GPUs in the server back down to eight from 16. But it has the enhancements that we just talked about that are at the processor level architecturally.

So it's interesting that they kind of cut that down. But they have this multi instance GPU capability. So actually they say you can run 56 applications. GPU applications, seven per GPU times. Yeah. And like you were saying, even though there's fewer here because the size increase of a 100, they kind of showed this picture in the, in the keynote which people can watch, but supposedly you can kind of reduce the size and footprint of your data center because you're doing more computation per box per DGX than you were before.

Correct.

And this is interesting, they were saying like, you know, each box, let's say, I think the price they said was like a million dollars. Right. So this is not what I'm going to be putting on my desk, but certainly within the range of like compute budgets for some companies. So like each one was that expensive. But you could do the same that you could if you spent previously like $11 million on your data center. So like scaling wise you can do more with less, I think is the idea.

Yeah. When I was originally looking at these announcements as they came out, I think one of the call outs here and this architecture does start to address that. But I think people in organizations that can afford to get DGX systems and they do choose to invest in those, they underestimate what it takes to get productive with them. And so they kind of just think, ah, I can go buy a DGX and just everything's going to work out.

After that and then all my training will complete in three days and I'm done.

Exactly. Nothing to do it. But I think the challenge is when you're scaling up to one or more DGX systems, then you are Talking about an overall, I'm not just talking about a DGX architecture, you're talking about an overall systems and software architecture in your organization and specifically data architecture that can support moving a lot of data around through training in an organized way that flows in with your business processes. And that is a big challenge. And I think. And being able to make all that work in your own organization is where a lot of organizations are struggling. And I know Nvidia works hard to throw them a bone. They work hard, they recognize that. And there are a lot of tools that they put out there to try to help you through that process. But I think this architecture has kind of accounted for some of those pain points of the past. And they're trying to make it easier for, to utilize, you know, in number of GPUs across multiple DGXs, which is good because there are cases, there are very highly scaled cases where you might be doing a lot of experimentation with like hyper parameter optimization and you want to try just an insane number of different possibilities.

Yeah.

When you're doing your training and have the ability not just to train one time, but to train many, many, many times and you know, thousands or millions even.

Yeah.

And I think they've understood that and that this architecture is starting to address that highly skilled use case.

Yeah. I think that gets to the point of sort of, you know, maybe something that is on people's mind as they listen to this is like, why not just the cloud and like use GPUs in the cloud, which you can certainly do. So like you could, you know, if you wanted to run a thousand experiments to test all your hyper parameters, you could spin up a thousand GPU nodes in Google Cloud or Amazon or wherever. But if you're doing that at any sort of frequency or length, that's gonna, the bill's gonna add up pretty crazy fast on that. So if this is something that like a company actually wants to do and you know, AI is central to their strategy, to their products and they want to get that very best model and they want to do that experimentation over and over and over again. If this sort of DGX system is capable of supporting the usability side of things like you're talking about, then they could run those over and over again as much as they are able to, usability wise. And so I think that kind of gets to the point for some people. You know, I keep joking that I'd love to have access to this, but I probably wouldn't. You know, just me myself, since I'm the AI person doing a lot of the AI things on my team and I don't have a team of 40 different people trying to run things all the time, then you know, I'm pretty okay with using like a GPU instance in the cloud when I need it because I might run a training for 48 hours or even four days or something, but I do that not very often and it's just me. But if you got like a team of 40 people or you got multiple teams throughout your organization and they all need to run that stuff, that adds up really, really quick.

It does. I have been pleased at just in general, when you combine the advances in NVLink and Switch, when you combine that with the multi instance GPUs that these A1 hundreds are at this point the scalability technology which without diving into it is called Mellanox ConnectX Dash 6. If I'm saying that right, it's a nice blend of architectural considerations to get you there. And you know, we haven't even talked yet about advancements on the edge.

Yeah.

And that is a huge, huge area at this point.

I'm glad you bring that up because it's probably even though, you know, I may not get access to the sort of DGX system, I am thinking about various applications at the edge. And in fact I had a conversation earlier today with another guy who's working on totally different stuff in manufacturing. But they're not a large company, but they do stuff at the edge in the manufacturing setting with low power devices already. Like think like a Raspberry PI and that sort of thing. But if you could bring the power of like this sort of GPU to like right to the edge, to a machine where you're doing computer vision to detect like anomalies in your manufacturing process or something like that, that's a pretty major advantage. And that brings that sort of capability to those sorts of people that are working on smaller teams and have that specific use case for running AI at the edge they have the Nvidia. So along with the A100 they have the EGX A100 which they're releasing, which seems to benefit from some of these things that we talked about with the A100. But they also talk a lot about security, security and an end to end and encryption of AI models, encrypt all the things. And I have some ideas about why that may be important at the edge. But you have any thoughts on that?

Well, we live in a time where we've had so many episodes where we Talk about malicious actors. And they could be anywhere from state level all the way down to teenagers that are savvy and having some fun. And we're in a world now where you just can't really assume that you can put anything that's not secure out of the edge and that doesn't have to be in the defense world, you know, where I live, that can be really anywhere, any industry at this point. So they have had obviously their previous kind of edge oriented offerings and we like to, you know, they, you know, there's, there's the smaller scaled stuff that we like to play with. You know, they have the NX out now, they've had nanos out the last couple of years and so like that. But as industry really gets serious about pushing inference out to the edge and having it both widespread and pervasive, having kind of a comprehensive and sophisticated security model that they can deploy onto these platforms is pretty key. And I think that's really, at this point, it's no longer a specialty thing. It's now something we're all having to, to acknowledge.

Yeah, because like if you think about products, some products that have come out over the past years, like if you think about a drone that's kind of come out, I think there's multiple drones now that have come out that have some sort of AI model running on them that, you know, it does something like it follows you around or like whatever the thing is, it does some operation object detection or something. If you're thinking about releasing a product that has this sort of edge GPU running inside of it, whether that be in a manufacturing sense or like the drone or robot sensor something really the AI model that you're releasing with that is part of your ip, right? And you've spent hundreds of thousands of dollars into it. So you got like the malicious actor side of thing. But you've also got the fact that like, oh, if I buy a cool thing to strap on my manufacturing machine that has one of these GPUs in it and it's like doing something sophisticated. Well, if they're giving me the model in this product that I'm buying, why don't I just like unscrew the hatch and like plug, plug my computer in and just take the model off of it and now I don't have to pay them for that product anymore. Right. So we've gotten to a point where the actual AI model is, is a piece of IP and is and is extremely valuable. So you wouldn't want like your client or your, you Know, competitor especially to just be able to buy one of your products, unscrew the thing and like, you know, copy, you know, cp, my model PB from, from the machine over to their machine and then they've ridded themselves of their need for buying your product. Right. So, yeah, I was just gonna say.

It'S funny, I've noticed this a lot lately and that when we talk about the fact that you're now seeing models being deployed to the edge, you know, just in massively parallel, deeply pervasive in whatever your business is, you know, as you know, I have a daughter who's young, we just went through a birthday and the toys that you can buy these days are now incorporating this stuff in it is you can actually buy toys that have convolutional neural networks in them. You can buy them that have NLP capability. And I think that's the moment where I find myself surprised because we're so used to talking about it in these kind of business oriented contexts.

Yeah.

But then, you know, that's also someone else's business is to make these toys and. Yeah, and I, you know, I keep being surprised at these toys that she unwraps and they have these capabilities. Of all people, I should not be surprised, I suppose, but I am just to see it in that context.

Yeah. Well, and especially in that case, like, at least depending on the age of the child, you know, it would be important for that AI model to run offline on the device. And like, let's just keep that thing offline. And it's good if it acts as a toy, but let's not connect it to the wild west of the Internet just yet. So, yeah, I definitely see, you know, you'd want to run that sort of model at the edge itself and upload it to the device. I think the other thing I wanted to mention is, so I was going, you mentioned the Nano. So if people are thinking about like Raspberry PI and this gets down maybe where it does bring some accessibility to a lot of people. So there's like Raspberry PI devices which are like single board computer devices which have been of course course, wildly popular. But Nvidia released a Jetson Nano, which, which is like a single board computer with a little, a little GPU on it. And I was actually thinking about getting one of those, but I don't know if it was in this series of releases or just very recently they released this Xavier nx, which is like a next greater version of this. They actually call it like a little AI Supercomputer and it's, it is a single board computer. It's got like something like a 10x compute or something like that of the Nano. And so when I was going to get that one, I just ended up getting the other because it seemed pretty awesome. And I think that a couple of things struck me about this one is I'm always trying to think of, you know, like, for example, the, the cases that we work with and the people that work in our organization around the world of course work in a disconnected setting a lot of times because they're out in the field. And of course a lot of people around the world don't have Internet. But then also, you know, we're not flowing with money. So like what is a way to like get things running at the edge in a reasonable way, in a disconnected, offline way, but also at a cost effective way. And I find it really interesting that some of these things are coming out that have sort of a GPU capability and the Xavier and X, it's interesting that it's, it's got the GPU and you can run inference on it, but you can actually update your models as well. So they talk about doing transfer learning, which is like an update of a model. So you're redoing some of the training, maybe you're training some of the layers or you're training additional layers that you add onto your model. So I'm really curious when this comes in. I actually, it should be coming in today. So I'm kind of watching out my window right now.

Yeah, I'm going to reveal before we started the episode when Daniel and I were talking, he's waiting, he's watching out the window for UPS to show up.

Station right by my front window.

We might possibly get a package opening here on air with that unexpected.

Fingers crossed. Yeah. So I'm curious to kind of try. What I want to try actually is just to start with a small model and see like how the training compares to like in some like with a better GPU and paper space or something. And then like try it all the way up to like how far can I push the training on the, on the nx, can I actually train like how big of a model can I train from scratch on it and then like how big of a model can I do transfer learning on? Because yeah, I find that incredibly interesting. The other thing that they talk about with the cloud native, things at the edge and I know both you and I are really big fans of Docker, so I find it interesting whereas Before, I didn't see them emphasize a lot of things about using Docker at the Edge to run like AI related workflows. And now they're saying, well, this is how you should do it in this device. And I find that really interesting.

Yeah, not only Docker, but Kubernetes as well. It's, you know, I mean, and we've talked, I know we've talked about this on other episodes when we were hitting slightly different topics, but we really, this whole kind of AI revolution that's happened over the, especially if you're looking at the last three to five years, we really, really benefited from what had the revolution that had just swept through the software development world and software systems deployment world out there and that Docker and Kubernetes became the systems to build on. And we landed on top of that and just took that over. So it's really good to see all of the hardware, whether you're talking about, you know, the lower end GPUs such as, you know, the Nano that you talked about all the way up to, you know, the latest here, this, you know, DGX A100, all using that same architecture. And so if you learn it one place, you can use it from the most scaled down to the most scaled up version and you can use it in the data center and you can use it in the Edge. And that is a wonderful, wonderful thing that we've enabled. Inherited.

Yeah, I totally agree. I've really enjoyed talking about all of these things. I've got a lot to learn on all of these fronts. And if you're thinking like, oh, all of this GPU stuff and like, accelerated AI is very new, don't be afraid. There's still like, I didn't come from a sort of computer science background, but there is tooling that's accessible for you to get into some of these topics. And one learning resource. A lot of times in these fully connected episodes, we like to mention learning resources. So actually Nvidia themselves have what they call the, I think it's the Deep Learning Institute, Nvidia Deep Learning Institute. And they have a series of, of courses that talk about everything from like getting started with AI on the Jetson nano. That's that little single board guy that we were talking about all the way to, you know, more advanced topics with high performance computing, high performance computing with containers. They talk about various GPU accelerated frameworks like rapids and AI in the data center and all sorts of topics. So if you're interested in this sort of accelerated AI topic, you know, you might check that out. We'll definitely link it in the, in the show notes as well. I know I have a lot to learn there myself, so I'm going to.

Go slightly off topic, but I and it just occurred to me as we were talking about this, for the learning resource I'm going to throw out there, it's going to be one that a friend of mine mentioned just earlier today that he has utilized. For those of you who may be familiar with the learning site. Udemy u--e m-y.com There is a course on there called Docker and Kubernetes, the Complete Guide. It's not expensive, especially there's a lot of coupons. You can get it at a very low price, like 10, 12, $13. And so this person had gone through that course and was like halfway 2/3 of the way through and just thought it was fantastic to ramp up on it. So given that recommendation, I'm going to recommend that to everybody and we will put a link in the show notes because if you're going to be in the AI world, it really pays to understand Docker and Kubernetes.

Well, awesome. Yeah. Well, check those things out. Reach out to us on our Slack channel or on LinkedIn or Twitter with any questions or thoughts that you have and hope that this has been a fun episode. It has for me.

It has been.

We will see you next week. See you, Chris.

See you later. And I apologize to the Nvidia people who are going, oh my gosh, those guys, they need to know more about it before they talk about it.

We accept feedback and in the show notes and everything is is on GitHub so you can submit a PR.

So yeah, okay.

Feedback. Welcome.

See you next time, Daniel.

Bye.

Thank you for listening to Practical AI. We appreciate your time and your attention. Word of mouth is the number one way people find new podcasts. If practically I has helped you on your AI journey, please do tell a friend and hey, they'll thank you later. Special thanks to Breakmaster Cylinder for the Beats and to our awesome partners for their support. Shout out to fastly Linode and Rollbar. If you and your organization would benefit by speaking directly to the AI community, you should sponsor practical AI. Podcast advertising is highly effective and we would love to work with you. Head to changelog.com sponsor to learn more. That's all for now. We'll talk to you again next week.