CUDA and the Wolfram Language

So, good morning, my name is Tom Wickham Jones. I've been a developer and implementer of the Wolfram language for many years and today I'm going to talk about CUDA and the Wolfram language. So let's get going. So we talk about Cuda, we're talking about GPU computation, I.e. graphics processing units for general computation. And like in some old, maybe older terminology, this is using the GPU as a CO processor. This can lead to a massive acceleration for computations, but they typically need to be very large scale problems. It needs special hardware, the gpu, but it's not suitable for all computation. Now CUDA is a programming platform, language and tools for Nvidia GPUs. It's not, it's by far the most popular GPU system, but it's, it's not the only one. But you know, these Nvidia Cuda cards are what, you know, they're doing enormously well, for example, driving all this sort of, you know, AI functionality that we're all interested in and using. Cuda comes with a very extensive set of libraries, so it fits very nicely with the Wolfram compiler. It's not available on the Mac. So actually I'm running here, I'm using a Mac, but I'm running a remote kernel over to a Linux machine. So I'll show various models, try to help. You have like the cpu and this is the graphics card with some grid of little processors and I'll talk more about what these are. So the ideal workflow is that you have your, your cpu, your regular computer and you load some data onto the gpu, you then run a program that gets downloaded to the GPU and then executes, you extract results and then you sort of tidy everything up. So just a quick statement of terminology. It's very tempting to talk about CPU and gpu, but technically and in books and things it's a bit better to talk about host and device. So for example, you know, there's other things on the main computer like memory that's, that's also part of the host, but that's not really part of the, you know, cpu, right? So let's talk compare the host or CPU versus the device or gpu. So on a cpu, the philosophers, it's very much a single core maybe and you know, in modern processes expanded to a smallish number of, of cores, but much more than 24 is getting kind of exotic and, and the way that performance is gained is by reducing latency in a single thread of execution on one core. And, and there's huge amounts of work have gone into modern processors with caches and pipelining and branch prediction, etc. To reduce this latency, this sort of waiting. And that makes CPUs good for general computation. So GPUs have a completely different philosophy and their goal is to boost throughput of an entire system. And they might have thousands of cores like the machine I'm using now, not totally exotic, has, has nearly 6,000 cores and a paradigm that is, it's quite convenient to think about. The way this works is it's single instruction, multiple data, so simd. And they're not good for general computation. So let's look at some numbers. This is like maximum theoretical raw speed that I found from, from a couple of years ago. So an Intel 24 core system has, you know, this, this system, it could do double precision computation in about a third of a teraflop. That's floating point, you know, operations per second. And single precision a about twice as fast. In contrast, the Nvidia Tesla card would do double precision in, you know, nearly 10 teraflops, but a huge jump going to single precision. So, so they really are optimized for single precision and there's various reasons for that. And they have a half precision as well. Some of these advantages of these smaller things is there's less data to move around, so the memory is super important. So on CUDA GPUs there's an entirely separate memory system. And so part of the computation is always going to involve copying data between the host and the device. And this copying can be slow. And on the device there's different sorts of memory that have different ways, different speeds and performances. Bit like caches on the cpu, but you have a bit more control over it. Also accessing the general global memory on the device, often that can be done in parallel and it's quite easy to understand how that might work. And so that sort of helps to turn these things into SIMD operations. But despite all of this, memory access is typically the bottleneck that prevents programs from reaching the theoretical performance of the system. And again, as far as programming goes, you have code that runs on CPU code that runs on the host and CPU code that runs on the device. And it's inevitable in working with these things that you have to manage these two styles of programming and make them work together. Now let's talk about CUDA Link. So Kudalinka Wolfram language tools for working with cuda, they've been available for many years and they had all sorts of features for working with data and programming. And what we're talking about now is an updated version of CUDA Link using the Wolfram compiler. So this is providing features such as automatic handling of data on the gpu, generating CUDA programs and getting easier access to CUDA functionality. So let's have a little look at this. So first of all, this was something that often in the past has been kind of complicated and kind of a bit, a bit challenging and that is how to set up the system, the configuration of the system. And starting now in version 14 we were starting to try to make things a bit easier, especially for debugging problems. But we have this thing so package of CUDA tools and we can load to get information and running this thing has very few requirements. It does require that you have a CUDA card in your machine and if you don't have a CUDA card then Cuda obviously can't work. But it has very minimal requirements. And what we're seeing there is similar to, you know, stick a card in and install drivers which is necessary for the card to work. You'll get something like a system information. And this, this, that magical screenshot of this, this is like obviously looks a bit Linuxy and there's my Nvidia card and we can see here that we have CUDA cores is 588 and sorry 5888 so that's nearly 6,000 little processors that I have available. So that's much more than you get with your multi core intel or ARM cpu. So that's the hardware. But in order to take advantage of Cuda you need software as well. And Nvidia provide a nice Cuda toolkit that's also required for things. And again we've written, try to optimize and streamline some tools for working with this. And we have like a Get Cuda toolkit and this tells us, you know, the toolkit with a path to the toolkit and there's like an Nvidia provider compiler for their things and we've set these up so that we can support different versions of the CUDA toolkit. So when Nvidia come up with a new version we can support that and then you can, there are ways of switching between them if that was important. And you probably get an idea that the version of the CUDA toolkit that I have here, there's a bit of a clue in the, in the name of it there. So CUDA Link is the Wolfram language toolkit for working with CUDA and for this to work, there must be a CUDA enabled card, so hardware and there must be a Cuda toolkit and that's software and our Cuda tools functionality make it easier to test your setup and identify problems and resolve them in case they're not. In case it's not working. So let's do some basics. And now I'm going to show how to create some data stored on the gpu. So I call we have this thing called the CUDA vector. So this is a one dimensional layout of data and it's got a certain type and I could make a CUDA vector, give it a type argument and then it makes a different type of, you know, different thing. So these are a little bit like compiler data structures. It's got a different head, but if I get a normal of it, it's returned in a numeric array because you know, I might have different size data and if I get normal of the numeric array it prints out, you know, the actual numbers. So key thing about this is that these CUDA vector, there's Cuda matrix, there's sparse versions. They are, they fit in. This is a nice feature from using the compiler technology. They fit in with the expression system so that if they're no longer being used then they automatically, they notify the GPU system to free the memory, which is, you know, kind of how things ought to be. So what can you do with these? So one thing you can do is to do some linear algebra type computations. So here I'm generating a matrix and a vector on the. I don't want to install Q new software right now. I'll go back to answer these questions. There's questions, they're good questions. I'll answer them at the end. Okay, so, right, so I've done like a linear algebra computation and that's now I'm going to make copy, make GPU versions of the matrix and vector and run this thing called Cuda dot and this runs and this, this particular case is running, is running faster. So the implementation of this is calling into a library Cublars, which is a BLAS layer from Nvidia. So BLARS is like an API that gets supported for different hardware platforms. It's a set of function definite, you know, the signatures that you can call to do things like matrix multiplication. Right, Here we go. Right. So I want to make a comment on the speed of operations because this is like, can be, you know, can cause a lot of confusion. So the legacy all parts of CUDA link also had ways of running GPU operations on the gpu. But it tried to be too clever. And so often it would copy the data from the host to the device, run the operation and then copy it back. But that can be much slower than running the thing itself. And so this led to endless confusion where people didn't understand the speed and the performance of things because it was just so confusing. The solution to this is to use the CUDA vector and the CUDA matrix. We've built a whole range of operations to work on these things to create them. And you get some flavor of these and I'll give some summary of this is a nice way to create a, you know, it's like range, but it's doing it on the card and I can get out the data that we see here and then I could run say like a map. So now I'm operating on the vector with a function. Now typically when things like CUDA map run, it has to code generate for this function the first time it's executing. And that can be kind of slow. But it caches the result of the compilation. So it, yeah, I mean people are answering questions. It caches the result of the computation. So you know, you don't have to do that again. And there are ways of running these things at some slightly lower level. We have a CUDA table so we can run that. This has one iterator, so it produces a vector and we can have do the ubiquitous Mandelbrot computation. So this is a CUDA table with two iterators and this produces a CUDA matrix which, which we can plot out. So we can do prepare data and things like that. So we've got access. This is using Nvidia built in libraries. I mean, I'm only generating 10 of these, but you know, probably you want to work much bigger, bigger sets of data so you can do random, There's a sorting and devising algorithms to do sorting over this sort of hardware is pretty interesting. Now I think one key point that again can be endlessly confusing is these are not necessary individually. These are not necessarily faster ways to do these sorts of operations. So they're, they're, they're running on the device and they can fit in part of some bigger algorithm or computation that might need fast access to floating point, you know, to random numbers without having to copy them back and back and forth. So I think that's, that's kind of a key, key point. Now I want to talk a little bit more about how these things are implemented and see about how we can use the compiler for this in a more explicit way. So as I said, there are. Two types of programming and we have a compiler environment that's set up, that runs on the host and this has all the definitions for working with CUDA libraries and things. And it's possible that you might invoke function compile using this compiler environment. And this is going to generate code that's executing on your cpu. In addition, there's a device environment and this is when you're co generating to run on the device. But this often gets used automatically because there's a set of functions that make that easier. So let's look at some CUDA programming. And the idea here is that we are using the Wolfram compiler to generate CUDA code. So you know, it's actually co generating. This is a pretty nice feature and a consequence of the compiler. This is how we implement CUDA table and map and everything. And it can make a nice concise workflow and it can save all sorts of details of working with CUDA and allocating. So still it's useful now to have some knowledge of how the system works. And this is some one version of this. And so you have threads that are organized into blocks and then blocks are organized into a grid. So every thread is running on one GPU core and the threads have an ID inside the block and then each block has a certain number of threads, a dimensionality, and each block has to have the same dimensionality and each block has an id. And there's a shared memory system that runs for everything. Now you can also have, you know, a more, you know, 2D or 3D grids, but kind of the same, you know, similar types of things. So that's the layout of the system. So now let's see, like a very simple basic CUDA kernel. So it's a Mathematica function that we have and we've got a C array of reals. It's got a, you know, the number of, you know, the length and we're going to add something to it. So we enter that and we call a CUDA compile export CUDA compile export kernel. And this generates a CUDA kernel object and we'll see a little bit about what that is. So we can generate some data and then we can run the kernel over the data and this particular kernel overwrote the input, which we can see has been changed. So this is a pretty streamlined way. When you work with this in C or something, you've got to write quite a lot of non trivial code in order to get that to work. So when the CUDA run kernel is running, you have to set dimensions, you know, the block dimensions and the grid dimensions. If it was two or 3D dimensional, you know, these would be lists and key element. Here again is it, you know, like, like, by default it's able to figure out suitable block and grid dimensions, but you can change these if, if, if you want. And because the number of threads in each block has to be the same, there's going to be more threads than there are, than there is data. And so that's why you always have to have this test to make sure you're not accessing outside of the range of the data. And then another point here is that here we are reading and writing data. And because we are reading and writing the data, it's very often this is done because of the way these threads fit together. This can often be done in parallel, where blocks of data are loaded into a whole block of processors, which, that's what SIMD is. So that's all nice and efficient. So let's have a look at some details of the CUDA kernel. It's a bit like a compiler data structure. So we get the CUDA kernel and we can pull out information. For some reason, information always prints out in a small. It's a bit hard for people to see, but we can pull out, say, say, like the types of the function or ptx. This is the low level language. So this was generated from the Wolfram compiler, it uses llvm and this is the assembler code for CUDA that we generated. So to do bigger, more complicated things, algorithms for tiling or sorting or matrix multiplication or aggregating or something, you need more functionality. And to make things efficient, you might have to do techniques like tiling and so on, like that, which are all quite interesting. And you need more functionality that we've built in support, such as shared memory and thread synchronization and dynamic kernels. And so we've got quite good support in the Wolfram language for these things. Now you might still want to go find some C code and use that, but use it in a simple sort of way and you can do that. So here I've got a. This is a pretty similar kernel, actually. I should probably have, you know, IDs less than length. There we go. Right. And I can then use the same CUDA compile export kernel, but now this is using C input and it compiles this and then I can run that in the same way that I ran before. We've got A smooth, uniform interface to programming and executing these things, whether it's Wolfram language or C plus. So often, as I said, you're not, you know, you don't have all the code. You might have some library of code that is that, that, that you're using. And this, these, these libraries are, you know, CUDA computing toolkit that Nvidia ship has a whole load of them. A lot of them are written in C and they make very heavy use of C templates. And so one, one particular example, there's this thing called Thrust, which is this high quality, high performance library that's bundled by Nvidia. Now these, some people you might know, some of you might not, but C templates, the C way of doing parametric polymorphism. So the Wolfram compiler also provides parametric polymorphism. And so we built this way to line up the Wolfram type system with C templates. And this allows us to take, for example, this is a C little C snippet that's making a call into the Thrust library. So it's a thrust sort and it's a template. So the point of templates is you haven't said what T is. T is a parameter that gets figured out at some instantiation time. So this code would work for different reals or doubles or, or different sorts of things. So we can build a function declaration that references this. And that's what I can see here. So that's the name of the function and the type is given polymorphically using the forall type. And it's going to line up the type parameters with the template type parameters, which, which I think is pretty cool, maybe nobody else does. So here's a function that's going to call this CUDA sort declaration and it's passing in a CUDA vector of integers and I shall compile this. So this has to run, it has to instantiate. Quite a lot of the work here is instantiating C templates which take a little bit of time and then this will run and it comes back. So there we go. It would also be possible to do function, you know, export library or put this in a compiled component or something. And notice in the function compile, I set the compiler environment variable to be the CUDA compiler environment and then now I can execute this. So I make a CUDA vector, I call my function on it and this has sorted in place the vector of data. So we called a library of C that was templatized and this is how we write many of the functions that we have. So I Think that's all kind of interesting, right? So I think I'm coming to the end of this, so I'm going to summarize and then I'll just run through some questions and things. So I looked at CUDA GPUs and their potential for high performance computing. I looked at the Wolfram integration for this and how we can probe the setup, the cards and the tools help you get, get things running. I looked at the basics of 12 day, 2D data with CUDA Vector and CUDA Matrix and some built in operations. I looked how to compile Wolfram language code to CUDA and then integrate with C tools for cuda. So some of the future directions that I think are kind of interesting are actually I found some very nice sources of lots of, you know, CUDA examples and algorithms and discussion of this. And I'm, I'm working now to try to translate these, which I might find an intern into Wolfram language versions because that makes sure that our code is working, our implementations are working properly. But it'd also be a really interesting set of learning resources and a way then we could start to build some unique algorithms and tools that could maybe enhance the war from language. This is also a great way to, to learn about and experiment with the CUDA system and it really is a different mindset. I think you have to throw away everything you've thought about your CPUs and start thinking anew. And it is a lower level type thing. Here's an interesting workflow that I would like to optimize which is prototyping these things in Wolfram and then just taking these CUDA kernels that you created and just, you know, their expression so we can just take them out and running them directly in some large scale, some large scale production system. So I'll just give a few thanks. So, so I've been working on this and there's members, particularly Fahim and Ilyan have been, been helping with this, this CUDA work in from the Wolfram compiler team. But I'd also like to give a call out, maybe mention Thomas Fry, who's a CS student from Bath University who spent a placement year with us working on this. And I think it's all been pretty, pretty nice. And that's just also calling out that we have an active intern program and we get quite a few interns coming through and they always have a really fascinating time and we're always keen to be bright young people and yeah, they always have a really interesting time. So that's the end of my talk. I think in the, I'm going to stick around. Some of you may have to go off to other things but I'm just going to go down through the questions and try and answer them. So question number one. Will there be a future support on Apple's GPU or others? Yes, I mean maybe, but you know, Cuda is enormously dominant. It just does a ton of stuff. More maybe if there was, you know, every time people always ask this but these, these other things just do an awful lot less. I mean there is huge amounts of resources and material and knowledge about, about these, these Cuda things and that's not, you know, when there's, when I see, you know, like a blase layer for Apple or something, then I will be interested in this. And you know, sometimes people, they send me some, oh, you know, Apple's faster for this thing but then they can never send me any follow up and backup. And so I believe in actual, you know, real, real things that, that I can see and feel and touch. So that's, that's, that's my answer there. So is there a handout coming? Yes, I'll be, I'm going to make a slightly polished up version of this notebook and I'll be uploading it and with my talk from yesterday. So if I turned an intel based Mac into an intel based Linux machine, would an Nvidia based GPU be able to run Mathematica Cuda? Right. So there's no such thing as Mathematica Cuda, there's just Cuda. So if you have a machine that can run that has a Cuda card in it, you know, so you can see like the system information, if it has a Cuda Nvidia card that is supported by, you know, Nvidia toolkit, then we will work with it. Okay, so that's, that's your question. So but if you have an intel based Mac, I don't know how easy it is to put extra hard. I mean if you're buying a Mac like you know, a MacBook Pro or something, you know, it's, it's pretty hard to put in extra hardware into it. Right. Does anyone have any experience of a adapting this tool set to Drive Google Cloud TPUs? Guessing this would require mapping to XLA. Sorry, I don't have a good answer for that. I'm going to have to research into that. But it's probably an interesting question. What's my take on Mojo? Mojo's a language that is using a compiler based system and it can also, it's pretty, pretty interesting. So I think usually when I look at things like this, I think there are things we can learn from it. So I think we have some advances and they have some, some advances. So David Galby wrote a CUDA core is an ALU, while a CPU core has many. ALU CPUs have 32 bit. So the name core count for the number of CUDA cores is very misleading. I mean all I'm saying is that's what the. So someone saying it's misleading to call them CUDA cors. But these are the things that are running the individual threads and in that sense they are coup de corps and that is what the Nvidia system information tool uses. So I think it's, you know, we're just following, I think it's not unreasonable to follow Nvidia practices. So Jason Harris asks, is the output CUDA vector stored in the notebook or is that a reference to structure on the device? Could you do command shift E on one of those outputs? I think it ought to be not stored in the thing but. And we've been doing a bit more work on this. Where are we? So it's, I think that's just the, it's not really storing the data, I don't think, which it shouldn't, it shouldn't store the data because the data can be massive. And if you were shipping it back and forth between the, you know, host and the device and the front end, that's, that sounds like a great way to become very slow in a Mac laptop environment. Is there a convenient and efficient way to use CUDA GPUs? So there is. The convenient and efficient way to use CUDA GPUs is to buy a non Mac. Sorry, I mean that's, that's, you know, you can't use, you can't use CUDA on a Mac. That, that doesn't exist. Macs do have a GPU system which has some sort of ability to program and, but it's much, you know, it's, it's, it's, it's lower level and it doesn't have all the features and things, but that, that, that is possible. Will there be tools for users if we want to develop our own device link, say as a replacement instead of Nvidia cards? That's a slightly. Right. So will there be tools for using variety? I mean, that's a slightly vague question. I mean in some sense, yes, there's tools, you know, that we, you know, that's, that's what we can build. If there was something important that was a replacement for Nvidia. Fine. But you know, Nvidia Sell, they are the market leader by a long, long way. And I'm going to fall follows success when there is something that's important or, or if someone comes along with some card and they, they, they want to have a consulting project. Yeah, sure, we, I'm sure we could extend to support that, but I think it's, it's, it's just the case that Cuda is very dominant, you know, and there is an awful lot of resources from that working with these other tools. You're going right back to the beginning and you'll be implementing everything. Now that might be fine, if that's okay for you. Right. I think I've answered all the questions, I hope and I think I'll wrap up if there aren't any more. I'll be. Oh, here we go. Right. So I meant to ask, will there be comprehensive documents and, and walkthroughs and tutorials? Yes, that's a good question, Sylvia. So Sylvia is asking here, she meant to ask, will there be comprehensive documents and walkthroughs and tutorials? Yes, we're going to try to sort of improve the documentation and make lots of examples. As I said, I think this is a really fascinating way to learn about the technology and to use it. So I mean, I think that's a great way that the warfarim language tool set is, it's a great, great way of learning about things because it lowers the barrier for accessing them. So I think that's, that's all quite useful. Right, Okay. I think I'm going to exit unless there's, there'll be opportunities to ask me questions and that sounds good. So let's make sure. If I click on the right button, I'm going to exit. And thank you all very much for coming along. Thank you.