Advanced Graphics with Shader Graphs

Hi, my name is Alec, and today I'm going to be talking about advanced graphics with shader graphs. Shader graphs are an upcoming feature in the Wolfram language. Today we're going to take a quick peek at what we have so far, so a quick overview of today's talk. First, we're going to start with a brief introduction to what shaders are and how they're used in the Wolfram language. Then we'll go into shader graphs, how they're structured and the motivation for actually using them. Then we'll end with some demonstrations on using shader graphs with our kind of current state development so far. So to start us off, if you want to know shaders, you have to understand the graphics pipeline. This pipeline takes all the data from our scene, so that's all objects, lights, cameras and sound information. Takes all of that, processes it, and converts it into the final 2D image you see rendered to your screen. This pipeline is made up of multiple stages. These stages are run out of GPU and each one sort of takes the previous output, processes it, and then outputs it to the next stage. The details here aren't too important. However, I want you to notice the two types of stages here. The first is the fixed function stage, shown in gray. These stages are controlled by the graphics API, so we're not really able to change much with how they work. However, the second type, the programmable stage, shown in red, these are, as the name implies, programmable, so we can customize the behavior when rendering. These ones are called shaders. And here there are the two main types. We see the vertex shader and the fragment shader. The vertex shader is responsible for the position of the vertices, so how they actually projected from space to your screen. The second one, the fragment shader that controls the color of the pixels, that kind of map to the object. So that we have kind of a basic understanding of shaders, let's see how they're used in the Wolfram language. So it's kind of done in three ways. We have the standard shader with directives. We have the alternative shaders like material shading. And then we have the surface appearance, which is a directive that uses the OpenGL shading language to make custom shaders. All right, so the first off, we have the standard shader. This is probably what you're used to if you use graphics before. If we don't add any other information and we just render something, it'll be in the standard shader. Here we see our torus rendered with a nice sort of matte material. Now, this shader does take parameters. For example, you can specify directives like the faceform being a color here, a specularity. So in this case we'll have a red shiny torus. However, in these cases we're not actually changing the shader itself, just a parameterization. And then next up we have alternative shaders. So these are actually separate shaders than the standard one. So they're kind of their own custom code that they're running on a gpu and each one's meant to sort of achieve a specific effect. So we have our 3D ones here called the shading symbols, and then we have the equivalent 2D called fillings. Now like I said, these are all meant for kind of a very niche purpose. For example, kind of artistic toon look, or a kind of hand drawn effect, or even a photorealistic effect. Yes. So these are also customizable with parameters I can see with this red torus. However, they're still limited on how much you can do. They're not that flexible. All right, that brings us to our last one here, Surface appearance. This is the most powerful of all three. So it provides full customization of both the vertex and fragment shader stages. And it does this through writing the OpenGL shading language or GLSL programs. So these are powerful, but not that user friendly. For one, it requires detailed knowledge of the graphics pipeline. This includes things like transformations and vertex interpolation. Two, compared to the local myoid, it's much lower level, which you may not be used to. Plus even if you are, there are some unique restrictions such as no dynamic memory allocation and some weird stuff with control flow that can make it difficult to use. Third, here it can be inconsistent support across hardware. So what works on an AMD chip might not work on an Nvidia chip. It kind of depends. And lastly here it can be difficult to debug. So unlike a normal program on your CPU where you can either do a breakpoint or print out values, as it's running for shader programs, you kind of just have to return a color and maybe check to see what color is being returned by your shader. So it can be very limited on how much you can debug. So with all that said, that kind of leaves us now with a gap between the kind of very easy to use but limited standard shader and this much more powerful but complex surface appearance. So the solution is usually shader graphs. This has been kind of common solution in recent years for a lot of different applications from game Engines to kind of different modeling software. They find this shader graph finds a nice balance between the kind of simplicity and complexity. But yeah, so here you can see kind of a list of companies that currently use shader graphs in some form. Yeah. So now that we know, like why we want to use them, let's see how they actually work. So what is a shader graph? A shader graph consists of nodes that act like functions. For example, here we can see a function on the left in Wolfram language. It's very simple. It takes two inputs, A and B, and adds them together for a single output called out. If we were to make the equivalent node, it'd be shown on the right where it's our plus node that takes two input ports, A and B and has one output port called out. Now, once you have these nodes, which can be as simple as like a single plus or as complicated as a full shader, once you have them, you can combine them into the final shader graph. This is done by connecting those node ports to each other and they form a directed acyclic graph or a dagger. So this means that there are no loops and also that each output goes into an input. So there's no sort of like for loops or any sort of messiness there. It's very easy to see where things flow in and then flow out of your shader graph. Yeah. So there are a lot of advantages to this kind of shader graph structure. One is discoverability. So instead of having to kind of dig through some documentation on a shading language to find certain features, you have just a nice, easily scrollable with some nodes to see what you can actually do. Second, there are guardrails. So because there's these kind of predefined pieces and building blocks you're putting together in the form of nodes, it's much easier to kind of not do something incorrect that will break your shader. It's also easier to not fall into common pitfalls that you might have when writing the code yourself. For example, things that only work on certain hardware or things that seem simple but run really slow due to the details of how GPUs work. All right. And third, here we have debugging. As I mentioned before, debugging is pretty tricky for shaders, but it can be a little easier with shader graph. The fact that we can kind of dig into certain nodes and see their values. It is much more helpful than just having the kind of final result of the image that you might have with a more kind of standard shading language kind of related to that is the explicit data flow here. So like I mentioned before, it's very easy to see kind of where data starts and how it flows through your entire program. So it's very easy to track what's going on. And the last advantage here is portability. So shader graphs are often much more portable than actual raw shader code, which means it's much easier to transfer them between different applications. So, for example, it's much easier if you have a model rendered with a specific shader in one application, like Blender, moving that into a different one while keeping the same appearance. I'll talk a little about a little bit more about that later. All right, let's actually get into some demonstrations here. So before I start, quick disclaimer. A lot of this code is still early in design, and it's very subject to change as we work more on it. So this is more to give you a taste of what's possible rather than the specific syntax you need to do this stuff. Also, the syntax currently is heavily based on Netgraph, which is another computational graph in the Wolfram language currently. So if you're familiar with that, you'll probably be kind of familiar with the syntax as well. All right, so our first example here is going to be to create the standard shader that we saw before as a shader graph. So we start here with our shader graph symbol. It takes two arguments. We have the list of nodes and the list of edges. For our nodes we just have the string names, as these are already predefined. So we have this one here called standard shading, which has already been defined to mimic the kind of built in standard shader. And then we have the second node here called setpixelcolor. This one basically acts as a return value for our shader that whatever color you give it, that will be the color returned to the final image. And then lastly here we have the edges. So because this kind of simple graph and both these nodes have the one input and one output, we can just sort of pass the output of node one here and to the input of node two. So if we go ahead and run that, we'll perform all the type checking and validation. And then we get our final shader graph here. There's our properties. For example, we're going to get like the list of nodes, and here we see them in their full kind of shader node wrapper. And then we go ahead and take that graph and render it. As you can see here, we're using this Kind of property called service appearance. This is kind of a temporary thing, but this cache shows under the hood that we are generating that GLSL code and using service appearance to render it. So I can do that. And we see our kind of nice default sphere. Nothing too special about this. Or this can also work with the directives as you would expect. For example, if we do a color and opacity, those flow through to the graph like you would see in a normal standard shading. And if we're also curious about the code, I have sort of a custom utility here that lets us see the code for the two shaders that were generated by that graph. We see on the left we have the vertex shader. On the right we have the fragment shader. So you can see there's a lot of kind of built in uniform data about different lights and a lot of functions for calculating different lighting. So as you can imagine, this would be pretty tedious to write if you want to do it yourself. That's why it's very nice to just have these things as a built in node. So you just specify the name and it automatically get all that boilerplate for free. All right, so let's now talk about a custom property. So if we want to have some sort of property that's not one of the default directives be passed into our shader, we use a custom property. Right now there is kind of a lot of extra stuff here for determining the data type and the name. This will be simplified later on, but overall this node here will kind of take in a custom color property called main color. Now we can just pass that node to the same graph as before, where we have those two same nodes and we're just adding it to the start. And then that node will take its main color that outputs and pass that to the diffuse color port on our standard shader node. So we can go ahead and actually create that graph. And if you look at our standard shader node here from the node list, we can see that it has multiple input ports, one of which is that diffuse color port. Now if we go ahead and run this, we can see we have our sphere shown in red and we can change that main color property and it updates dynamically as we'd expect. So that's all working well. However, it may not be the most exciting because you can kind of do this easily with the standard shader using a manipulate. So for our next example, we're going to look at something that you can't do with a standard shader. Go ahead and Go here. So now we're going to do something similar, except instead of using a kind of property for that node, we're going to use a custom subgraph called checker. So one nice part about shader graphs is that you can actually nest them. So instead of just using kind of some top level nodes, you can essentially build up functions and reuse them in different places by nesting shader graphs. So here I define one called Checker. Even though it's a shader graph, I can just use it like any other node and this other shader graph, so I can take that and pass its output into that same diffuse color we saw before, go ahead and create that graph, render it. And now we see this checker pattern being applied to our object, we can change the scale to define, like how fine the grid is that we're using. And one thing to note is that it's not like a normal 2D texture mapping that applies some flat image to the surface. It's actually looking at the position of the pixel and 3D space and determining the color based on where it lies. So if I do a different model like Beethoven here, we can see it maps to it fully and kind of see as we change it how, depending on the position of the pixel, it changes the color based on the grid. Yeah, so that's kind of an example you can't do with the standard shader. Next up, we're going to see a similar example. This time, instead of a checker pattern, we're going to try something a little more useful with a contour line. So the goal here will be to render kind of like a 3D elevation map. We will have regular contour lines at a certain interval and thickness. So I'll create our little helper subgraph here called Contours. I'll pass that to our actual main graph here. I'll define some models and then we can render it here. So here we're rendering kind of a 3D model of Mount Everest and we have these elevation lines. We can change the frequency to be more or less frequent. We can change the thickness to be very thin or very thick. We can change kind of the offset so like how it slides across the surface. And of course we can change the color like so. And then this shader can be applied to any model. So here's it with the kind of blender Suzanne, Monkey Head. And we also do it to regular sphere and once again change the color. So I'll just update dynamically. Yeah. So once again, this is something you can't do with the standard shader, but you can do when you write your own custom shaders with shader graph. The last example here will be vertex displacement. So in all the previous examples, I was basically editing the fragment shader. So that's the one that just determines the color of the pixel. However, we can also change the vertex shader in order to kind of alter the geometry on the fly after rendering. So a common use for this is vertex displacement, where either for some animation or effect, you want to move the vertices on the model without having to kind of manually do that to the geometry on the CPU side. So to do that, we can create this subgraph here, which will kind of change positions based on some vector. We can pass that to this graph, which will be our vertex shader. So what this will do here is we'll have our position of our vertex. We're going to pass that to this distort subgraph, also give it the normal as well. So that's the surface normal direction of that vertex. And then that subgraph will sort of offset the vertex along that normal direction and give it a new position that passes to the set position node, which is essentially like the return statement for a vertex graph. Once we have all that, we can now render our sphere. Right now it's pretty, pretty normal, but if we change the amplitude, we can see that it's kind of kind of moving the vertices based on our normal, and it's actually doing it based on a sine curve. So we get a nice little wave. So I can actually increase the frequency of that curve. So you can see we get more waves, and we can actually kind of shift that curve across the sphere. So we see this kind of nice wavy effect on our surface. And we can even see the original vertices, if you want, with this edge form where you can see some of them are being pushed inwards, some are being pulled outward. And yeah, so it's a neat effect. And all this is done on the GPU without changing any of the actual sphere geometry on the CPU side. Yeah, and this effect can be used for many things, from grass blowing in the wind to like a flag on a flag pole kind of blowing in a breeze. Yeah, so it can be pretty useful. All right. And that was my last example, so I'll enter out some plans for the future. So. So these are things we kind of want to continue to add as we develop. So while we definitely want to simplify the design to make it very easy to use and very kind of avoid any boilerplate, make it as like minimal code as possible. Next up, we want to add more custom nodes. So you already saw we can do stuff with for example, shader graphs to do nested nodes. However, we can also potentially do custom nodes with GLSL code that you write yourself, or potentially even multiple language code that's simple enough to be compiled to glsl. That could be a possibility as well. Next up, we want to kind of continue integration with MaterialX. So we've already made some strides here. And for those of you who aren't aware, MaterialX is a open source library file format for shader graphs. The goal is to make shading more portable across different rendering software and do that by sort of having their one specification that different software can sort of develop to. So while the code might be different for different softwares, they all sort of agree on the final effect that should occur. So anyway, with our share graph, we wanted to make it compatible with MaterialX. So we too will be able to sort of export or import shaders from other softwares into Mathematica or from Mathematica. And lastly here we have a gui. So this is still have very much a maybe, but we could potentially add a nice graphical interface editor that lets you sort of build this shader graph by clicking and dragging stuff instead of having to write all the code for it. Yeah, with that, that is my talk some other points of interest. So I believe right after this talk we have Yuzu's talk on GLSL with service appearance. So if you want to get into the nitty gritty of writing GLSL shaders, you should go watch that. And then there'll probably be some more shader content going to the Wolfram r&d YouTube channel, so you can subscribe to that if you want to see more of that stuff. But yeah, now I think I have a few minutes for questions. So if you haven't had a chance already, you can type your questions in the chat and I'll try to answer them here. All right, thank you. All right. Hi everyone. So we got a few questions that came in through the q and a.1 was just about when shadergraph will be available. And right now the plan is for version 14.1. So it's not in the current 14 release, but hopefully the next one, it should be in there. Yes. Next up, is it reasonable to expect an importer that can convert shader scripts in the wild to shader graph objects? This kind of depends. So as I mentioned, we're trying to have some sort of custom code node that could Hopefully Take for example a function written in GLSL and just sort of use that one. So potentially yes, for like smaller snippets in terms of whole shaders, that can be a little trickier because it kind of depends on the platform it's written for. There can be some differences there. So probably not the whole shader, but potentially smaller snippets. Yes. Let's see. Will there be a connector to bring materials from other 3D programs? So yeah, so the goal here, as I kind of mentioned a little earlier, this would be. One moment. There we go. So I mentioned MaterialX. So this kind of a growing kind of open source format library for doing just that, trying to transfer these shaders, which are usually very kind of locked into certain software and try to make them much more shareable between softwares we have information. So this kind of library is backed by a lot of kind of big names and sort of rendering from Pixar to Adobe to even more recently, Apple, particularly Apple has recently announced as like the kind of standard way to make materials on their new Reality Kit Vision OS hardware. Yeah. So ideally we're trying to kind of write our code so that we can import and export to the Material X format and that would allow us to then in turn import export between these different programs if they're kind of compatible with MaterialX. All right, all the current questions. One thing I didn't want to touch on, I didn't mention in the main talk, is that I just showed it for 3D graphics, but also kind of support for 2D graphics. Just have the regular 2D graphics. We are planning to add the shader graphs for those too. They might be have kind of their own separate symbol or name. We're not quite sure yet. Although we do want to add support for those as well. All right, I'll give it maybe another minute or two here. If there's any last minute questions. Yes, if you have any, please put them in the Q and A. All right, looks like that's all the questions for now. So thank you for coming to this talk. And once again after this one, Yuzu will have his talk on actually writing those GLSL shaders with the server's appearance. So I recommend you check that one out. All right, with that, thank you and have a nice day.