Customize Mathematica Graphics Rendering with GLSL

Okay, let's start.

For those of you don't know me, my name is Zhu Lu and I.

A front end developer mainly working on graphics.

So today I'm going to talk about.

How to using GL slang to customize mathematic graphics language. First of all, you might wondering what is GLSLAN?

GLSILON is just OpenGL shading language and it is a piece of.

Program could be run on GPU directly if you are using OpenGL. So we have an internal function, it's called Service Parents, its graphics feature and allows to customize 2D and 3D graphics rendering through this surface variance.

There are many newly introduced features, graphics.

Features building on top of it, mainly fillings and shadings.

And even though it's an internal function.

It is also accessible for users as well.

So next I'm going to give a.

Rough overview about what is Surface Parents and going deeper a little bit and.

Then list a few features implemented with.

Surface parents so you have a rough idea how much the surface appearance can achieve.

And also I'm going to use a.

Very simple example to walk you through and you will know better how this Suki experience works step by step. And after that I'm going to give updates about a few new things added to Sufi's parents and give two demos about how to use that to do multiple path rendering for both 2D and 3D. Yeah.

Okay then.

So what is Surface Parents? To have a better understanding of that.

We probably need to introduce a little.

Bit of how mathematical graphics works.

So after the kernel and also the user are done with all the fancy.

Computing and send data to front end with graphics language.

With primitives, with directives.

With those options, front end will decompose that even more.

And like for those polygons, all the primitives decompose that into triangles and all.

The camera like a viewpoint, view vector.

View angle, all the things and also even the rotation geometry transformation will end.

Up resolved as a four by four matrix.

So that then with that data we.

Will push it to hardware and together with it we're going to push internally.

Defined.

Shader to GPU GR san shader to GPU so that GPU know what to do.

And at this stage the surface parents.

Could tell the program that do not send the system shader, instead.

Use the pixel shadows and the vertex shadows associated.

With surface parents instead.

So to make that work smoothly, the surface parents need to have access with.

All the resolved primitive vertex data like position, like a vertex, normals, colors and the texture coordinates and also option values and directives as well.

So here is a list of.

Features.

Implemented with service variance from the beginning that the toon shedding and also the CS G reading, which is a solid.

Rendering using ray tracing and to the very newly introduced hollowing. And they are all implemented with Surface parents.

This gives you a rough idea how.

Flexible and how powerful this Surface parents can do.

Then how do we define a Surface parents?

So here is the infrastructure.

Here you need to give it a.

Name so that later you can reference it with the name.

I need to tell the program that.

What kind of attribute you need to access, like a position normal, colors.

And other than that, you might need.

To also want to pass along other parameters like a list of numbers, a list of integers, sort of things like that, and the colors.

And last, and you also need to.

Pass in the pixel shadow and vertex shadow so that it could be used for hardware.

So together with this talk, I also.

Attached a notebook which is a reference page for Surface parents.

There are a lot of details about the Surface parents. So if you're interested in that, you.

Can look through that and get a much better idea. I just give an introduction a very overview way.

And for example, like you have access.

Vertex attribute and which is a position information. You can also access normals.

Normals are not available for 2D, but.

Available for all 3D. Even if you do not ed ing.

With vertex normals colors works with both.

2D and 3D, but only when you are specified that with vertex colors and texture coordinates will be always generated automatically.

And then and of course you can.

Specify in your coordinates and it will.

Use instead so parameters. There are several types.

It could be a color, could be a number, a lister number or a size, basically integer and also matrix and also texture.

So there are also a list of predefined values you can use which is.

Resolved like directives, diffuse color and also these camera informations and the clipping, the lighting related information.

The most important thing is a metrics. And this is the matrix used. If you apply that so means other rotation.

With geometry transformation and all the resizing will just works. Of course, you can choose not to define your own and it will just work the way you want it. Let's get back to the notebook.

Now I'm going to walk you through.

With a very simple example. Here is customized texture mapping and it.

Will use the position and also the.

Texture coordinates as input.

And also other than that, we need.

To use the matrix as well. So the resize works and things like geometry transformation will work as well.

And also Importantly, you need to feed.

In the pixel shadow or the fragment shadow and the vertex shadow.

And to make that these two loaded.

Correctly you need to specify the light directory. And here is more straightforward view of that structure.

So let's take a look.

What is this vertex shadow?

And like I mentioned, you need declared.

That I need a position and texture.

Coordinates and also need the magics.

It will output the text coordinates to pixel shader. And this is just converting the position.

Applying by that matrix. So and pass around the vertex corners for the pixel shader.

We tell the hardware that we're going.

To use the texture and also the.

Text cone is passing from vertex shadow and output this color.

So from here you can see that.

It'S pretty straightforward computing that converting a sample and converting the color into grayscale. Originally it should be RGB color space. So now let's try add it to the to the library.

So we need to have this private.

Add surface parent definition to add it to the library so that later it could be referenced.

And after that you can check if.

That is added correctly and you can just list it by the reference name.

If it is added correctly, everything will.

Show up and these are the attributes.

And the parameters and the shared programs.

Will be loaded and if not it will just return fail.

So then everything looks good and then we just need to use it and.

Reference that by circuit parents and it will look into the library and pick it up.

And also the program requires a texture.

Then you need to feed it in the texture as well.

Now it is you just get a.

Customized texture nugget and with that you can just all the interaction just works and resizing also works as well. Then next is I will give you.

Some updates about Surface parents. It is Surface appearance is a very.

Active project, ongoing project.

It is more than just have minimum.

Maintenance so that we try to keep.

Add more things to it. Like for example, during the development of.

All the drop shadowing, blurring and hollowing that the developer need to access more information. Like what about the ATL setting option from user and what is the DPI of the hardware and what is your viewpoint size, sort of things like that.

And another thing is we are trying.

To make it work with more primitives.

Since for the past year we make.

Inset work with suffix parents in general cases and also make the graphics group work with drop shading, glowing and hollowing.

Let's take a look here. This is inset previous last year it just only minimum like drop shadowing and.

The blurring work with inset. So now all the these surface parents, customized surface parents should just work. And you can also define your own one. And.

So and the graphics group. During last year we have the job.

Shadowing and it is applied box by box, polygon by polygon.

But for some cases user want them.

Grouped together and there's something.

They cannot.

Put everything into one polygon. So. So here comes a graphics complex, sorry graphics group.

It could group all the box together.

And then apply the geometry as well. And this is an example of the newly introduced hollowing that you can choose to apply one by one or you prefer to apply just them as a whole object.

So another important thing we added in the past year that is image effect to circuit parents.

It will only work with 2D.

And this makes the drop shadowing, make.

The blurring, makes hollowing possible. And then what is image effect? It is basically a post processing which is a computer graphics term.

And it need to have the previous.

Rendered object or primitive result as input texture and then computing on that, adding decorations on that and render that as a final result.

So that is how the drop shadowing works. And we add a list of options.

So this tells the surface appearance that it is image effect.

And I want to mention that we also have this option that allows user other than the image effect user with this on you can also do an extra pass so that rendering the actual.

Original object on top of the image effect. So that is job shadowing. How job shadowing works and the image.

Effect functionality work with all the 2D.

Primitive including inset, raster and graphics group. I will show you more examples with.

Insect and the next update main update.

Is the texture mapping.

For this new version of version 14.

We adding a new option which is called texture mapping.

It will work with all the 2D.

And 3D face based primitives.

And we're adding a list of named.

Texture coordinates over here.

So it will generate text colonies for you and feed.

To the hardware and.

You can specify planner or box. The planner is just applying the UV coordinates and then from X Y coordinates.

And also extrude along Z direction. Box is something that applies six directions.

And the same thing for spherical as well.

It will assume the data is a sphere.

If it is not, it will have.

Some warp on that.

But that is expected. And also at the meantime improve the.

Automatic text conus generating there for surface variance.

And the previously we are using the planner.

It's just applying from top to bottom along Z direction.

And now it applies all six directions. Now you are seeing that the hatch.

Shedding looks much much Better now with that improvement.

So.

Now and I give a rough introduction.

About the image effect functionality itself.

By nature it is multipurpose technology because.

It need.

The result of previous rendering.

And also we make it work with inset so that it could be nested, which means.

Developers and users can do multiple paths or end paths for rendering.

And also since it works with inset.

Everything work with inset like a controls.

Or something else where they all work.

With this predefined image effect. For example, like eye drop shadowing or customized design.

It's not just the 2D graphics. So here I'm giving some comparison with.

This image effect showing original true and.

Plus I'm going to show this with.

A simple example with edge detection.

These two surface parents defined are same use the same pixel shadows. The only difference here is just oh.

This flag said to be 4, this said to be true.

And let's see, we added to the.

Library and then we will take a look. What is this edge detection, this pixel shader. You can tell that this is the edge computing.

By sampling the pixel and.

Its neighbors and also turning it into grayscale. Very straightforward.

So now we see this is none.

This is the original rendering. And notice that it does. Now we didn't have any edge here.

And this does the edge detection and.

The render this as an image effect.

And also we have this flag set to be true so that based on.

The image effect you can have extra passive rendering of the original object to combine them together. Which means this one has to be passed.

But for some cases you want more.

And then we can go through with inset.

So this is the object that we.

The 3D path object we defined above. And we want to apply blurring and the drop shadowing. And it just, it just works.

It's not just the drop shadowing and blurring which is pre defined. You can define your image effect as well here I defined it.

A very.

Older, older school tv. What it does is just have the signal together with all the random noise.

It will just look like this and resize it. It will just works.

Then how about 3d graphics image effect is for 2d but it is also possible for 3d as well just because.

It works with inset.

So now here is a very typical 3D graphics example and we, we wrap around it with inset of course several other things together so that it could preserve the interactivity. Now you can see that all these 2D feelings now works with 3D as well.

And you have all the interaction here. And then you can even Zoom in.

And this, this is one pass. And in case you want more paths.

And you can apply nested insect here we apply the. The very old school have very special effects here.

And here we are applying together blurring.

Together with that the TV effects. You can by comparing these two you can see that both of the image effects are working together which gives a multiple pass rendering result.

And still all the 3D graphics functionality still works.

So.

Now from these demos it's pretty straightforward.

That surface appearance can allow you to do customized rendering.

And if you're having existing gslanresources files you can just easily reuse it and.

Make it work with all the mathematical functionality with very minimum work.

And also as the pixel shared I.

Showed you that all the computing happened.

On GPU and which means it saves.

All the time happened from CPU like image processing. It could happen just on GPU directly.

And also you notice that what we.

Pass along is the GS long program.

The algorithm itself, it's not the data. So we don't need to pass around the computed data saves a lot traffic.

Through mass link and also the memory consumption.

And that's basically moving all the computing.

From CPU to GPU.

And and one of the possible important.

Use could be videos and we do have some experimenting option turned on. They are in the prototype build but.

They could do more.

You can do more.

And the last most important things it makes things much easier to adding new features. As long as you are following the geoslung 330 standard and according to my.

Past experience if you are adding some.

New fancy feature, let's say if you take you one month to make it.

Work with one computer it will roughly taking 10 months to make it work with all computers, all platforms and all graphics vendors.

And so that is our responsibility to.

That you make it work on one computer. Our responsibility is to make it work with all hardware and so roughly just speed up all the process of adding new features.

So that's main things I want to talk and also right after this talk.

We are going to have James talking.

About all the graphics new features.

We have quite a few new features there.

And also tomorrow we're going to have.

Visualization office hour and the Notebook front end office hour. There's a group of developers there and I will be one of them.

So you're welcome to come and let me see.

I have four minutes to go through questions if there's any.

Yeah, there's a question.

Could the jslang script in the present examples we shared at Hanout I.

I tried but the current it does not.

Allow me because the file extension is.

GL slot and I do have two notebooks there. But you can find me in the office hour.

I can try to find a way to pass it along.

And also I will try to see.

If I can post the whole package for download after all the conference is done. Another question you mentioned having a texture mapping option in the future for primitives. Will you have fine control over the orientation of these textures or will only support the quick and dirty text map.

Orientation? That is pretty good question. Currently the only way you can apply. Actually there do have several orientations that the text mapping.

For example, like a planner we do have provide parameters that you can apply.

X direction, Y direction, Z direct C direction and same thing for the sphere as well and spherical as well. And so other than that, if you are you. There's. There's always the. The possibility that you rotate the texture itself and then before you apply contribute to the texture.

Yep, that.

That answers all the questions.

So I think that will be it and the talk is done.

Enjoy rest of the talks.

Thank you. Bye bye.