Graphics: New Features and Improvements

Hello, welcome to this talk on improvements and new features for graphics and graphics 3D. I'm James Lane, I'm a graphics developer on the front end team and I'm going to be giving an expose of several of the new features we've done to make using the front end in terms of graphics even easier. Making it some things render faster, extensions we've done to primitives getting things to look prettier and as well as ways of exploring your graphics to kick off with. We've made texture mapping easier by introducing named vertex texture coordinates. And this is typically when you've got vertex texture coordinates, you'll specify a set of coordinates and there'll be numeric coordinates or you'll specify a function. Now you can very simply just say for instance, for a cone, you can say, I'd like to map my texture across the cone using a conical mapping. And you get this nice uniformly distributed mapping of the texture across the cone. One of the things we've also introduced here is an alias for vertex texture coordinates called texture mapping, which is, is quite a good name for what we're doing here. So for instance, you can also say I'd like a cylindrical mapping for a cylinder and some other primitives. And here you can see an example of this. It's mapped this texture across the cylinder. One thing to notice here is that the distribution of the texture is nice and uniform across the texture, despite how long the cone is so at the base and along the body of the cone. Yusu, the developer for this, did a really great job with some direction from John. We also support spherical mapping. And one of the interesting things is spherical mapping is that it takes a second argument. And this second argument specifies how the which axis would be effectively up. So for instance, here we could specify that we want the Z axis to be up. And if you look at the, the texture over here, effectively up in this texture is going say from the tip of Africa, South Africa up to Europe. And then that's been mapped for this first one running up the Z axis, which is, is what this argument says in contrast. Or another version of this is for instance this negative X which is this example on the end you can see the tip of Africa. At the bottom. Here it runs along the negative X axis. And that, that's the, the way that you can describe what's up for the texture. For graphics complex, we support planar, cubic and spherical names to accomplish a mapping across those graphics complex. For B spline surface, we, we also support, we support these named forms UV wrap, UV Unified wrap and UV Unified wrap with stretch. And the. You can see these last two do a really good job of preserving area across the B spline surface where UV wrap kind of squashes things here and things get stretched out there. I'm going over these kind of quickly. There are a lot of. A lot. There's a lot to cover in the talk, but I will stay in the room for questions afterwards at the end, just to answer anything, any questions that come up. To make things even easier. For the texture directive, we now support a second argument. And for that you can specify automatic and it'll choose the most appropriate named version or mapping for the shapes that follow it. So you can have a set of primitives and this texture will be mapped to all those primitives without the need to specify a texture mapping or vertex texture coordinates for them. So for instance, in this example, we can see that the texture has been wrapped across all these primitives and shapes. The default for this, by the way, is none, which basically then requires you to specify a mapping for the primitive to get that. That's the default, but you can say automatic now, which makes it nice and easy. You can also specify any of those other named forms for the second argument, and that will apply that mapping to all the primitives that follow that texture directive. So here's a plane mapping to all of them. You can see it's kind of stretched out the texture along the bottom. This box mapping is pretty uniform across all of them. Cubic mapping, cylindrical and spherical do funny things on certain shapes. Yeah, but this will hopefully make life a lot easier when it comes to specifying textures and working with them quickly. In Mathematica, previously textures, if you your vertex texture coordinates exceeded the 0 to 1 UV coordinate range, the texture would repeat as as it runs over the primitive it's mapped to. So we've introduced a new method option called Texture Wrap. This accepts the these arguments. You can now the default for this is repeat, which is the current behavior prior to version 14. But now you could also say that I'd like my texture clamped to the edge. And what that does is it'll use the color of the texels on the edge of the texture and carry those across the shape or the primitive. You can also say I'd like to do a mirror repeat. And this does a kind of reflection of the texture as it carries along across the shape. And then finally there's Clamp to Border, which uses the border color. In this case, it's light blue for the background and it just carries that across the Rest of the shape. We provide another method option to help speed up textures in certain situations. One of the developers, Kevin Daly, he, he's been doing amazing stuff with creating games with Mathematica. And in one of the scenarios he was doing a Doom type game and there were loads of textures. And we found that this was just slowing the front end down. So what the front end typically does with textures is it tries to be memory friendly. So when it encounters a texture, it will load that into memory and apply that to any primitives that need that texture. Then when it hits a new texture, it'll then drop this texture out of memory, load the new texture, and apply that one to what needs to be done. So we did this to just keep Mathematica memory friendly and not build up loads of textures in memory. But now you can override that with this casual textures method option. What that will do is in a scene like this, it'll just load all the textures into memory. And then when it needs one, if it's in memory, it'll just pick it out. Here it can switch between textures and pick them up without swapping between textures. This did show speed ups in the game. The thing to keep in mind here is there's a trade off between the memory footprint and speed. We've also accelerated rendering of things like polygons and tubes. There have been some situations where we've had millions of polygons that need to be rendered, or for instance, complex 2D polygons with holes that have been quite slow. And we've put in place a few things to accelerate polygon rendering. One of them is to we cache the polygon holes to speed up the rendering of complex 2D polygons. So for instance, in this case, where they are polygon holes, they get cached and then they can quickly be reused. That speeds things up. We also employ a stencil buffering technique to avoid tessellating very large concave polygons. So. So for instance, in this example, this is using a stenciling approach. So it just renders these 2,500 polygons super quickly. If we tessellated them, it would take much longer just to say as well, some of the other things we've done in the front end to speed up polygon rendering is that we've employed multi threading for loading and formatting of polygons. So. So for instance, we had a very large drawing, hundreds and millions of polygons, and the bottleneck was the actual loading of the polygons and formatting them. So Some of that's now multi threaded and we employ a background thread. When we construct certain spatial data structures for stuff like hit testing or when you move your mouse over and you want to see. You want to do mouseovers that. That are responsive to that. We set up data structures on the polygons and those are computed in the background. So you get them just in time. But you. We don't necessarily need them when we first loading and drawing the polygons. There's this community post where drawing millions of polygons is really. Or millions of tubes was really slow and using lots of memory. So we've done some stuff in the front end to address that. There's a two points method option. This was in version 13 and before a fixed default of 14:2 and it just specifies how the tube will be subdivided. We've changed that to automatic. Basically this speeds up rendering a lot. In this example I am using 2 points automatically, but if I say 2 points 42 it'll take a lot longer to render that. So what automatic is doing is it's sampling the tube based on. It'll decide the sampling to use to subdivide the tube based on the scaled radius. And so therefore it'll use fewer vertices and so on. For tubes with small radii. And this you can see, does yield a significant spe, but it also reduces the amount of memory that is used. We also do other things like release vertex normals when a graphics box is hibernated. There's one other tip that one can use as well to reduce memory such as in this example and you can use there's a method option called resolve as triangles. And what this does is when the tubes are being rendered, we'll often cache the triangles to try to accelerate the rendering of it. So when it comes to drawing them over again and if you set this to false, it won't cache that and that can for instance in that example from the community post, it reduces the memory a lot and so you can end up with loads of tubes being drawn really quickly and not using too much memory. In terms of extensions to primitives disk and circle. Now accept multiple arguments or multiple centers for this first argument. So this will create three disks. I've specified three centers and it'll create three disks there and two circles. So they now instead of saying disk, disk, disk, you can just have one of a single disk with those something to note those the disks all have the same radii and or radius. And if it comes to editing them, they all. When you Edit them. They all get edited as. As one primitive. So this is helpful because it can speed up rendering of load lots of disks. Like in this example, what happens in the front end is it's only a single disk box that has to be internally formatted and instead of, in this case, thousands. And that makes things much faster. We've also extended axis object to 3D. So here you can see a small example with three axis objects created in 3D. And we've introduced this slightly less verbose and I think, intuitive form. So this is an axis that runs along X. That's why the infinity and through y equals 0. X or y and z equals 0. Here's an axis running through x is 0 and y. It's an axis along the Y direction and Z is zero. The original form was like this. Which. Which you can say xyz, you know which axis you're talking about. But then you have to think, okay, it's X and Y are the other two. It's the mapping of the point that it's passing through. Just to say about 3D axis objects, they apply to all the same primitives that 2D axis objects do, with the exception of circle, because we don't do circles in 3D. And also it uses the same options as 2D axis objects. So for instance, tick lengths works exactly the same. There are nuances with the 3D axis objects to get them to work similarly to 2D axis objects, but to get them to work simply in 3D. So for instance, text direction automatic. It'll try to choose directions that are orthogonal to the axis but also orthogonal to the viewpoint. So for instance, in this case, you can see I was looking on this from this front view, but if, if I rotate around it, you can see it chose an up direction. If I release it, it'll choose a direction that's orthogonal to my view now and adjust those ticks. And there you can see they've been shifted to be orthogonal from that viewpoint. One. One of the. Some of the things that we do here is that we also try to reduce the. The. We minimize the rotation between ticks across the curve while trying to keep them orthogonal to the view. And yeah, so access object is still experimental, but should be in 14. We've also done some rendering improvements to reduce artifacts. And one of these improvements, for instance, is if you specify that you want pixels that are less or lines that should be less than a pixel, some of the drawing layers would use opacity. They would make those lines slightly. They'd make them more transparent to give this effect of them actually being thinner. And we've rolled that out across our other other renderers in terms the, the Z or depth buffer can introduce artifacts when it, it draws something at a, a very large, with a very large view range and if there's a very small view angle. So for instance, this planetary visualization, you can imagine if you set up a camera on Earth and you were looking at Saturn, you'd have a very small view angle to get this picture right and Saturn itself would be very far away from the the camera. And this, we were finding that this was introducing artifacts and we employed a multi pass view range range rendering technique or approach to solve this. And so basically what it does is internally we set this well internally we'll do this if we find that the view angle is really small and the, and, and the view range is quite far. But when you can set this as well by using this method option view range pass. And what it does is basically it just divides up the view range into however many passes you'd like and, and then it will render each, each section of the view range and then bring the composite or bring the results together to get the final drawing. And what this does is it, it just reduces, it improves the precision in the Z buffer which avoids the artifacts. Z fighting is and has been a thorn in the side of computer graphics for, for ages. And one of the options we employ in the front end to alleviate Z fighting is this method option relieve DPZ fighting. And it's been by default turned off. What it does is it adds a small offset to vertices in the direction of the normals and, and this helps to overcome it. But we've now turned this on by default. So now if we evaluate the same thing that Z fighting is overcome or conquered there it's not a silver bullet, but it does solve many problems with Z fighting. One thing to notice, for instance, here we're drawing two spheres on top of each other. There's no Z fighting. There's very little Z fighting. You can still sometimes see it coming through. But what it'll do is it does applies those offsets to the vertex normals or the vertices in the direction of the normals. And this ends up working like primitives that overlap in 2D. The last specified primitive will end up on top of the primitive before it. And we've rolled this out across several of our primitives and it fixes numerous Z fighting bugs that the front ends encountered. When it comes to exploring your 3D graphics, we've now Rolled out absolute current value and absolute option to work for all the graphics and graphics 3D options. So here's an example of using absolute current value I basically we evaluate this box and now we can query it the absolute current value of it. If I change this and re evaluate this, you'll see these values will update. And this is for options like View Matrix, which we hadn't covered previously. In terms of exploring and better understanding your view options, we are introducing a new Wolfram repository function called Visualize3D View. And basically this will work by saying Visualize3D View. And you'll just give it your graphic that you want. And then what it'll do is it'll render your graphic, but it'll produce this attached cell. And this attached cell has got these kind of three visualizations that help to better understand the viewing options. The first one here on the left displays the viewpoint, the view vertical. So the up direction of the camera, you can see the viewing frustum. And inside here is the plot range as well as whatever's been rendered. It also illustrates the lighting. So here we've got four directional lights and you can see the colors of those lights in the second view we have just a very simple illustration of the view angle and the view range. So here the near plane corresponds to this blue line, the far range to that line. And then finally we've got an illustration of the view center. Now I'll get to that now, but this is all interactive. So as I rotate my original graphic, the the different views update interactively. So for instance, rotating around a graphic, you can see the, the view center is staying constant and it's actually the viewpoint that's rotating or swiveling around that zooming. And I want to look at this for a different example to help understand some of the new 3D interaction capabilities that we have. For instance, dialing a camera forwards and backwards and look around functionality. Let's go have a look at, at this example. So here I've rendered a scene and all you're seeing, we're seeing is like this little, this single sphere. If we zoom into this and you look at the view frustum, you can kind of see there's a single turquoise sphere inside of it. So that's what we're seeing there if we zoom out of this graphic. So I'm going to zoom out now. So that's command. I'm holding down command and I'm dragging backwards. So now we can see in this picture here the view angle changes as we, as we Zoom out. You can also kind of get in see it happening in this view. As we zoom out, that frustum gets bigger or narrower based on how far we zoomed out. And just to contrast that to dialing the camera, if I hold shift and command down and I dolly the camera backwards, you can see dollying. The camera actually is moving the viewpoint that's changing, but the view angle staying the same. So with this visualization you can get a better idea of how the view options are actually changing in response to interacting with the scene. One thing to say about dollying the camera. So if I use my mouse wheel, it scrolls the notebook there. If this graphic is selected and I hold down the dolly camera keys and I scroll my mouse book, it'll using the mouse wheel, it'll now do the same as if I, if I dragged. So the nice thing about. One of the nice things about that is the cursor just stays in the same place. And yeah, it's just, it's. It can be quite easy to do. And this is a technique used in several 3D packages like Blender or AutoCAD. One of the new options that we have is the look around camera. So look around. This is really useful to understand the difference between rotating around versus look around. So look around you'll see is rotating the view center around the viewpoint. Notice how the frustra itself is moving away from the plot range. Frustum plot range. So look around is actually rotating the view center where if I rotate around this, it's rotating the view center or the viewpoint around the view center. One of the things that we've improved here with the look around camera is if I drag on a certain point and I move it, that point that I drag on stays pretty much underneath where the cursor is. So that's an improvement from version 13. Just to illustrate this view, if, when we pan the camera, if you click and you pan the camera, you'll notice that nothing changes in the these 3D settings. The viewpoint stays the same. A pan is actually achieved by a screen space space displacement. So as I drag it up here, you can see it's applied some screen space delta to. So it takes the viewpoint in 3D space and then it makes sure that lines up with this delta in image image space. So that's I think helpful to understand how panning works in the system. One of the other options, some of the other functionality we have here is viewpoint picking. So if I hold shift and the option key down, it will do view center picking. So if I click on a point of interest, it'll move that to be in focus. And what's happening here is previously it would just jump. So the improvement to the system is that it will now smoothly transition to that. You can see in the bottom image as I do that it transitions and then. And then the mouse wheel also works here so one can dolly the camera in and out of a point of interest, just more. More smoothly and intuitively. So we've introduced the mouse. You can use the mouse wheel for these interactions. Look around is option command and drag. And that drag vector stays tightly underneath. Underneath the pointer. And the new 3D visualized 3D view helps to better understand interactively those navigations and it's useful for understanding a 3D graphic scene. Maybe you don't see why something's rendering and then you'll be able to use that to see. Maybe I've got no lights on my scene. Just to mention tomorrow there's a front end office hour and that's going to be a great place to ask questions. A lot of this, the work that's been done and that I've just touched on in this was done by Yutzulu and it's a great opportunity to be able to ask questions where we can take you deep into what we've done and yeah. So I'll just draw your attention to that. I'm going to end the presentation and then just take questions now. So some questions here on the side. Thank you as well for coming to the talk. You're welcome to stay in the room. I know it's. I'm over time now but I'll. I'll answer the questions and you're welcome to stick around. So one of the questions is I'm just going to go through these from. Okay. At the bottom. Thank you Sylvia for the question. Is spline Access object in 3D also supported? Yes it is. B splines bezier curves just like as in 2D. They. They all. They'll all work in 3D and they'll try to minimize the. The variation between tick directions across those if the tick direction is automatic and. And so on. Yeah that. That should be in 14 but it's still experimental. We ironing out one or two things. How then there was. Is there. Ian asks is there a simple way to scale an axis say the z axis in 3D by some factor N such that the graphic displays with the x and Y axes at natural scales. Box ratio is automatic but the z axis displays at N times natural scale. I currently use absolute values to get the box ratios which I manipulate. But this is a bit messy. This might be better answered by Brett Champion. But. And let me think about that. The z axis. Okay. So the box ratios are automatic. I might need to get back to you on that one. Ian. This, this would be a good question to direct to Brett Champion. Sorry. Then Sylvia also asks, could we also ask for dolly zoom or Hitchcock shot camera control? I will have to. To look up what those are. But are dolly. Okay, so the. As you move in the. The. You mean that the, the view angle changes while you. You're moving the camera forward or backwards. But you can, we can definitely try to do that. I'll. I'll follow up on that. And we, we Then Thomas asked, how about a drone camera and representing its view path. Bezier control paths might be nice. That would be really neat at the moment. I think you'd have to set up your own. You'd have to kind of set up a bezier by yourself and then use maybe dynamics or so to get the camera to move along it. But if you mean built in functionality for that. Yeah, that would be really cool to do. Then there's a question. What about logarithmic axes? Very good question. So Minsang Peng's been working with scaling functions and we can use scaling functions on axis objects. And that will allow you to do logarithmic scales and have logarithmic scales on your axis objects as well as. As any kind of scaling function. Yeah, so. So that is. That will be. That's not fully complete. In, in 14 you can do some logarithmic scaling on axis objects. But we'll, we'll roll out the final version of that in the next release. Okay. Sylvia says Mari's box ratio is automatic. Automatic four. Okay, so Sylvia has answered one of the other questions. Oh, thank you, Sylvia. And then there is. Read the drone camera question. Okay, John's answered that. Okay, John's answered that. So the video team's looking at the drone camera question and yay. I think that's all the questions. Thank you so much again for your questions and for sitting in on the talk. I hope that you have a great conference ahead and remember the front end office hour for more questions where. Yeah, you'll be able to get even better responses than we're given here. Thank you so much everyone. Have a good day. Bye.