3D Game Development in Wolfram Language

Hello everyone. Thanks for coming to my talk. If you don't know me, my name is Kevin Daly. I'm a user interface developer at Wolfram Research. I normally make interfaces like toolbars, dialogs, etc. But I'm going to be doing something a little non traditional. I'm going to be using graphics 3D and dynamic to make a video game today. So this is sort of a spiritual successor to a talk I gave back in May which I've linked down below where I made a 2D video game using notebooks in the Wolfram language. And so I'm going to be building up a 3D game in this talk and it's going to consist of all the things you would expect inside a game. Basically using the notebook as a game engine. There's going to be graphics, physics, user input. I'm using a Microsoft controller just because it has Bluetooth connectivity and it's easy to hook up to my Windows machine. I'll show collision detection, etc. It's all going to be there. And just to preview that I am not lying about anything, I am going to turn on the controller and show a game in action. So here you see, I've got a heads up display. I can move around in this 3D game world. I don't clip through walls. The unfortunate thing is you're not going to get sound through this webinar. But basically if I hit the X button you hear a pew pew sound and so I can take out these crew member targets and if I walk over these spheres you'll hear a bloop and it'll increase either my armor or my health, you know, walking upstairs. And of course my ammo goes up and down as I use the button. So this is just a quick preview of what we'll get to by the end of this talk. And what makes this all possible is an extra method that was added in version 14 cache alltextures goes to true. I think Yutsu or James mentioned that earlier there's other methods that have existed that also make this possible. I have been using sort of pixel art level graphics and so texture sampling goes to nearest is useful as a method option. And then because my camera is within the box of graphics 3D it's good to turn off any edges. So edge depth offset goes to false. Another thing that sort of makes this possible is I developed a resource function that lets me import geometry from the web in a useful and convenient way that doesn't throw away any texture data. So the game we're going to be building is in graphics 3D. If you're not familiar with it, this is a way that you can place primitives inside the world like cubes, spheres, etc. You have control of a camera. You can rotate around the viewpoint. And what I'm going to be focused on is making a whole bunch of polygons, these 2D surfaces. And we want to move objects, including the camera, around in the world. And how we're going to do that is through dynamic, a very quick refresher of dynamic. What it wants to do is display exactly as it is with the state of the variable. Actually anything that's wrapped in dynamic as it is. So it's showing up here as 0.5, because I've set it as 0.5, the symbol x above. And you can do neat things with this. Like I have X here as the control variable in the slider. I move the slider around and you See it updates output109 as fast as possible. So there is a cousin to dynamic dynamic wrapper. This gives us a bit more flexibility because now instead of the display being sort of tied with the expression, we now can separate the expression that's going to be updated with what's displayed on the screen. And so I can still interact with that thing, which is just resetting the value of x to zero. The other thing this is demonstrating is a runaway dynamic, which is usually something you want to avoid in a interface, but I'm going to actually use it to my advantage. So by runaway, I mean the symbol X gets updated, which then triggers. The system realizes it's updated, so it needs to reevaluate the entire payload, et cetera, and just does it indefinitely. One last thing in the dynamic family is dynamic module. This is a convenient way to localize the symbols. So you see, even though I have X here and a global X here, the hue, this color here is updating independently of this button, because I have localized that symbol within the dynamic module. And with just those three things, we can make a first game. So what I have here is just a little shuffleboard. If I press and hold the control key, I can get that disk to slide along this ground here. The goal of this game is to basically have it hit the blue area. I lost, I guess. But the point is that we're using dynamic module as a localization construct. D Dynamic wrapper. This is what's driving the game forward. It's running all of this code as fast as possible, and then it's updating a position which what we're actually displaying on screen is just sort of lazily updating that position and then drawing accordingly. One other thing to notice here is a time factor. This is needed because as a runaway dynamic it's going to run as fast as the CPU allows, as fast as dynamic runs. So this is a scale factor that converts from basically the updates per second into roughly a game speed, game time. But we don't want to have this outside view, we want to have a first person point of view. So we need to understand how the camera works in graphics 3D. There's already been great talks by James. He showed some pretty nice camera stuff. There's also some community posts that I have up in this first paragraph here. But basically I'm going to be focusing on the view angle, view vector and view vertical. And actually the view vertical is going to be locked at straight up and down. I'm not going to have any tilt to the camera. The view angle, you can do some fun things with it. So what I'm showing here, the outside point of view, this is the camera frustum. I'm not showing the front face. Frustum is a pyramid with the top chopped off. So I'm not showing that here, I'm just showing the whole pyramid. But if I increase the view angle you can get kind of a fisheye lens. But we're gonna probably keep it about 60 for the rest of this talk. And then lastly the view vector, which is a list of two points. It is the camera point which is the peak of the pyramid here and then the direction you're facing. Now this aim point can be any point away from the camera point. But conventionally for this talk I'm going to keep it a unit distance away from that camera point. So with those three things in mind, we need to start moving the camera point around. So I'm going to consider a body fixed sort of coordinate system where the camera is the origin of this fixed coordinate system. And then the aim point is a unit distance away. You can project that down into the plane and and that gives our forward and backward motion. And then I can take the perpendicular to that which gives the transverse motion. And so with those two vectors we can translate the camera. So here I have the view vector at unit length. I have the forward motion, I have the transverse motion and I'm putting those into a dynamic right hand side for the view vector. And so long as I translate both the camera point and the aim point by the same amount, I effectively slide the camera around in the same orientation. So that's how we're going to do a translation of the camera. But we also want to be able to look around. So we're going to use some of the same vectors. We have the same forward and transverse motion vectors. We've added a localized vertical vector, and we're going to have a combined rotation around both the vertical and the transverse vectors. So I'm going to keep the camera point fixed for this demo, and I'm just going to be updating the AIM point, which is the camera point plus some direction. So now I can sweep the camera around and I can sweep up and down, and it's going to be the combination of those two rotations that will give me the full, like, 360 viewability. Okay. But we want to also connect this to a controller. If you weren't aware for a long time, we can actually have been able to communicate with external devices. There's a number connected to my machine right now, but I am focused on the Xbox 360 controller. And you see, as I do the joysticks, things move on screen. We have Boolean buttons. That is amazing. That's awesome. No effort. It just happened for free. And we access the controller information via controller state. So this is the function I'm going to use to access where the two joysticks are. And then I have my slider buttons on top. And just in case. I doubt it's going to happen, but if my batteries die or I disconnect the controller. So if I ever fail to have a controller state, I'm going to wrap all of my future demos with this function. That'll just kind of gives me an escape hatch so I don't get just runaway errors. And you might have noticed the joysticks did not come back to zero. So that's known as joy drift. Joystick drift. And so we can just chop that away. And so I'll be doing that in the rest of the talk as well. Okay, so you're gonna see a bit of code spew, word of warning, but don't be alarmed. This is actually everything we've seen in the previous slides. We get our controller state coming in. We are cutting off any drifting of the joysticks. We have our view vector, forward motion, transverse motion. But the only addition here is now we're taking the input from the controller, adding some velocity and time factor to it, which gives us a displacement. And our aim is just two rotations, one after each other. But this is everything we've already seen before, just put into a single function. That we can now use. So here I'm going to make a world of cubes. And so now as I move the controller around, I can fly around this world. There's no hit detection yet. We have to build that still. But this is a great milestone. So now you already know how with just this is roughly 100 lines of code connect a controller to your graphics 3D. See, but we want more, we want hit detection. So this could be a talk in and of itself. I chose to implement the Gilbert Johnson Keerthy algorithm, which it's a really efficient and extensible algorithm for calculating distance between objects. I'm only going to take the predicate of this, so I just assume there's a function which I've called impact cue. You put in two objects and it tells you yay or nay, are they touching each other? So I'm going to create an octagon of polygon surfaces. And so here I have a sphere which could say, represent the player character moving around a world. And if we have an impact, you see, I have. If impact is true, I color the impacting surface red. So just a demo to prove to all of you that I have hit detection implemented. But if I were to eventually build a world with hundreds or thousands of surfaces or objects, it would be extremely inefficient to always be querying whether or not these things are hitting the player. So we need to do something more efficient. So what we're going to do is partition the world. So in Minecraft you might hear this parlance of blocks and chunks, but we're going to create a sort of hidden data layer which gives us a map of where all of the static surfaces are. So what I mean by this, in a 2D example, let's say a polygon exists in the world. This is now overlaying this hidden data layer on top of it. And then what we can do is draw an axis aligned bounding box around this thing and then that gives us basically the membership that object in this, you know, map of the world. So it becomes extremely efficient to just query where things are. So what this data layer ends up looking like. So again, this is my octagon example. Surface 1 is in this block, Surface 8 is in this block. They're just, you know, integer indexed. I have a fixed block size. I get my world size in there as well. It's all good. It's this nice data layer I can query as I run my game what this looks like in the same octagon example. Let it bounce around a bit. Here we go. The logic that we're going to have here is again, assuming this sphere is the player character, we first ask where is the player character? Then that gives us the nearest neighbors in this invisible data layer. And then all we have to do is look up in this basically lookup table which surfaces are nearby. And so in this example, the nearby surfaces are colored yellow and the true and then we do the harder calculation of where the hits actually occur. And so that's this red surface. So we basically excluded 99% of all of the surfaces and objects with a very, very fast lookup. And then we do the hard thing. And we're eventually going to move to not just these octagon colored surfaces. We want to go the full textured polygon route. So I got to remind us of how to use textures with polygons. If you're not familiar, you can use vertex texture coordinates. This is also known as the UV coordinate system. It spans from 0 to 1. And what it gives us is if you say, I have these square polygons here, you'll get the full texture back if you go with the full range from 0 to 1. If you go beyond 0 to 1, you get a repeated texture as shown in the middle here. And if you do something smaller within that range, you basically get a cutout of that texture enlarged. And this is great. I'm using all the same texture, just cut out in different ways. And so I can move the camera point around. It's all smooth and fun, no problem there. One word of warning, if you are using, say pixel art, so very low count for your images of pixels. The default is to blur those together, but you can use a graphics 3D method option. Texture sampling goes to nearest and this will give you a more truthful representation of that pixel art you were aiming for. Last word of warning, with textures, you might be familiar. As soon as you use more than one texture, it kind of hurts because now as I try and move the camera around, it's sluggish. I'm not sure exactly how many what my bitrate is in this webinar, but that's pretty slow. And so what saves the day is a new feature. A new method option in 14 cache alltextures goes to true. So now we're caching more of the textures. And now this is back to smooth motion keeps going off screen. So if you're in 13, 3 or older, there is still a way to do this. You basically have to merge all your textures together and then cookie cut out each individual thing. Which I did before. I just asked my colleagues for help. And so you can do this. I have the code here in this bonus section. So you could potentially even do what I'm doing prior to 14. I'm running a version of 14 right now. So another thing that I developed that makes this easier, I have a resource function that lets me grab stuff, grab geometry from elsewhere on the Internet, because I am not a very good artist. And so here is a map of the first Doom level. And this comes from VR. Chris at sketchfab. One of the things we kind of have to worry about though, is actually I'm gonna reload this where you get things from. The Internet matters. And that might not be in the same coordinate system that you're expecting. So using my function, I pulled this. And it's not. I mean, it looks flipped. That's because this game world was probably along the Y axis, but my game is along the Z axis is the vertical. And so just with a couple of rotations though, we can get this back to where we need it. So here we go. That's the game world I was expecting. Okay, we're gonna make our surface data, but we're not quite done yet. So now we've got our surfaces. We know hit detection works, but there was a little hiccup that I experienced when making this. If you have steps in your world and you don't jump, you can't even get over the tiniest hill. And so what I mean is here this red line represents, say a given displacement. If you hit a surface, I'm just going to take basically the triple vector product, which gives us the component of that motion along the wall. But if I were to stop here, every molehill becomes a mountain. So a trick that I'm going to do is I'm going to take any vertical surface and I'm just going to kick out the bottom vertices and turn everything into a ramp. Because I can run up a ramp, I can't run over a step. So with all of that together, we now no longer have free motion. So now I have this movement function which still uses my free motion code. I just take out the Z axis, but now I'm going to find which surfaces I'm colliding with. So here's the player characters code, and then of the things I'm hitting, I'm going to just resolve the surface collisions as I explained above, and update the view vector's list of points accordingly. And so what this looks like convert ramps. So here we go. So this is again I would say probably it's pretty smooth. I'm not sure how well this comes across in the webinar, but you see, I can run up steps. These are not mountains anymore. But what did I forget? I forgot gravity. So let's implement that. So we're going to update our movement function just slightly. And this is actually similar to the same stuff we saw for the shuffleboard example I showed way at the beginning of this talk. We're going to add an acceleration in the Z direction, and that's just going to constantly pull us down. And everything else is the same. Same movement, same surface collision calculation. This line turns out to be pretty important. I didn't have it when I first built the game, and so the vertical velocity got so negative, it just slurped me right through a surface. So what this does is if you're standing on a surface that's pointing straight up and down, just set the velocity to zero. Pretty basic. So we've got gravity now to prove that it's working. I can now walk forward and go up and down these steps just fine. But we want more. We want interaction with our game. And so I'm going to talk very briefly about line of sight. The view vector defines a ray that goes from the camera origin to infinity. We can super cover it with the invisible data layer, these blocks, and then we can walk along the blocks and query at each one, are there any surfaces? And if so, do we pierce them with this ray? And so that gets us something kind of fun we can actually output. I've added a crosshair to make this a bit easier to see, but we can actually see all the polygons that make up the world highlighted in red. And so we get the index of the surface. I'm looking at the point in the plane, so I'm staring at the ground, it's at zero. I go down a step, it goes negative in the Z direction. And then you can also kind of funnily see the surfaces. I'm actually running up our ramp. So this is. The red shows us the actual surface collision. But even though they're drawing as steps, I run up a ramp. We want to do even more. We want to do some target practice. So I'm going to add yet another data layer. I'm going to draw some targets that look like crewmates. And then I have hit detection enabled for cylinders. Because even though I'm going to be shooting at a billboard, if you rotate a billboard around its center axis, it sweeps out a cylindrical area or volume. And so I Just did hit detection for cylinders because that was a lot easier to implement. And then we need a few more things in the movement function. We need to know the player's current angle based on where they're looking. We have to add a button. And this is the more complicated thing. We don't want the button to press all the time. And so we have a logic in here that basically says, is the button held down? If so, shoot once, do nothing else. And so as we press the X button, we're going to play a sound which unfortunately you won't hear today. And the previous demo I had was doing line of sight all the time. So this is now only going to happen when you press a button. So we're going to find the blocks along the line of sight and then we're going to find what surfaces are in the line of sight and then what cylinder primitives are in the line of sight. And if the surface primitive, cylinder primitive is in front of the surface, then that means we got a hit. So I'll play a ding sound and then exclude that cylinder from hit detection going forward. Okay, so then what this looks like. So now I can run around this level, which is now populated by crewmates. I press the X button, which then if I have a hit, I can then eliminate that target from the game. I do not have a good aim. Okay. Last thing I want to add is a heads up display. This is just a convenient way to keep track of what's on the screen. One thing to note is the text here. I'm actually using haloing to make it this kind of boulder effect. That was a feature added in 13.3. These are actually more like the interfaces I'm used to building. Give me a couple of sliders and some text and some dynamic and I can make something look nice. So we're going to reuse the same locations as some of these targets we had. I'm just going to replace them with health and armor pickups. So this is yet another data layer items which are now spheres is the thing I'm going to have hit detection with. And then this is the repeat of what you saw at the start of the talk. Oh, I forgot to actually rebuild my targets. Shame on me. Oh, there we go. So now if I walk over spheres, you hear a bloop sound, it increases my health. Or if I walk over the green spheres, it should increase my armor. One other thing I should mention, I actually place these different things in the world, both the targets and the spheres, using a geometric transformation. It's a very convenient way to make multiple copies of stuff. Not just convenient, but I'm pretty sure it's efficient too. So, yeah, actually, that's as far as I intended to get today. Thanks for your attention. I'm going to now check to see if there's any questions. Some of you may have recognized that this is the first level of Doom. I should say it's a facsimile of it because, yeah, it's Creative Commons license. I don't know who made it. Well, V.R. chris made it, but Evan asks, do I know how many frames per second it runs at? You can actually time this again? I don't know how. It's being presented through this webinar. It's running smooth on my machine. That's about what I can say. The fun thing about this time factor. There it is. Is you can adjust this based on how fast your CPU is. Oh, I'm going to live dangerously. I'm going to double this and then rerun this game. Oh boy. Now I'm like running at lightning speed. And so that it really is a time factor. If I've implemented it correctly, changing that will make the game appear to move faster or slower. Let's see if there's any other questions. Cheeky. They asked if multiplayer is coming. I don't know. It's also, this doesn't run on the cloud either because this is really hammering hard that connection between the front end and the kernel. Getting back to actually the FPS question the frames per second, it is probably possible to implement something that's checking how fast the updates are happening and then in real time sort of trying to adjust the time factor every, say 10 updates, it could take an average of how fast the game is running and then reset the time factor. Yeah, that could probably work. I don't see any other. Oh, can I open doors? Oh, Kyle. Yeah, okay. I wasn't going to get into this. Oh, I should actually cite all the people that I actually used here. Leave that up for a sec. So Kyle asks, can I open doors? Yes. Did I implement that for this talk? No, I did not. So I made a developer tool as I was making this. So what I mean by that is here I will make a new game. So I made this developer tool that as I walk around, I use my line of sight and I actually placed things on the ground. So what this is doing is this is giving me a new data layer which I called logic. And so what this base, I built the world like all the targets you saw. I didn't just pull those origins out of nowhere. I literally walked around and placed enemies and pickups and so that let me actually very conveniently build the world out. So getting back to can you open doors in this developer mode? I did the same thing. I built this thing that lets me open doors so I could basically make any surface a door that I wanted to. So I could make this a door. Well, I need to add a door first. That's right. So I could make that a door and then it takes the surface away. And so you this. It gets complicated because you have to add multiple surfaces to a single door. But once all those surfaces are added, so you see here surfaces 406, 366, etc. There's a bit more development to do. But yes, I did develop doors that open and close, but it was just way too complicated to get into the weeds for this talk. Matthew Adams asks what's the downside or trade off when using the texture caching method? The trade off is that it actually works. If you were to have even one extra texture in Mathematica 13.3, the camera would just sluggishly move through the world at like maybe five frames a second. It was brutal. And so by caching all the textures, I can actually move the camera smoothly. At least apparently smoothly through this graphics 3D. Andy asks, is it easy to have user rotation or interaction with objects and graphics 3D? Interpreting your question, the rotating billboards turned out to be very expensive, so that's why I put them inside of the. I just forget what it's called. The geometric transformation. Rotating the player though, that's very smooth. Like that's just a couple of rotation matrices that I've pre calculated. That's not bad at all. Oh, okay. So to clarify, what Matthew meant is why isn't cache all textures on by default? We debated that, but then we decided because of memory issues or possible memory issues, that we'll let the user decide to turn that on. Most of the cases, people are probably just building worlds with single textures anyway because this is more of a technical presentation, technical software. I'm using it in a very non traditional way which requires having multiple textures. And does this texture caching work on the Minecraft demo someone made 10 years ago or so? I know the demo you're talking about. I almost made a Minecraft demo instead of Doom, but I don't know. I mean you can turn the caching on, but if I recall from that Minecraft demo, he did the technique of having one texture where he cookie cuttered all of the different individual textures out, but you saw how smooth my Cube World demo went. I wouldn't be surprised if a Minecraft level would be fairly smooth nowadays. I don't want to take people away from other talks because I've hit the end of my time. Thank you all for attending. This was a lot of fun to present. Hopefully you can take the code that I have in my notebook, which is in the handouts forgot to mention, and start using that in your own exploration of graphics 3D by ALT.