Unity Performance Profiling Ninjas with Ruben Torres Bonet

Foreign.

Get ready to discover some of the most actionable performance engineering testing insight reliability device for some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE podcast, and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome back to another episode of the Test Guild Performance and Site Reliability Podcast. Today I'm going to give you a special replay from a session we had at this year's Game Dev Guild. I'm not sure if you know that I do an online conference with a influencer in this space who's a game dev specialist, Jason Wyman. And so we're going to actually do this every year. So we had it this year in May, we're going to have it next year again. I believe it's in May, so definitely stay tuned for that. But this is a session called Unity Performance Profiling Ninjas. So in this episode, Ruben, who is a Unity performance expert and runs the website the Game Dev Guru where he has a really cool program on a performance task force that can help you with your performance issues, does performance consulting, whole bunch of cool things around performance, but specializing in performance of game development. And in this episode he's actually going to share his approach to solving gaming performance issues. So you're going to discover his three step workflow to find your bottlenecks and how to craft an action plan within minutes. And you're also going to have takeaways to probably help you with any type of performance bottleneck. This just happens to be geared towards unity performance issues, but obviously a bottleneck is a bottleneck, so you're definitely going to pick something up. Either way, you definitely don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why Smartbear created load Ninja, a SaaS load testing tool to help teams get full visibility performance so you can release quality software that faster than ever make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more.

Welcome to this talk on how to become a Unity Performance Profiler Ninja, which is part of the Getting Juicy Cash. As a performance Expert series, I will give you the mental tools that you need to destroy the performance bottlenecks that are haunting you 247 very quickly to let you unlock higher tier jobs and gigs if that's what you are into. And I will give you proof of that. But first, who am I to speak about performance? I am not yet another just content creator. I personally want to be always useful and I cannot do this if I hadn't have enough experience. First, I am a game developer. I started on the mobile sector making games for Vuga, you know, games that have millions and millions of players. And I also collaborated in bigger games for bigger studios like Assassin's Creed for Ubisoft, like Star Trek for Ubisoft as well like a lot of games for Forcefield. You know, there is a lot of games that I have contributed over my lifetime. I have been always focusing on performance optimization. The second thing I am as well is a performance consultant. Because as you will see, there is a real need in in this world for people who know about performance. Otherwise I wouldn't be paid every single week to do this kind of stuff for different clients, right? So basically here I help studios destroy the bottlenecks that are slowing them down before they are able to ship the games that they want. The third thing is that I am a high performance trainer, which means that I help studios and I help people as a mentor understand how performance optimization works to get the frame rate that that the players demand. I have done training for Unity and Unity clients have done masterclasses for XR Bootcamp. I also maintain my Unity performance task force. A lot of content on optimization that has members over 25 different countries and I have hundreds of posts in places like gamasutra, gamedev.net and even in my own website, if you know me, which is called the gamedev Guru. And I tell you this, I only got here because I committed long time ago to to continuous learning, okay? I committed to do the things that other developers didn't want to do. In this case, it's performance optimization because it's hard. If you want to get to this point as well, where you are able to work for any studio that you want and get paid for it, a lot of money for this, then you have to commit to apply the learnings that I'm going to give you in your current and future projects and just know that this is not going to happen automatically, you have to do a conscious effort, right? Just like with any habit, right at the beginning it is hard. You need to do this manually, but at some point it will become automatic. Just remember that these learnings I have gotten over decades of experience doing performance optimization. So don't take it lightly, my friend. Let me address first the most common objection. The group that I call the non believers of performance optimization. Chances are high that you will come across people who tell you premature optimization is the root of all evil or you don't need about optimization nowadays, all of these things there are. This is crap. And the reason for this is because if you have no problems with performance ever in your life, chances are high that you are not pushing your game hard enough in terms of features, quantity, quality, everything. And of course, if you also set your minimum specs to an RTX 4090 Ti and a Ryzenoff seventh generation, where of course your game is going to run well in every single computer, right? But then the problem is that you are forbidding many players from playing your game because you set the requirements too high. Of course your game is going to perform well. In that case, the key thing, the key takeaway is this. Performance will always be a problem if you do things right. A problem or a challenge to take care of. Okay? And if you want even more proof, here's more proof I have every single week cleanse who are asking me for help in terms of performance optimization. If the employees or the game developers did this right, or knew how performance actually works well enough, then they wouldn't need to pay me large sums of money. There is a huge market, a huge potential for developers who know about performance optimization both if you want to be just a game developer as an employee, or if you want to be a consultant or a freelancer. And I'm not a magical person. I started like a noob making web development for. For less than five bucks an hour. And I had no idea about game development. I had no idea about performance. I didn't know this is how it started. And now I got to a place where I charge a lot of money for consulting and I can work for every single company that I want out there. And I don't say this to brag. I say this because if I can do this, you can also do this. You just need to commit. Now let's get started with the truth behind performance. Truth number one, there is no single bottleneck, okay? Bottlenecks, always evolving, evolve over time, evolve over different runs, evolve over everything, even between frames, because we always target different hardware and different software, right? You don't know where your user will be running your target game on, okay? Unless you are running, of course, just for one specific device, like an Oculus Quest, for example. But even then, from day One to day two, the performance might change because as happened to me already. Yes, in a matter of days, Oculus pushed an update to the operating system and my performance dropped like 20%. It also depends because the level or the gameplay state changes. For example, level one might be indoors, Level two might be outdoors, or even within the same level. If you have AI, you know, AI tends to be quite random, sometimes might change the way things are displayed, and might change as well, your bottlenecks. Finally, the bottleneck might change because your device status might change. Just think of, for example, the temperature or the battery level. If you have low battery, chances are high that the operating system is going to tell you, hey, dude, relax. With the cpu, it's going to tune down the frequencies and you are going to be suffering from extreme performance loss. Just think of the CPU and GPU levels on Oculus Quest. Things like this will change your performance bottlenecks, even between frames. Okay? So you have to be careful. Key takeaway is that you must dominate profiling because you will have to do this over and over. Does it make sense? Let's go to truth number two. Great intel leads to great optimization, or no intel at all leads to poor optimization. First, you need to understand everything about your bottlenecks. And only then when you know everything about your bottlenecks, then you can do the right optimization. I tell you this because this is one of the reasons I have so many clients. They do it the other way around. They start optimizing and then realize they are not getting the gains that they need and they start panicking and they start putting thousands of dollars on people like me because they don't know how to approach it. So you need to do profiling over and over again. Having said that, what are the steps to performance optimization? Three steps. One, profiling. This is what this talk is about. Second is planning, which is, once I know everything about my bottlenecks, what are my options to deal with this performance bottleneck? And third is perform, which is, you know, I choose this tactic and I execute on that. Then I get rid of my performance bottleneck. Okay, so the profile phase. The profile phase is all about answering three questions. Question number one. Where is my performance bottleneck? It can be in any of the four performance pillars. Professional pillar, cpu, GPU, or memory pillars. We will see this in more detail. Second question. Why exactly is this actually a bottlenec? Need to understand the why of this. And third, once you know this, you can craft your intel briefing. So let's get to the first question. What is your Performance bottleneck here. What we need to do is to find in which hardware components we have this bottleneck. It can be in the professional pillar, right? You are the bottleneck in this case. It might be that you are taking too much time to develop something or that you are waiting too much time across different iterations. Things like this. Okay, Maybe that you have developed build system or no build pipeline at all. Right? And you don't want to be waiting four hours between builds. Right? So that's the first pillar. Second pillar is CPU pillar. Things like AI, physics, animators, networking, things like this. Third one is gpu. So basically graphics. When you are rendering polygons, you are shading fragments and all of these things. Fourth one is memory. When you are using too much memory, memory capacity or you are transmitting too much memory, memory bandwidth or too big build sizes. Right? So how do you know where this is coming from? You know this by doing what they call playful experimentation. And then you can also use generic profilers. Okay, so what is playful experimentation? This is a technique that I like a lot because it's fun and it's very fast. It is all about doing something and seeing the result of that. For example, if I don't know where my bottleneck is, I might decide to, you know, just activate or deactivate entire hierarchies of game objects and see what the effect is on the profiler. Right? I can also say, hey, let's reduce the work that the CPU is doing, for example by turning off AI and see if performance improves. If that's the case, then you know, this bottleneck might be on the cpu. I can also say, hey, let's reduce the work that the GPU has to do and see if performance improves. For example, I could reduce the resolution from 4K to 10 pixels and see what's the result of that. I could also work on the load bias, etc, etc, etc. If you are doing this in the editor, it's pretty easy. You just need to do the switches in the editor hierarchy, right? But if you are not, if you are doing this on a target device, then you can use custom scripts that turn features on and off, for example every 20 seconds so that you can see the difference in the profiler. You can also use tools that I like which are for example the NG remote scene or remote hierarchy tools, or HDG debug. Those are very cool ones. So let's see an example in Unity. Okay, I don't want to get too philosophical about this. So here I might see that my performance is pretty bad. Right. It's like 12fps and I don't know where it's coming from. So what can I do? Playful experimentation, my friend. What can I do? For example, I can check is this coming from my cpu? I'm going to check the animators, for example, and I'm going to disable them all. Okay, Another question is, is my frame rate much better, 50fps? No, I would say not. Maybe there is a marginal gain. Maybe there is even nothing. Okay, so let's assume it is not because of my cpu. I could also do, of course, some other changes, like to reduce or to remove some scripts or the AI and such. But I want to keep this short. Okay? So the other question is, could it be my GPU itself? How can we test it? So what we are going to do is to reduce the resolution from three aspects, which might be almost 2K resolution, to 10 by 10, and see if the frame rate increases. Indeed, that is the case. We can reach now even 30fps, and that was not the case. Now, what we learned from this is that indeed, just by doing quick testing, the problem might be on the GPU side of things. This is all about moving fast. Okay, we need to really quickly find out where the performance bottleneck is, right? In which hardware component, and in this case here, we are bottlenecked by the gpu. Once you have more experience with this, you will be able to tell more of this information just by looking at the profiler itself. But until then, playful experimentation is a really good option. One option that I still execute on nowadays. Okay, it's very important that you learn this technique. Now, the second question is, why exactly is this bottleneck a thing? So by now you know where your bottleneck is, but you need to know why this is happening. Otherwise it is as if you knew that your arm hurts. Okay, it's hurting here, but you don't know why. Maybe it's just, you know, that you burned yourself with the sign. Or maybe it's just a pimple, or maybe it's cancer. Okay, I don't know. You really need to know why. Otherwise you cannot do anything about that. So how do you know this?

Why?

By using two things, specialized profilers, and by doing research. Okay, things like doing online research, asking on discord, reading blog posts, checking on the Unity forum. And you can also even join the UNITY Performance Task Force, which is my huge library on performance optimization. So by using these tools, you should be able to find out why you have a bottleneck. And using your brain, of course, that's the most important tool. Okay, so what are these specialized profilers I'm talking about? Well, it depends on the performance pillar where we are having the bottleneck in. For example, if you have a bottleneck on the professional pillar, you have several tools. The simplest one is a timer. Whenever you find yourself waiting too long for something, just press a timer. That will help you measure where the problem is. Exactly. Then you can find the reason for that. Okay. By using other specialized tools, you can also use the profiler on the editor as well. Okay, so in other words, you can profile the editor itself. This is something that is very powerful and I see barely anyone using if your builds are taking four hours. You can also profile your build pipelines by putting timers around the process and seeing what's happening. If you have long iteration times, for example, if you are waiting for one minute when you start playing or stopping playing, then you need to profile why this is the case. And for this you can use the Unity eip, for example, the Editor Iteration Profiler. Those are specific tools that you can use. For example, this game takes a long time to stop playing. In other words, what we have is a long iteration time. So what we can do is to use the Unity eip, right? The UNITY Editor Profiler. And when you activate that, you will be able to find information about the exact reason this is taking so long. So in my case, for example, I have awake scripted objects, right? This is taking the biggest part of the chunk, right? And I see stuff that I might have control over and stuff that I might have no control over. But in any case, I would learn why that is the case. For example, I have here a dock area on Enable. Okay, that doesn't seem to be something I have control over. So I can just keep going and at some point I will reach some spot where I can say, hey, this is something I can do something about, right? For example, I see plugins about the Pro, Builder, Rider, even plastic scm. I will be asking myself, do I actually need plastic scm? You know, at some point you will find some useful clues. And this is all about finding the whys. It doesn't mean you can do something about it, but you need to understand what the problem is. Once you know what the problem is, then you can do something about it. The point is that you have several tools that you can use together in order to reach the conclusion about why something is happening. The second pillar is CPU Profiler. So which specific profilers do you have? Well, here's the truth. Always start with a Unity CPU Profiler, that's number one. That's simple, that's effective, that's fast, and it just works. Unity knows a lot of information about itself, while other profilers do not know so much. So that's why I always start with the Unity CPU Profiler. Now if I need to complement that information with lower level information, I might turn to go to Flame Graphs. I might also use tools like pics or Intel VTune. I might recur to vendor specific tools like Snapdragon Profiler, the OVR metrics and all of these things. Okay, so there are a lot of tools, but you have to be very selective on your time. For example, if you're developing for PlayStation or for Xbox, they also have their own tool set. So just use it. Okay, so just make sure that you find the right combination of tools and that you know them, right? Because once you know the tools that you have, then you can choose when to use them. Maybe you never use it, but maybe at some point you will have to and that's going to save your butt. Other tools that I really like are Perfecto and Perfio. So let's have a look again. I would start here with the Unity Profiler. I really like the timeline view because it lets me visualize my target budgets and my current state. So for example, if my budget is 30 milliseconds here, I know that something is taking a lot of time and I can tell visually what is happening, right? What's taking so much time. But you possibly know already how to use the Unity Profiler. So. So we are going to skip that. We can also learn about perfview, but of course we cannot make this talk about that. I'm just going to show you very quickly that you also have access to other tools, right? For example, I could click on Collect and say let's start the collection. Then I could play a bit with Unity, then stop it. And then here you have access to a lot of metrics, right? For example CPU stacks. I could open the Unity process, but as you will see, this tool doesn't know as much as Unity does about itself. Right, about Unity. That's why you have to pay a price when you use these tools. However, by using tools like Purview, you get access to information that Unity hid on purpose from you. For example, every time you see something about Unity, this is also an opportunity for you for learning. For example, here we have Unity profiling interleave mono sample. That's taking 4% of the time. Okay, so here you might be wondering what's that and then that might lead you to investigate other things. You have a render play mode, view camera internal, okay? And then you need a navmesh agent custom set destination injected. The point is that there are so many tools, okay? And then you just need to find the right ones and coordinate the usage of these tools in order to get the intel that you need. So how do you find the why? Again, by doing research. Here is very clear. You can just zoom in in the profiler. Why is culling taking 5 milliseconds for example? It's a lot of time. Well, one of the reasons is because Reflections is taking half of that. Reflections are taking 2.4. Can we do something about reflections? I don't know and I don't care. Right now we are just trying to explain the why. Now next one is GPU profilers. You have the Unity GPU profiler. You know, it's kind of okay, tells you the time that you spend on each type of draw call, but it just does not give enough information. You can also use the frame debugger from Unity if you want to analyze the render order. That's very useful for overdraw. But in general I'm not super happy with the tools that Unity is offering now. You still have access to other tools, for example RenderTalk, that's the tool that I use 80% of the time when I have problems with graphics performance. You can also use intel gpa, which is graphics performance analyzer. You can also use MALI Offline compiler, which is a nice tool that I like to profile the usage of shaders. And you can also use more render specific ones, for example OVR metrics, which is nice because it's going to tell you how much time you are using on fragment shading, how much time you are using in vertex shading, how much effort you are putting into pixel fill rate, etc. For example, I could really quickly go into the scene and click this, take a snapshot the button, right? And once I am in Render doc, I will be able to tell information about this frame, right? Why it's so slow? You have this button there is to time the draw calls. Here you see the duration, right? Which is very helpful, at least in relative terms. And somewhere you will find a way to to find how much time you're using each draw call and you will find something is too expensive. For example, here we have 3,000 microseconds in something, right? Is it a lot? Yes, it is a lot if you compare to the rest of the draw calls. And now that you are here. You can find relevant information about this draw call. Might it be because overdraw? I don't know. You can check this out here. Quite overdraw. Might it be because of vertex complexity or geometric complexity? I don't know. Let's see, it says 23,000 vertices. So you know, it's not a lot, but it is also substantial. Could it just be because of the complexity of the shader? While we can go to the vertex shader, we see custom terrain, we can go to view and you know, it is a bit expensive in terms of lines at least, but maybe it's not the reason after all. I could also just go to the pipeline state and then go to the pixel shader. And here suddenly you see that we have a lot of properties, right? 10 properties or actually 11 properties. That's a lot. 11 resources and samples and all of this. This can only be expensive if you open the shader itself. It is pretty expensive. So this could be one explanation for the GPU bottleneck that you have here. Okay? For example, the terrain is contributing to large degree to the cost and the GPU cost, right? And now that you see this shader is like, you know, is it complex? Yes or not, right? You don't know. If you have experience, you might say, yes, it is expensive. If you don't have more experience, then you need to do research again, okay? For this you can use different tools. For example, I could just go to the terrain itself and then look for the material that I'm interested in. Here we have the math material and indeed we see how complex this material is. It's crazy, right? So I can just go to select shader. And here, for example, I could go and hit the compile and show code, right? Once I get this, I could copy the relevant section, for example, this pixel or fragment shader, I can just call Mali OC and then blah, blah, blah, frac. And here we will see that we are spending a lot of cycles on these categories, right? Arithmetic operations and then we also having a lot on variant sampling and also texture sampling, right? Once you know this, you can do a lot of optimizations later down the road. But first you need to know why. And for this you need to use different tools. Last pillar memory. So if you are using too much memory, you can for example, use the memory profiler from Unity. It is pretty good. You can even go a step ahead and use the memory profiler package, which displays the same information in a more visually appealing way, plus also gives information about fragmentation and memory layout. If you need that, you can also use the ADB shell Dumpsys command, right? Dumpsys meminfo A and then you put the package in it. You can also use other tools like Unity Heap Explorer. And if you want to discover more about the memory bandwidth that you're using, you can check out Snapdragon Profiler which is not very stable in my opinion. But you know that's the choice that you have. You cannot use OVR metrics and in general you just need to google or be able to google for different tools that might help you in your journey. This list might be outdated in 10 years, I don't know. But just in case you need to know how to look for this, you need to know how to combine the usage of different tools together to understand everything about your bottleneck. If I were using a lot of memory, I could just go to analysis and activate the Heap Explorer and then hit on Capture and Analyze editor. And here you have very juicy information, right? Like top 20 native memory usage. Oh, we are spending a lot of memory in texture. Remember we are doing this in the editor and profiling memory in the editor is a very bad idea. But hey, I don't have 20 minutes to do this on a device. Okay. So here if I were, you know, suspicious of the texture, I have a button that says investigate. I mean this is already amazing, isn't it? Just click it investigate. And here we have information about, you know, how much memory we are using per category. So I mean cannot get easier then this, right? I can even sort it. So let's see. Two gigabytes on textures. Lm body normal. Okay, very nice. It's taking 42m. That's a lot. And all of these textures, this is a lot. Why is this happening? I don't know. Let's have a look. If I click here and then I check the import settings, I'm sure we will not find some good format being chosen for this platform, right? I might just decide to go for something better, right? I might just go for the XT5 or PC file. I don't know. That's up to you. I have a video on this on the task force that analysis every text to compression. Then this idea you have to do your own research. You should be able to explain why something is expensive and using these tools is going to help you do exactly that. If your performance problem is on the side of taking too much memory, please, please, please use the Unity Memory profiler. It is super helpful especially if you go to the detailed mode and then you take a sample, but please do this on your target device and not on the Unity editor. Now, once you know all of this, right, and once you know where your bottleneck is and why this is happening, then you can craft the intel briefing. This is all about taking a problem that is complex and crafting simple words to explain that. Because at some point, you need to explain this to colleagues. Colleagues who might be programmers. They will possibly understand you even if you don't find the simple words. But some of these colleagues might be project managers or human resources if you messed up really bad, or even the CEO. Okay? So you need to find easy words to explain a complex problem to everybody. Because let me tell you a secret. If you want support from your peers to do an optimization task that takes two weeks to complete, you need to be able to tell them how important it is, why this is happening, and what your plan is. You don't want to confuse them. If you confuse your colleagues, they are just not going to support you. I have lived through this over and over again. So just explain with simple words what the problem is and what you can do about that. If developers were doing this well enough already, I wouldn't be having clients every single week that come to me to ask, hey, Reuben, my team cannot manage to solve this problem. I don't understand what the problem is. Please help us. I wouldn't have these clients, but I do have them, because people don't do this. So be the smart one who knows how to craft intel briefing. Okay, cool. So so far you have learned a bit about the profiling phase, which is all about gathering the intel that you need for future stages, which is basically planning phase. That's all about finding the best alternatives that you have. And then you have the performance stage, which is actually about doing the things that you said that were good, right? To reach your performance targets. So the three questions for profile were first, in which hardware component is your bottleneck? Okay, this is one of the four performance task forces for pillars. Professional performance, if it's about you or the processes that you established, the CPU pillar, if it's about AI generating protocols, networking, physics, UI generation, all of these things, right? The gpu, right? If it's mostly about the graphic sides like geometric complexity, shading, overdo, and all of this. And fourth is the memory pillar. Memory usage, memory bandwidth, and even build sizes and things related to this. Now you should also know, why is this happening? And once you know all of this, you can explain this to your colleagues and even better to yourself. Once you have this intel you can make a good optimization plan. Remember these three steps in profiling, okay? Remember that great intel leads to great optimization. So what do you think so far? Have you learned something new? Do you see yourself applying some of the tools or the workflows that you have learned today? Another important question I have for you. Will you allocate just a tiny bit of time to use at least one new profiling tool for each of the pillars that you didn't know about? Can you see how learning optimization gives you access to higher tiered jobs? Now just go ahead and start profiling like a ninja. Remember the three questions. Remember that great intel leads to great optimization and what is left to learn? Well, what is left to learn is the plan phase. Here's about formulating three options to tackle the specific bottleneck that you are dealing with. And we do this in a structured way, a very specific way that leads to very short optimization cycles. And then the third step is to actually perform the optimization stage. If you really want to learn about the whole picture of performance optimization, then I am hosting a free class which I'm calling the P3 class that you can join for free. Just register to my upcoming class at join Thegamedev Guru P3 class. When you learn what I have to say about the whole optimization world, you will get automatically access to higher tier jobs and higher tier gigs just like I do every single week.

Thank you Ruben for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P94. And while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about our awesome sponsor SmartBear and their load Ninja solution that makes performance testing effortless. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end and full stack performant application awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, dev and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over totest guild.info and let's make it happen.

SA.