Debunking 5 Stubborn Systems Performance Myths

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Mark all about debunking five stubborn system performance myths and probably all types of other performance topics, if you don't know. Mark is the CEO of Jabperf Corp. An IT consultancy providing performance engineering services for small to medium sized firms and he's been around for a bit. He also serves as a Senior Performance Engineer at WH Trading LLC in Chicago. He's also a contributing author to the book Performance Analysis and tuning on modern CPUs which is now available at CPU. We'll probably touch on that as well. And he's also a former boxer, which is interesting to learn, so probably talk a little bit about that as well. Really excited to have Mark on the show. I've been trying to get him for years. He's finally here. You don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more.

Hey Mark. Welcome to the Guild.

Hey, how's it going, Joe?

Great, great. Great to see you.

Long time coming.

Yeah, yeah, no doubt. I'm really excited to have you finally. It's been I think a few years I guess before we get into it though. Mark, is there anything I missed in your bio that you want the Guild to know more about?

Just that JAT Firth Corp. Is for small to medium firms, primarily in the high frequency trading and fintech industry.

Okay. Now why, I guess right away I'm thinking why that particular industry? Is that where you see the most need or is that just seems to be your niche?

That's where I've seen the most me, because everything in that industry is very much geared towards low latency. As a matter of fact, I've been in this industry since 09 at some of the larger firms like IMC and Jump Trading. So yeah, low latency is of utmost importance. It's paramount. As a matter of fact, sometimes security takes a huge backseat if it means better performance.

Wow. So really ignorant question. Fintech does that like crypto and things like that?

Well, crypto is becoming larger, making a larger footprint in the industry. But traditionally before crypto was a big thing, it was just basically equities derivatives, like options on equities and commodities and things of that nature. Just, you know, the kind of trading you would see on the CME or cbo. But yeah, crypto trading is gaining more of a footprint in the HRT space.

Nice. So, wow. This to me seems like a really complicated area to do performance testing. Actually I'd probably be nervous to do it because usually it's not just a web app. Right. A simple web app. It seems like you could affect a lot of like millions and millions of dollars if there's a performance issue. It sounds like.

Yeah, yeah, the reliability is very important because there is a lot of money at stake. But the typical way it works, it's not web based, but it's typically a system where you're connected with an exchange. The exchange sends you market data information in the form of UDP multicast. And based on that information in the UDP multicast, which broadcast information that's going out, on, going on, on the market. Your trading algorithm that takes in this UDP market cast, it does its magic and then it decides to execute some type of trade based on algorithms that your quant team is getting together or your trading organization is put together so that the executions go over TCP to the exchange to execute some type of specific order, or to publish some kind of order, or to cancel an order, or to modify an order, you can do any kind of execution over this TCP link that you have to the exchange. So as UDP coming in, TCP going out. And you want that round trip to be as low latency as possible so that once you see something of interest over that UDP multicast traffic, you can execute as quickly as possible.

Absolutely. So it seems like if like the price dropped, you want to be able to get at the lowest price before it moves. Why latency would be such a big deal, right?

Exactly. Or if you have an order out there that's sitting out on the exchange and then the market moves to the point that the price that you have your order out there is. It's not. There's no longer the best price. You want to remove that from the order book on the exchange, so you want to cancel it. So or modify it as quickly as possible before the market moves against you. So yeah, latency is of. Is paramount in this type of workload scheme.

So this wasn't the topic of the episode, but I'm just fascinated by it. One more question then. Is what tends to be the biggest roadblock to latency? Is it multiple things or does it usually fall into one or two areas where you're like, oh, that's usually where the latency hit's going to happen. If we make that more performant, we'll have better, better throughput and better round trip response times.

Well, I. It's hard to say what is the, the biggest. You have a lot of data you want to sift through when you're going through the market data. Maybe you're only interested in a couple instruments, but you have to sit through a lot of market data to make sure that you can find the signal that you want to act upon. So that you need to make sure that your trading platform can sift through the market data as quickly as possible to get some information that you need and then to act upon it as quickly as possible. So think about typically when you talk about wanting to execute quickly against something, you don't want the caches to get cold because you're not doing something on a frequent basis. So typically, let's say that you, you want. Let's say this, take trading out of the, the picture of the conversation and you want something to perform quickly all the time. You may just have it always operating, you have it running on a on a frequent basis and you can make sure that nothing goes cold. Everything is in a the L3 cash or the L1 cash. So everything is nice and hot on a pipeline. Well, what differs about trading is that you may be getting into a lot of market data, but you only find something interesting every now and then. So your application is not executing along that desirable path frequent enough to keep your copass hot. So you have to fight against the way computer architectures work in order to make sure that you can execute as quickly as possible when you need it to do so. So there are various techniques that you have to go through to make sure that that happens, but that's just one. That is one area I would, I don't know if I would say that's the main area that there's. There are quite a few that you need to think about, but that's part of the reason that FPGAs are such a big thing and when it comes to high frequency trading, because if you can get all of your trading algorithm to work right on the card itself so that it never has to go up to the general purpose CPU and through the OS and everything. Then you get the lowest latency possible. But the fact of the matter is there's some algorithms that are too complex and they take up so much space that they can't entirely fit on the fpga. So there's only certain aspects of the business that can work directly on the hardware. And some you have to make compromise and have to run on a general purpose cpu. So then if you run on a general purpose CPU with Linux running on it, for instance, then you have to incorporate certain techniques to make sure that you can run as quickly as possible for that round trip to be as low latency as you want it to be.

So I am completely ignorant on this type of testing for financial, so I would assume that would make it difficult. Is, are you relying on a third party to be performant as well? Is it always on you to be for the latency or it's like you're getting a UDP with things of interest. You execute on a. You send a request and then is it, is it ever performance on someone else?

Well, the great thing, the great thing about that is that there are in a lot of these protocols that we're dealing with, like the market data fee protocols that we get from exchanges and then the information, our order execution reports that we get from the exchanges, there are so many different timestamps that arrive that are part of these protocols that you can compare against so that you can make sure that if there is some kind of latency hiccup that is not on your end, it makes it feasible to find out what's going on. And that if it's at the exchange side or wherever the case is, and then you can call up the exchange and complain or and then show them the stats and the data that you have based on all the timestamps that you have on your side and the timestamps that they provided from their side so that information is available in the ecosystem to help you decide if it's really not on your end.

Awesome. Cool. Cool. All right. So I guess I just want to touch on the main topic of this episode is you wrote a really cool article recently on, I think it was called Debunking five Stubborn System Performance Myths. I thought we'd dive into that a little as well. And so I thought we'd go through all five myths and just kind of dive into the. I'll have a link to the blog post so people can read a little bit more about it. I guess the one, one, the first one is tool expertise does not equal performance expertise. I guess that's something common, you see. But can you talk a little bit more about what, what that means?

Yeah, so there's. Okay, so what I've noticed is there's. There are two different groups of performance engineers. There may be more, but the larger group of performance engineers happens to be on the website, which I think you probably, you talk to more performance engineers on that side. Am I correct?

Yes, absolutely.

Yeah, yeah. So then, as I've been getting more into this LinkedIn community of performance engineers, I've noticed that the larger portion of the web performance engineering community, there's this tendency among our colleagues to highlight or to describe themselves based on the testing tools that they use as opposed to saying that they're performance engineers. So one example of this is that I won't name any names, but there's a person who has a tutorial type of thing where it's a blog plus YouTube videos and things of that nature and it's called a performance engineering school or it's touted as a performance engineering curriculum. But every video or every blog is only about JMeter. And I've seen a lot of LinkedIn headlines where they person may describe themselves as a JMETER expert or I'm a low runner master or something to that effect. And the, the implication is that because a person is an expert in one specific testing tool, that means that they're an expert in performance engineering. And nothing could be further from the truth. So you'll see sometimes some online discussions or group chats where topics like statistical analysis will come up or queuing theory. And you'll find out that a lot of these professionals who claim expertise in these testing tools don't have a lot of the foundational expertise in performance engineering. Some of these basic ideas that you have to have in order to be an effective performance performance engineer as a whole, whereas the tooling should just be something that complements your skill set. And so that's been a prevailing myth that I've seen over the past few years that I've been involved in a performance engineering community that's largely active on LinkedIn. I don't know if you've seen the same thing or not, but it seems pretty prevalent in my opinion.

Yeah, 100%. It's usually it's someone's more of a toolsmith than an actual, you know, tester in that particular area of performance, I guess the problem is it's so hard to like, where does someone pick up the information how to do Cune theory and system architecture and CPU Mucker architecture is usually that it's such a small audience that would know that and then also know a test tool. But I definitely see that for sure. So where does someone learn these skills then to make them really more of a performance engineer rather than just, you know, someone that knows a test tool?

Well, that's one of the great things that James Pulley, I'm sure you're familiar with James Pulley.

Yep.

Yeah, so he was talking about, as a matter of fact, he formed a LinkedIn group about coming up with a curriculum that at the end of which would have some kind of certification exam where these type of foundational principles would be taught and tested for before someone can say that they're a certified performance engineer. But as it stands now, there's no one. I'm not aware of any one curriculum that teaches the foundational aspects of performance engineering. It's one of those things where. Well, personally in my career I've had to seek out these various aspects of performance engineering before I had this holistic view of how it all works. I would say that my career began pretty probably like most people's career began. They probably didn't start off as performance engineers, but they started off as something completely different. And then they picked up a bunch of different technologies and then they just kind of evolved performance engineers. So I started out as a Unix and Linux engineer and then eventually I picked up Oracle Database Administration, then Cisco Network Administration and then SAN Administration. And so having that holistic view of an infant software infrastructure, to me, the logical progression, because I didn't want to get into management was to either go into security or performance, because those are the two type of areas where you need to have that holistic view of an infrastructure. And so I chose performance. And so in that evolution, I had all the different technologies under my belt as far as understanding how those work and how they interwork. But once I really got into the performance tuning aspect of IT and the trying to specialize in it, that's when I realized, okay, I need to know what this whole queuing theory thing is about so I can model performance and help me understand these large infrastructures and complex topologies. If I can model this with queuing theory, that would help me along the process. And then also when I was doing performance testing, I was getting this raw data, this, all this raw data. How do I process it and how do I make sense of it to make sure that when I'm comparing a before and after picture, I know if I've actually made a latency improvement or made a latency regression. And how do I do that in a sound manner where I'm not fooling myself? So that's when statistics came into the picture. It's like, okay, I never took statistics. I know what an average and a standard deviation is, but do I really know when that applies and when it doesn't apply? When I need something that's different than just comparing averages and standard deviation. What are statistically sound methods of comparing two distributions based on the type of distributions that they are? These are all kind of things that we need as performance engineers before we can do accurate testing and then reporting the results of those to our higher level executives like CTOs, who want to know the information.

It's a great point. Some of the times these tools, the reporting makes it seem like, oh, here's the average, but they totally fly. No, an outlier. And the outlier may actually be the thing that you need to really focusing on rather than just the average. So I definitely agree there.

Yeah, exactly. And then also, not only that, but a lot of these reporting tools, or a lot of these tools that give you these, the fancy reporting with the beautiful graphs at the end. Unless you have a good grasp of how computer systems work and what is like a back of the envelope type of calculations about how fast something should go based on what type of technology you're using. How do you know how to fact check the tool? Because the tool could just spit out some number. But if you don't have some kind of backup envelope type of calculation to think of, okay, this is what the ballpark should be.

Right? Right, Right.

How do you sanity check your result?

Yeah, yeah, yeah, that makes complete sense. And also I think this kind of goes into your next myth. A lot of these tools, they'll just focus in on, like the top four things that you normally monitor, like CPU and memory. And your myth four is memory consumption. Tracking is not enough. So what is this myth all about?

Yeah, it's, you know, a lot of these tools, they'll track how much RAM you're using, what percentage of ram, or if you're swapping, which means that you're using too much ram and it kind of limits it to that, and people think that's enough. If I'm not swapping and if I'm not taking up memory from some other application that needs it, then that's fine. RAM is taken care of. But the fact of the matter is it's not performance is not just about how much RAM you're using, it's also about how, how much throughput you're imposing on the RAM that's installing your system. Because on a multi core system you guys are accessing that memory through a memory controller and everyone on that same CPU socket, they're sharing these memory controllers. So if one is hogging the throughput on that memory controller, even though it's only using let's say 10% of the available RAM, so there's enough RAM to go around for everybody. But this one particular application that's using just a small fraction of RAM is nonetheless using the lion's share of the throughput going through that memory controller, then he's really crowding out the other applications on that same socket that needs to access RAM through that same memory controller. And so you may see some performance degradation, but then you look at the typical stat and you're like, well there's no swapping going on, you know, and I, there's no heavy context. Switching is like I'm getting CPUs when I need them, so why am I having this performance anomaly? So then you have this blind spot because you're not tracking how much of the throughput on your memory controller every application is consuming. So a lot of the modern tools, there are some things where I don't know if they're all, but I know the tick stack, there's Telegraph plugin that you can enable on Telegraph where you can stream that information into InfluxDB and then you can track what your memory controller throughput use is over a given period of time. So you can see if some kind of anomalous behavior happens and track it back to if it's actually because of memory controller or something else.

So yeah, it would be hard to track down. So what happens is there's a queue up and if not seeing things being queued up, you just see the CPU is normal and your threshold, your transactions per second are dropping.

It could be transactions per second if you're a throughput based, if your goal is to be throughput based and yeah, possibly could be that. And my line of work is mostly about latency if my, if we call it tick to trade latency. So tick being when a certain UDP market data packet comes in and then, so that's the tick and then trade is when your TCP trade leaves your trading platform to go out to the exchange. So that tick the trade round trip time. Once that gets to a certain Level, if that gets over a certain number of microseconds, then I'm like, okay, something's going on. What caused this anomalous? Take the trade number here. Why are we having this latency spike? And then I can look at the behavior of all these different metrics on that system for the past few hours to see if some kind of anomalous type of thing happened. Say, let's say it actually is because some kind of rogue application is running on the same socket and it's just like hammering the memory controller. Now, let me just qualify it this way. Memory controllers these days, they can take a lot of bandwidth, they offer a lot of RAM bandwidth. So I'm not talking about some application that's writing at 100 gigabytes a second and then it's, you know, icing out other applications. It doesn't even need to get to that level. It gets into details about how a lot of the memory controllers work, but it does cause the memory controller to have to make tougher decisions when it's ordering different requests at the memory controller level. So it doesn't even need to get to its max bandwidth. But just having one, that's hammering it enough to the point that now you no longer get as low latency access to RAM as you used to, or as you typically do, it can affect latency of your application.

Cool. So I guess the next we can talk more and more about this, but I guess I want to get through the five of them. The other one is I think sampling profilers. This might be a little controversial because I think everyone loves sampling profilers, but you say myth number three is sampling profilers work great for multi threaded apps.

Right? Right. Because the thing is with multi threaded apps, with a single threaded app, and if you throw a sampling profiler against it, you know where the CPU is spending most of its time based on the output of that profile. It just, it arranges it in descending order from the most CPU consuming or expensive function down to the least, and then you just start at the top. Because everything is serial in a single threaded application. But when you talk about multi threaded application, it's not so simple about which threads are in the critical path and at which point you may have all the threads doing something different, or they all may be cooperating on a certain workload and then they convene at some spot to combine their information and spit out one result. There's all kinds of different ways that multi threaded applications can cooperate to get a result, but you never know. You don't Know that. So when you walk you will know that if you wrote it. But it may not be obvious in that type of workload where the critical path is. So the profiler will tell you where you spend a lot of cpu, but it could be on some kind of thread. The highest CPU consuming function can be on a thread that really has nothing to do with the critical path. It could be some background thread that does some kind of housekeeping, but it's nonetheless taking the most cpu. Well, if you tune, if you take all your effort to tune this background funk this background thread, the function on this background thread, you'll find that you've wasted a lot of time, but yet you've done nothing to increase the performance of your multi threaded application. Whereas with causal profilers like Cause Coz, it actually will execute various experiments during runtime across the whole code base and then at the end of its experimentation it'll spit out results saying that hey, if you speed up this function at this line number of this source code file by this amount, we estimate that you'll improve the performance of the entire application by this amount. And I encourage everyone to read the research paper that was the genesis of the whole Cause project because it's very interesting that the technique that they use to get this or to accomplish this, it's very interesting reading. It's called Causal Profiling by Emory Berger and Charlie Kurtzinger. But it's a very interesting paper. But it's not just some kind of pie in the sky type of research. It actually has a tool attached to it called Cause that you can run. I think the Java version of is called jc. There's a Rust version called COS rs and then there's one for CC which is just called cos. But I encourage everyone to give it a try because you'd be surprised what you find. As a matter of fact, I mentioned in the article that I used it for one client engagement where the cause pointed out a function that was a part of a thread that had that the developers felt had nothing to do with the overall performance of the application. They felt it was outside of the critical path because was saying something different. So when I actually tuned the function that Cars pointed out, then we got the actual estimated performance improvement that COs spit out at the end. So everyone was surprised to find out that this particular function on this thread was actually part of the critical path.

Nice. And I'll actually have a link to this in the show notes and to your article. And I think in your article you actually have a Link to Cos as well. Cool. Myth number two is CPU clock speed is paramount. So what's the deal? Like CPU clock speed?

Yeah, that one's lasted so long that that myth used to be called the megahertz myth. So it's old enough to the time where CPUs were operating in megahertz, not gigahertz. So yeah, you just have. There's always a couple of guys in a technical department where they believe that if CPU A from vendor A has a faster clock than CPU B from vendor B, then we need to get CPU A because it has the faster clock. Or if you get two CPUs from the same vendor, but the older version had a faster clock than the newer version, that may be two or three generations newer than this older clock. It doesn't matter. They just believe that whichever one has the faster clock is the one we need to stick to. So I actually have a funny story about a scenario where that happened at a firm I was working at where one team, they were basically refusing to upgrade to a new generation Intel CPU because the older generation CPU that they were currently running with, that they were running with at the time, had a much faster clock than this newer generation intel cpu. So when I finally got them to try it out, they were surprised at how much it improved the performance, despite the fact that the new generation CPU had a far slower clock than their older cpu. So not everyone believes that myth, but I put it number two, because it's still floating around and it's been, it has impressive longevity.

Absolutely. And I guess myth number one is something I think people might be surprised by, is Big O complexity does not equal performance. This is kind of contrary to what people have always been taught in computer science.

Right, Right. Yeah. People will be, you know, they'll be surprised to find out that bubble sort, you know, for a certain size number of elements is faster than quicksort. But bubble sort is like the first sort that everybody learns. It's like a, a horrible algorithm because it's quadratic, you know, in big O notation. But because of the way caches work, it's faster than a lot of other sort algorithms that have a better big old complexity than bubble sort, all because of caching effect. You know, so we're not executing algorithms on some kind of vanilla landscape. We're running it on actual hardware that has branch prediction, it has multi level caches, all these different kind of things, hardware prefectures that can get in the way of what you estimate the complexity of your algorithm to be. So big O can't be the big old complexity analysis can't be the only way that we examine algorithms. We can't just go by counting the number of steps that an algorithm takes and then deciding that okay, this one takes fewer steps, therefore this means this is going to be the faster algorithm. It's much more complex than that. So I think college courses to teach students to also profile their application for these type of machine effects like caching and branch prediction that should be a part of their analysis and not just big old complexity analysis.

Absolutely Great advice. Okay Mark, before we go, is there one piece of actionable advice you can give to someone to help them with their performance engineering efforts and what's the best way to find or contact you?

The best way to contact me is on Twitter. My Twitter name is Medawson JR me Dawson J R. Or you can reach me on LinkedIn. The best piece of advice I can give someone for performance engineering if they want to enter into this discipline, I would recommend that they get Neil Gunther's Performance book. Any of Neil Gunther's performance books will give you a nice foundation about how performance works, how modeling works. I think that will give you a nice general overview of the entire landscape and then you can build specific knowledge from there.

Thanks again for your Performance Testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com.

And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.