Performance Testing in Troubled Times with PerfGuild

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And this episode is just going to be a solo episode, just me and you, talking about why performance testing in troubled times is more important now than ever and how you don't want to be like Robinhood. I'm also going to share some audio for some of the video sessions that we're going to be having at this year's Perf Guild Online Conference that I think could help you avoid being like a company like Robinhood. For those that don't know, the Perf Guild Online Conference, taking place June 1st to the 2nd, is an online conference dedicated 100% to help you succeed with your performance engineering and testing efforts. Our goal is you learn a tip, tool, technique or best practice you could put into place right away to help a new or existing performance testing project. And make sure to listen all the way to the end to receive a special exclusive discount just for Test Guild Performance Podcast listeners. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary, no changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

As we enter May of this year. I don't think anyone predicted the troubled times we'd find ourselves in in 2020. And you know, more than a few companies were caught off guard by how COVID 19 actually impacted the traffic to their website. And the bad news is that some sites completely crashed due to the spike in traffic. So just imagine if Netflix went down for 24 hours. Netflix usage has actually gone up amid COVID 19, but how would you feel if you cannot access it for 20 due to a performance issue? You'd go crazy, right? It would totally impact your perception of Netflix and whether you would continue as a customer in some cases depending on how annoyed you were. So here's a real worst case scenario. This actually happened to the stock trading app Robinhood recently. It went down. It went down twice. So the first time I think it went down for like 24 hours during what was probably 1 of the stock market's most significant swings. And it cost users millions of dollars because users weren't able to log in and sell or buy anything for 24 hours. And do you know what the reason for this significant outage was? I'll give you a guess. That's right. Due to a surge in traffic, their system struggled with unprecedented load and it crashed. And actually the founders of Robinhood went online and explained why, why it was down. And this is what the founders said. They said, we now understand the cause of the outage was stress on their infrastructure, which struggled with unprecedented load. This in turn led to a thundering herd effect, triggering a failure of their DNS systems. So multiple factors contributed to the unprecedented load that ultimately led to their outage. And you know, of course it had to do with high volatile and historic market conditions, record volume and record account signups. You would think that companies would think about this ahead of time. I can understand how some companies don't. A lot of these are kind of black swan type events where they're outside the norm. But this is a stock trading app. They should have built in the idea that there's going to be swings in the market and then know how to at least react to that in their systems gracefully. I can understand how some companies don't. A lot of these are kind of black swan type events where they're outside the norm. But I think as performance engineers, we actually overlook a lot of these and sometimes we just go on maybe happy path type performance testing issues rather than really stressing our systems or what could be a real use case scenario, even though it would be outside the norm, really talking to our team members and say, okay, what happened if this? And do we have a user case for it? Or what is the risk if this happened, no matter how remote it is, if it did happen, how would it impact us? And how can we plan for these type of events? And as I mentioned, this didn't happen just one time for Robinhood. It happened, I think a week or two after. Same type of deal, it was down for about four hours and same type of issue, the systems are just overloaded. So here's the question, and it's a valid question because even if you're just an employee, think about how this could impact your job through layoffs due to loss in market share or to the inability to recover from such an event. How many users and how much long term revenue do you think Robinhood lost because of that bad performance? You know, they probably were thinking, oh, we're going to Go, we're going to go public and we're going to sell our stock. Right? But their brand took such a hit right now that I think they lost a considerable amount of money because of their incompetence. Or maybe that's kind of strong incompetence because like I said, it is a kind of like a swan event. But companies should be planning these type of events because this is what happens. This is why you need testing. This is why you need to think outside the box and really invest in not only functional tests, but performance testing, security testing for these type of scenarios. And remember that, you know, the top engineering companies consider performance not as a nice to have, but as a crucial critical feature of their applications. And just back to Netflix. As I mentioned at the beginning of the show, their usage is actually up as you can imagine. More people are home, more people are streaming, more people have time to binge watch shows on Netflix. And notice you didn't hear anything about Netflix in the news of their services being downed. Why do you think that is? So Netflix, actually because of all the different infrastructure over time they had to support an increasingly complex and really a lot of resource hungry type activities with a huge, huge user base. They actually invested in a tool or they came up with their own in house tool which they created called Chaos Monkey which was used to proactively. So they proactively were able to cause these type of failures at random places, at random intervals within their system and see how their system behaved and how they could recover from it. And actually this is becoming a really hot type of type of testing the past year or so as chaos Engineering. And this is one of the reasons why I think more companies should be investing in their performance engineering testing efforts. So another site that was affected by COVID 19 without them thinking how could a virus affect a online business? Well, I don't know if you heard of sba. They're a US government agency that represents the US Small Business Administration and they provide support for entrepreneurs and small businesses. So this was the administration that was actually responsible for the APAC protection program, which was A I think 700 billion business loan program established for Coronavirus Relief and Aid and Economic Security act to help certain small businesses or self employed workers, sole proprietors, to continue paying their workers. And the weird thing about this loan, it's not really a loan, it's, it's forgivable. Meaning you get free money or supposedly free money in air quotes. Government's just printing money. But anyway, that's a different story. But just hearing that free money. So if you're just hearing about this for the first time, you're probably the first thing you thought is, wow, that's probably going to be a very busy program. That site is probably going to get slammed when that program goes live. Eventually they ran out of money. That's how many people applied to this program. And they got hit so hard that once again their system went down for four hours because was just hammered with users. And once again, the whole point of this program was to instill confidence in the business community that, hey, don't worry, we have this, and hopefully instill confidence that eventually things will get better. And the first experience is the site goes down, people are already panicked, people already stressed. And I think it had the opposite result. And once again, I wonder, has anyone in that IT department even thought about performance? Everyone as a company need to realize the risk that their application might have and address that before these type of scenarios happen. Using capacity planning, performance planning, performance engineering, having teams really bake this into their systems ahead of time so they can avoid this or at least have a plan in place for when it goes live in the wild and some sort of anomaly occurs. So the reason why I'm telling you all this is that I've seen over my career, I started in 1999, 2000 with performance testing and then I eventually got into functional testing as well, functional automation. Most engineering teams I've spoken to also on my podcast, we're over now, 300 episodes on my automation podcast. So I've interviewed a lot of people and their teams don't regularly test the performance and scalability of the applications and infrastructure. And actually most of them actually lack the knowledge to even do so correctly. And the good news is I think that these examples I've given really highlight the importance of and the need for more performance testing. I also have kind of a controversial view of testing that if you have tester in your title, you should be responsible for any type of testing, not just functional testing. I think a tester just sometimes puts the head down and just thinks that they're only responsible for functional testing. I think testers, as a tester myself, we should be expanding our reach into all aspects of testing within our teams. We're the ones with the testing expertise and we should be able to shepherd our teams through any type of testing activity they may have. Whether it's unit testing. We may not be able or be the ones actually coding the unit test, but we should have an understanding of what unit testing is and how we can help Our teams create more effective tests so that when we do our testing, our applications have testability baked in from the beginning. But not only testability from a functional perspective. How about if we got involved earlier with, say, security testing? Once again, you may not be a security expert, but if you tell your teams early enough in the process, hey, not only do we need to make our applications testable, we need to make them secure, we need to make them perform it. Like, once again, if you've already created an application, never thought about testing, or you didn't make your application testable from the beginning, you can't bake testability into an application. You can't just say, okay, I'm going to automate it, when no one thought about quality or testability to begin with. You need to bake that in from the beginning. It's the same exact thing with performance testing. If your application wasn't designed to be performant to begin with, if you're testing it, you can't test performance into an application. You can only highlight how bad it is at performance and your development life cycle. And that's why performance testing really is a growing area of performance engineering that I think we all need to evolve and embrace with our testing plans. So after you heard these examples, you probably could think of all the ones that actually affect your industry. How's your company going to approach their performance engineering? And as we move forward, COVID 19 hopefully we'll get out of it soon. But it's not going to be the last type of event we see. History shows us anything. These things happen. Every five to 10 years, something happens that we didn't plan for that's going to impact the performance of our applications. So how are you going to prepare for this? This is one reason why I created my online performance testing conference, Perf Guild, way back in 2017. And that's why I'm really excited to announce the opening of the third annual Perf Guild. And look, I know what you're thinking. Everyone's going online now. Everyone has an online event. But as I mentioned, we started in 2017. We've only been focusing on delivering the best of the conference experience, filled with actionable advice, all online. This is not a pivot, it's not a fad. I'm not jumping on the bandwagon. This is not a second option. We at Test Guild conferences, we've only done and will only do online conferences. That's all we believe in. You know, on site's okay, but, you know, if you really believe in global warming, Climate change. If you're traveling around the world to speak or attend an on site conference, it's insanity to me. And then on top of this COVID 19 it just, it just does not make sense. So we've always believed in online conferences. It's not not our second option and it's not like a nice we're going to add it on now. We only have and we only will do online conferences. It's the only thing we focus in on giving you the best value we can. All online. Even companies that have embraced online events, it's still almost like an afterthought to them or like a. It's like a webinar like I believe try to deliver the best online experience possible that mimics an on site conference without all the headaches and all the expenses of having to travel. So you can trust us not wasting your time with a quickly pulled together, poorly run online event. So I just wanted to give you a sneak peek at what you're going to learn at the 3rd annual online perf Guild. We have some of the top performance testing experts in the industry that's going to share with you a tip, tool, technique or best practice that will help accelerate your performance engineering efforts in 2020 to handle COVID 19 type of events and even just everyday performance.

That's how backwards I think some of companies performance efforts are. They're not even up to subpar performance. They don't even think about performance. So we want to get everyone up. It's a great conference for beginners. We have a lot of sessions at this one that is geared towards beginners because I really want more folks to embrace performance testing. So here's some high level topics that we'll be covering as well as I'm going to give you some audio samples of some speakers of what they'll be covering at this event as well. So we're going to have hybrid load testing using real browsers. So a lot of times what teams struggle with is creating well new type of performance. Engineers is using protocol level performance testing. You need protocol level performance testing in most instances. You can't do anything without protocol level performance scripts, but you can leverage real browsers to do performance testing as well. And so using a hybrid approach of using browser based automation which most testers I know are already familiar with. With protocol level performance testing you get the best of both worlds. So you can get up to speed quicker, faster using a real browser type of performance to get going with performance testing. So that's one of the topics we'll be covering. We'll be covering effective performance engineering. It's an intro level type of session with Scott Moore and here's just a quick sample of what Scott's going to be sharing with you during that session.

Hey everybody, this is Scott Moore and it's really a pleasure to be here on Perf Guild again this year. I learned from my experience last year that I need to keep the scope of what I talk about down a little bit because we do have a limited amount of time. There's so much I always want to share. Now this year I really want to focus on some basic things. One is going to be around how to start moving from just a performance tester to thinking like an engineer and getting earlier in the lifecycle and just a starter point of how easy that is and what I would do. Then I want to share with you a couple of templates that I use when I do a standard performance testing project. Maybe just some suggestions to get you started. And then finally I want to talk about the digital experience. Now that gets thrown around a lot these days, but not a lot of people really define it. I want to put some definition around it and help you understand it. So some of this stuff for you advanced users or people who have been doing this for a while, this may be first grader stuff for you, but you might learn some things as well. So I hope you'll enjoy this as well as people who are just starting out and just beginning. And if you're a performance tester wanting to think like an engineer and gain some additional skill sets, I think that you'll get something out of this as well. So this is just for you Perf Guild audience. Let's go ahead and dive right in.

Awesome. Thank you Scott. This is going to be a great session. And as Scott mentioned, these videos or these sessions are created specifically for the Perf Guild. We're not reusing sessions that we did. I know a company, a few companies have done online events. What they've done is they just rehashed videos of what they had for their on site events, put it online and charged for it. We don't do retreads here at Perf Guild, so we try to give you some fresh content that's unique to the conference and that it's relevant for what people are struggling with now with their performance efforts. We also have a session on leveraging your automation scripts for performance testing. We have sessions on a code first approach to performance engineering. We have one on Java Performance tips. And just let me give you a Sneak peek of that session as well, because that's really valuable if you're dealing with any type of Java performance testing.

Hi there, my name is Stephen Townsend and I am a performance specialist based in New Zealand. When I attend conferences and events, I find the most interesting ones are when people share a real experience that they had from the job. I find it the most engaging because I want to know what happened with that person. How did it go? I often find it's the most honest. You hear different sides of the story and I often find I learn the most from it as well. Often something that the presenter thought wasn't that important becomes a really interesting talking point, something to discuss later on. So today I'm going to give an experience report on an engagement that I did in the last few months.

Thank you Stephen for your session. Once again, like I said, it's a real world example and as Stephen mentioned, I find the same thing. A lot of times you may think, oh, how does that relate? I find real world examples, some of the most beneficial ones when I do attend conferences as well. Another session we're going to have is how to add front end performance to your software development life cycle. With more and more modern applications using these really front end heavy JavaScript frameworks like Angular and React, this is becoming more important now than ever. Also, we're going to have a session on shift, right? Digital Experience Analytics. So here's a little sneak peek for that.

Hey, I'm Jonathan Wright and today I'm going to be talking to you about Digital Experience Analytics. Now I've only got 25 minutes so it's going to be fast and furious, but you're going to learn loads. So let's start quick. Shout out to Scott Moore who'll be talking about effective performance engineering. My friend Todd DiCaprio who hosts the website effectiveperformanceengineering.com and has written the book which you can download from O'Reilly which will give you a great source on how some of these patterns can be applied. Now we're looking at OpenStack here, so everything we're going to talk about will run on a Raspberry PI or some kind of low compute device. Okay, so you may have heard of performance application lifecycle, known as PAL. This has been around since the early 90s. So you can see here. Back in 1998 I was performance testing against what we would class as a client server written in Java.

Awesome. Thank you Jonathan. Like I said, this is another hands on real world example. Jonathan really digs in here Jonathan is an automation cyborg. You won't be disappointed with this session. We also have a session on shift left, so we have shift right covered. We have shift left, we have how to create performance engineering, how to code. So we have really end to end life cycle performance framework pretty much that's covered. And this particular perf guild, another one I'm really excited about is we have two sessions I think are special specific to the type of what COVID 19 to me represents. It's like a black swan event and I think it's caused by, you know, variants. James Pulley is going to have a session on the beauty of variants, how variance is so essential to creating realistic performance testing scenarios. I think a lot of us miss that when we're doing performance testing. We do really like a sterile type of performance test and not think about maybe outliers that we should be focusing on. It's just, you know, happy path, as I mentioned earlier. And we also have a session by Paul McLean on distortion, why it makes music sound great and how it's relevant to make your load test more realistic. Once again, it's another session on practical tips on making load test and your SRE test more realistic by injecting appropriate spikes into load test. And so as I mentioned, those two examples earlier were Robin Hood and and the sba. Those were due to spikes in their normal performance profile that they weren't able to handle because they never modeled it. They never checked to see what would happen if that occurred in the real world. And these two sessions I think are going to give you a little background on how to mimic kind of like an unknown.

Hi, I'm Paul and today we're going to discuss distortion, why it sounds great, and why you should consider including some sort of distortion in your load and performance tests. I'm going to step through this analogy of sound distortion and fidelity and compare it with making a test more realistic. So the focus is on test realism, making it so that your test translates the requirements for testing into a realistic repeatable test that's more likely to mimic what's actually happening in production, which is what we want to do. Hope you enjoy it.

And if you're listening to this the first few days after I release it, there's actually a discounted rate on the site already. So if you applied the code Guild Coin50, that's G U I L D Coin C O I N50, the number 50. If you apply that coupon code, you'll get $50 off the already discounted price. Rates are going to go up, but this I'm going to keep this discount in effect as a thank you for being a loyal listener. So all you need to do is go to guildconferences.com and click on the PERF Guild 2020 link and apply that discount to your checkout card. For links to everything of value we covered in this episode, head on over to testguild.com p.

And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about SmartBear's Marden Performance Testing tool Solution Load Ninja, which actually uses real browsers to help you do performance testing. It is one of the topics we will be covering in this Perf Guild as well. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.