Six Top Trends for Performance in 2020 and Beyond with Joe Colantonio

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Once again, it's just going to be me and you in a solo episode, but if you're sick of hearing my voice and you'd like to hear more interviews and and you think you have a tip, tool, technique or best practice, you think you can help someone with their performance testing or their site reliability engineering efforts, let me know. Drop me an email@joestgill.com and I'd love to get you on the show to talk all about it, but today we're going to talk about the six top trends for performance testing in 2020 and beyond that you need to know. You don't want to miss this episode. Check it out. And if you haven't already, I would highly recommend that you register for this year's Perf Guild Online conference. It's the third annual online conference Dedic 100% to helping you succeed with all your performance testing and performance engineering efforts. You don't want to miss this year's event. It's taking place June 1st to the 2nd, so you only have like a week or so left to register. And for being a loyal Test Guild performance Testing podcast listener, if you use the coupon code PODGuildcoin25, you'll get $50 off the ticket price. That's P O D G U I L D c o I n 25 discount discount code at checkout. To get $50 off your ticket price, just head on over to GuildConferences.com and click on perf guild 2020.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code or almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

All right, so here are my top performance testing trends you need to be aware of in 2020 beyond now, many of these trends have taken place earlier with other software development activities. If you've done any sort of functional automation, maybe back in the day and you notice that a lot of companies have changed the way they've implemented automated testing. It's the same thing that's now going on with performance testing activities. Except performance testing activities seem to have been lagging and now they're just catching up to say what we did with our functional automation maybe five, six years ago and the first one is a shift from performance testing to performance engineering. So one of the biggest trends I've seen in the last few years is that I've been seeing the move from doing performance testing to more of teams embracing a full software software development lifecycle. Performance engineering practice. You may be asking, what is performance engineering? According to folks that I've spoken to on this podcast and on my other podcast, performance engineering is a cultural shift in the way organizations view their essential processes. Performance engineering really embraces practices and capabilities that build quality and performance throughout the organization and throughout the software development lifecycle. Todd Dicapia, who is the author of the book Effective Performance Engineering, says that performance engineering means understanding how all parts of the system fit together, knowing which metrics matter, and building in performance from the first design. Also, Scott Moore, who is a performance testing consultant and speaker at the 2020 PERF Guild, he expressed the same thing when he said that the days of testing performance into a product are over. It's just too risky and it causes too much technical debt to go back to do so. This really is the same type of shift that occurred with functional automation years ago. Rather than having one person responsible for automation, everyone in the team really becomes responsible, becomes a whole team approach. So that's something I definitely see going forward with performance testing as well. So you might be asking how can you help your company to transform over to a performance engineering culture? And you might think the first thing is technical or it's a tool. But I have a real actionable, non technical advice you can use right now and that is to MIP it. So one suggestion you could do anytime, anywhere with performance or any other quality concern is to MIP it, which means mention it in passing. And I got this from Scott Barber. And the way Scott explained it to me is that he would just MIP to anyone he passed in his office. He'd say, hey, how's the performance today? So he'd be walking down the corridor, someone go by, he says, hey, how's the performance today? He didn't even know who they were or what to pop they were in. He would just say, hey, how's performance today? And that one person could be anyone, not just someone involved in the project. And so he said he would just MIP it to anyone in the building. How's performance today? And at the end of the week he said performance increased significantly only by saying, you know, how's performance today? By asking the question to anyone that would listen and not even doing it or coming out and saying we need to take care of performance just by saying how's performance today? So I think this is really effective. And another effective approach is to definitely check out Scott Moore's effective performance engineering session at the GIS Perf Guild. It's really a great introduction to folks new to performance engineering that want to know some actual ways to get started. Of course, Scott Barber's approach is just a start. The real push really is to get your team thinking about performance as early as possible, not after the fact, after it's already baked into your application. And that's where the next trend comes in. More and more teams shifting their performance testing efforts, more left in the software development lifecycle process. And once again, just like we are trying to get developers more involved in functional automation activities by really embracing the same languages and tools that they use in their day to day workflow, moving performance testing earlier in your software development lifecycle. Because of this I definitely see a trend towards creating tests using developer friendly tools and languages. This allows performance tasks to fit in really nicely with the developer's current ecosystem. So there's a lot less context switching. So if the heads down coding, it's really hard to get them to use say a tool like JMETER or something, it probably would be better in some instances to use a tool that actually uses the same language they may already be developing in. So say you have a bunch of Python developers, you can introduce a tool, a performance tool like Locus which allows them to create Python based performance testing scripts. So if you use Locus, everything is in Python, everything is in code. You can even use PyCharm as an IDE. So it's probably the same exact tools techniques that your developers are already using. It just makes it less resistance by just using the same language and tools they're already using. That's one way you can do it. Another popular option I've been hearing more about is K6 and this also focuses on giving developers really the best experience for load testing. K6 takes a code like approach to developer test scripts. Same thing with tools like Gatling which uses Scala Scala type approach to performance testing. So these tools really allow you to treat your performance testing scenarios or test cases just like you would any other application code, which in turn will enable you to shift your performance testing efforts earlier in the software development lifecycle. Another quick hack I have for you is from Christian Daniel Marquez Barros. I interviewed him in an earlier episode of the Test Scale Performance podcast. He's a front end developer, he has a bunch of experience and he said he gave me an excellent recommendation to get developers more involved in creating more performance software. He said that all front end developers should be using at a minimum Google Lighthouse metrics that's available in their Google Chrome tools that they're probably using already. All you need to do is when you open up the dev tools in Chrome, just navigate to the Audit tab and you could find the Lighthouse audit. This will tell you the performance of your site. It simulates mobile on the desktop versions of your browsers. It does a lot of cool things. They also I think just recently released a 6.0 version of Lighthouse with new functionality and it's the perfect tool to really to start with to get your developers kind of hooked on as a developing code. Also start thinking about hey, how can we not only make this functional but how can we make it more performant? Just like anything, you really can't really bake that in after the fact, really. A pillar of performance engineering is that performance needs to be baked in into your software from the very beginning with developers creating more performant code in your product from the start, not you trying to test it in and around the same thing. We have another session at Perf Guild. It's Allison Henry and Matthew Andres. They're both from blazemeter and they have a session on why good test scripts aren't enough, Shift left and automation as keys to the future of testing, which is really, I think, a great example of how to achieve this. You could definitely check out that session, head on over to perfguild.com, register for that today. So another way to get your whole team involved in performance testing is to leverage your existing functional tests using real browsers for performance. And this takes us to our next trend, browser based performance testing testing. So another trend I've seen is more companies leveraging their automation functional tests for performance engineering activities. Now this is not perfect. It's not a silver bullet. I don't recommend it for every single performance activity, but it's a great way to use this for certain activities that get your whole team involved in performance testing. And you know, you might be wondering why shouldn't I just use a non UI based protocol? This is the preferred way if you're doing a real enterprise level end to end performance test. But nowadays A lot of modern JavaScript front end frameworks like Angular and React, they make it very difficult to create performance scripts at a protocol level, which is kind of the traditional way Testers have been developing performance tests since the mid-90s. In such cases there's really no interaction between the UI and the server like you'd find in a typical old school client server application architecture. And with with protocol tools you would have to do some additional scripting and programming to capture those kinds of transactions. Just makes it so the process of setting up a script comes very tricky. By focusing more on browser based performance script creation, you end up spending less time doing any of the manual editing, correlation and programming that would otherwise be necessary with a protocol based recording approach. Also, because this approach uses a real browser and more people are familiar with interacting with an actual browser rather than a protocol, this really does help to shift left the performance activities that more teams probably should get involved in. Does it replace performance level performance testing? No. And even Avengers would say it doesn't replace it all. For example, Nicole Van der Hoven from Flood IO is having a session at Perf Guild on how you can combine both browser and protocol level performance scripting to create a really cool hybrid load testing solution. So it's really the best of both worlds. You don't want to miss that. Head over to perfguild.com register for that today. But it really is just another tool, another technique you can add to your performance toolbox to help transform your teams towards becoming a performance engineering powerhouse. So another trend is a shift right using AI. The first time I heard this term was from Jonathan Wright on AIOps. AIOps is about expanding AI out of the just functional testing type area that I've been hearing more about it there in the functional area and applying it to all your software development shift right activities. And although many companies have started to make the shift right, they still tend to focus on simple analytics and you know digital experience analytics is incredibly important. But. But I really think we as industry need to stop focusing on a webpage or a mobile app and we really need to start thinking about the entire customer journey. And this is something that Jonathan Wright actually recommends doing. He has a real world scenario using all open source tools for the shift right digital experience analytics type AI type approach in his perfguild 2020 talk. So what is the shift right AI types approach? So just think of it more like monitoring things in production and when it's monitoring things this algorithm then is able to take that monitoring information and automatically create a Performance test model for what is actually happening in production. So a lot of times you may have assumptions that you build into your performance scenario thinking how your users use your application. And once you release it in the wild, you're like, it's behaving totally different. Why is that? Well, with this approach, because you're running something in production, it's actually listening to the actual transactions your real users are using in a typical day. It's then able to take that information, roll it up into a model, and then generate some performance scripts for you that mimic what it really looks like in production for you. So that to me is like a holy grail of creating a really realistic performance scenario. And it really helps to introduce, help you make some really informed decisions by taking that information, getting that feedback and feeding it back into your life cycle from the beginning, all the way the end, kind of iterating on that feedback and making things better along the way. It can also help inform your decisions using machine learning to bring up key insights that you might not have noticed in your pipeline. In fact, James Pulley thinks this is an excellent play for AI in performance testing. If you look at what we do inside performance testing and performance engineering in general, it's really pattern based. And the conventional AI used today is less expert system based and more AI pattern based. So it doesn't ask questions and assign weights. What it does, it behaves more like looking at a pattern and observing matching conditions and saying, you know, is this a normal condition or is this an exception? And if it's an exception, does it match other exception patterns? So when you talk about things, say like resource utilization and things like that, AI really could be useful because AI is great for looking at behavior of systems, going through the logs and saying, here's what 90% of our people are doing, here's how they're traversing our application, and here are some of the common paths in building a model for what it should look like. And it's able then to bubble up things say this doesn't look right based on previous performance by analyzing all the historic data for you and then bubbling up key insights saying, hey, this behavior looks different, this possibly could be an issue. And there's also a subset of this that AI really can play in actually generating the actual performance scripts for you. So I really think monitoring things in production is great way to know or to plan for unknowns that tend to happen once your system goes live that you didn't anticipate. So speaking of Unknowns, if this COVID 19 pandemic has taught us anything. It really is that we need to make our software more resilient. And that is my next trend to Chaos Engineering. Resilience testing, variance testing, however you call it, anything that helps you to plan for these more unknown type of events in production. So we've seen scenarios in production that most teams probably have not even anticipated. How do you plan for unknown behavior? And I've seen examples of this in the past few months. If you looked at the financial app Robinhood, if you looked at the Small Business association from the government when they did the payroll protection program that they rolled out a few months ago, even Epic Games I think released a free game and people were because everyone's home due to the pandemic playing games, I don't think they anticipated how many users were actually going to get that free game. And so all these systems went down due to unexpected spikes due to out of the normal type circumstances. So you know the best approach to achieving chaos is to test how the system responds under stressed proactively. So rather than to wait to once again into production and say okay, now I know I need to handle this, you could be a little more proactive for these type of events. You can then identify and fix failures before they impact your customers or cause damage to your reputation due to poor publicity an outage can receive. The idea of Chaos Engineering is that it compares what you believe will happen in your distributed system to the reality of what is happening. And so to learn how to build resilient software systems, you can use Chaos test tools to break things in your environment on purpose to see if it actually fails the way you believe it would. Chaos Engineering is achieved by using thoughtfully planned out experiments that helps to reveal a weakness in your system. Dawn Haynes, the CEO and testing expert at Perf Test plus, also is a big advocate of adding more robustness testing to our test plans. She says the industry definition for robustness testing is how does your software respond in the face of invalid input or unexpected or stressful environmental conditions? Which is exactly what a lot of these companies have seen that cause our application to go down. But Dylan really likes to define it as applications delivering value to customers consistently and not disappointing them or frustrating them in any particular way. Once again, think about Robinhood. That financial app people went in, logged in during probably was one of the worst case scenarios for the stock market and probably ever with people selling and buying and their stocks crashing. They weren't able to log in and sell or to buy. So it really cost the users. It literally cost the users millions of dollars and just think about how that impact future customers really using their service in the future. It probably was not positive. So a lot of people kind of have a hard time visualizing this because what would a requirement look like for robustness? Well, maybe we just look at how we handle invalid inputs, or maybe you look at how we handle stressful environment conditions. But if you expand it to it being all about the user experience, how robust are we for user interactions? Do we fall down when we get surprised by something, or do we handle surprises in a way that we could do? Mainly two things. Inform the user about what's going on and make some sort of suggestion on what they could do next. Rather than just crashing, really crash more elegantly, or say, hey, when I'm able to do this right now, do something else. Really trying to inform the user along the way rather than just throwing up a random error message and locking everyone out. Or if you're surprised by something, if your user's surprised by something, can you do something other than fail? It would be great if you could create some sort of mechanisms into your applications that could check to say, all right, this failed, but how can we fail gracefully so that when someone is surprised, it just doesn't trash the whole system? So I really do expect variance chaos and resilience testing to be a significant area of focus for teams in 2020 and beyond due to the COVID 19 scare. And that's why really, I am excited that we have two sessions around these type of testing approaches. We have one about distortion by Paul McLean and one on the Beauty of variance by James Pulley that really show you how you can use variance and distortion and things outside the norm to help you create more realistic load tests to help you plan for these unknowns. And the final trend is performance testing in CICD. So where does performance fit in the DevOps pipeline? So one expert I spoke with, Paula Rosaro, mentioned that there is a trend for performance tests to become part of our acceptance test. So automation is vital and having a reliable reproduction of the production environment is a must. So rapid feedback also allows resolving issues before they reach the user. This is what is known as continuous testing. Continuous testing really is a key component of continuous integration, continuous delivery. And what's good about having a streamlined continuous delivery process is if the only way of finding out if your application performance is not working well is via a ticket being opened by a complaint of a user, you have issues. I would think, if that was the case, that you know you're going to have a lot of unhappy customers when in turn they're not going to use your application which is going to cause loss in revenue and maybe you lose your job. There's a lot of things that occur because of performance that you may not even think of and open tickets an unhappy customer is not ideal so testing and continuous integration really go hand in hand. Each step during the continuous delivery pipeline also has testing associated with it. So from the development environment, integration, testing and test environments, functional tests and then pre production environments with performance and accepted testing up to production when you have synthetic monitoring and real user monitoring and gathering monitoring to make sure the performance is working well. This really is the type of ideal situation. All these steps we really need to have as part of our continuous delivery and continuous testing and performance testing really works well together with this type of framework. And I hope to see more teams start to integrate performance tests into the CI CD pipelines in 2020 and beyond. So those are my top six predictions for 2020. So what do you think? Let me know. Did I miss something? Am I wrong on something? Also, to really properly prepare for these trends, I really do highly recommend you register for PERF Guild 2020. You only have a few weeks left to really get a jump on these trends. Just head on over to perfguild.com and register today. And for links to everything of value we talked about in today's episode, head on over to testgill.com P34. And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance test tool Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.