Performance Tour 2023 BBQ with Scott Moore

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device for some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE Podcast and my goal is to help you succeed in creating application performance awesomeness.

Hey it's Joe and welcome to another episode of the Test Skill, Performance and Site Reliability Podcast. And today you're in for a special treat because we have the great Scott Moore joining us today to talk all about performance testing, his performance tour and some secrets that he's going to reveal. So you don't want to miss it. You want to stay all the way to the end if you don't know Scott is a PERF Evangelist, host of the Performance Tour at perf tour us. With over 28 years of IT experience with various platforms and technologies, he's worked with some of the largest applications and infrastructures in the world. Scott also founded Load Tester Inc. In 2004 and co founded Northway Solutions Group in 2010. But in 2015 he brought his Awesomeness together and created Scott Moore Consulting llc, which he set up to provide consulting services around performance testing, performance engineering and performance monitoring. He's an active thought leader in the performance engineering space, so he's created a lot of awesome things. And if you don't know, Scott is also the creator of such performance musical parody hits like two 503s ain't bad, technically Dumb Black Friday. My site is down and much more that you need to check out his YouTube channel. These are high quality musical video parodies geared just for performance testers and testers that you definitely have to check out. But we'll talk about that probably as well. You don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head on over to Loneinja.com and learn more.

Hey Scott, welcome back to the Guild.

Hey Joe, it's really good to be on your show. Thank you for having me. That intro was a mouthful. I got to get a shorter bio. It's got to be Guy with loud mouth and make stupid videos. Here's Scott.

He never give me a bio, so I always use this one I've had for a while, so I even know if it's correct. So is there anything I missed in your bio that you want the Guild to know more about?

No, no, no. You're pretty much right. It's that I don't think anybody cares about that stuff anymore. They just. They want to see the stupid videos now. So.

They're not stupid. They're awesome. And people need to see them. They should be going viral.

So thank you. I've got to improve my marketing on that. No, but thanks again for having me on. I was told by Paul Grisafi that if I was on your show, I would be one of the cool kids again.

So that's good. Awesome. Always good to hear about Paul. So I guess some things you've been up to, some of these videos are part of your perf tour. So I thought maybe we start off by, you know, I want to touch on 2023, but before we do, maybe let's talk a little bit about, you know, what is the perf chart first for people that are unaware of it.

Yeah. So in 20, the late 2019, when everything was normal in our world last, I decided I wanted to go to a conference. It was going to be the Dynatrace Perform conference out in Vegas. And my wife said, you're going to have figure out a way to pay for it yourself, buddy, because I'm not giving you any money. And of course, you know who the boss is. So I had to figure out what to do. And I thought, you know what? I'll just take my iPhone, I'll travel from where I live in Florida to Las Vegas by car. And I know people in almost every state from Florida to Vegas. And I'll just stop and talk to people, record the conversations, and let's see if I can get a sponsor to pay for my trip. I sent an email out the next day, and within 10 minutes, I had a sponsor, which was Catch Point. Mehdi from the CEO of Catch Point, said, I'll do it. And I thought, wow, I should have asked for way more money. That was the first thought. No, But I said, you know, if I'm going to do it, why just stop at me holding an iPhone, talking to some friends? Let's. Let's go buy some cameras. So now I got, you know, I got to have more money because I got to buy equipment and all these things. That was the start of my content Creation phase. So I drove. It took me. I was on the road for about 33 days total. So I just didn't go straight to Vegas. I stopped in all these places, and I went over 5,500 miles in my truck and learned from vendors, from experts in the space at this conference. I mean, I talked to everybody, and I was asking them the big question is, where are we at with regards to performance these days? And what's. What are the problems that we need to solve and what do I need to learn? And I learned a lot from that trip. When I got back, it was right at the beginning of April 2020, which, as you know, what happened then was as soon as I got home, everything got locked down. And here I am with all this footage, and I'm like, what am I gonna do now? Fortunately, I had enough footage to release 17 full episodes of my journey. Part of that was with James Pulley. Part of that was on my own and with other people. And so the performance tour show actually just kind of came out of that. And one of the big things that I learned were we're nowhere near where I thought we were supposed to be in this day and time with regards to performance engineering, the skill sets and all that. And because of that, I'm still producing shows about it because apparently there's still a lot that needs to be done.

So, you know you did in 2000. You know you did this year, 2002. Were there any key takeaways? Like, how do you know? Because you're actually speaking with some of the best of the best in performance. And usually when you get that, you get kind of a skewed result, like, wow, everyone's doing performance. But what are you hearing that makes you think, well, there's more to be done.

Yeah. When you get out there in the field and you talk to the companies that are trying to implement some of these things that the experts, and I'll point it myself, of things that they're supposed to be doing. It's easier said than done. So, oh, we're just going to do continuous performance. Oh, yeah. Just check by. Just. You just click a couple of buttons and you're doing continuous performance. It's difficult, and everything's becoming more complex. And when things become more complex, you don't need less testing and less stuff. You need more of it, which increases the complexity on that end. I think it's kind of unfair for someone to ever think, oh, we're going to make our development side more complex, our systems more complex. But testing should Be able to just click a couple of buttons. And so those. When you get down to where people are actually doing the work, that's when you find out there's a lot of stuff that's still missing. There's still a lot of blind spots.

Absolutely. What I thought was kind of interesting, this past year, I've heard more and more testers talking about performance that they need to do performance testing. So to me, it's like that's a good thing. But it almost sounds like it's almost impossible because like you said, things are getting more complicated. And maybe now they're expecting testers to have no background performance to even continuous performance, which is really, I mean, kind of something that's kind of a little more advanced, I would think. What are your thoughts on that?

We're expecting a lot out of people today, I feel. And there is a tendency to get away from specialization. And I think with regards to performance, that's the pill that I'm willing to die on. That's my soapbox, is to be very good at performance in all areas you're going to need to specialize. I can't be a security penetration tester, a great functional tester, exploratory tester, and a performance tester and be equally as good. There are things that a performance engineer needs to think about. And it's not just the testing piece, it's the engineering testing being one of those disciplines that an engineer does. A lot of things that we have to think about that aren't just about your code, they aren't just about the database, they aren't just about the infrastructure, and it's everything in between. And one weak link can cause a problem. So when you're talking about performance playing a more important role and the whole shift left effort, it's a great to shift left and get the feedback about performance as early as possible. But we don't stop there. We still need that because there are still things that slip through the cracks. And if you just have, you know, great discipline on the left side, you shift everything so far left that the whole ship capsizes to the left. You still have to have all these other disciplines in place. For example, somebody still has to pay attention to the network performance. It's, you know, that might not ever be an issue for some companies, but it might be the Achilles heel for others. And somebody who only thinks about their code. Well, my code is fine. Yeah, but how does your code affect that the network? Is it real chatty? Are you pulling too much stuff across the network? That you shouldn't be too many times. And those are things that you don't. A developer isn't going to necessarily think about at the time they're writing the code. It's found out later. So that's where all these things come into play. And we're expecting a lot these days when you hear about site reliability engineers, they're expected to be the find it and fix it people who find the problem, finding code around the solution, reduce all the toil and help operations, make everything a smooth running engine. And it's like, come on, I mean at some point you're going to start seeing, well, we are seeing in some of the surveys burnout. You know, people are just getting overloaded and burned out and you're going to have some bad fallout from that, I think.

So how should it work then? You know, I know with automation, a lot of times people have like an automation architect that sits outside of sprint teams and he kind of coaches each of the sprint teams and they have like an embedded tester that kind of helps lead everyone into the testing effort. Do you still see performance as an activity that should be done outside of the sprint as a separate team or do you. I guess it depends. Or do you see it more as like a coaching feature that could then help all the different sprint teams as they're going through their features?

Well, I've always been a fan of teaching people how to fish. Right. But I don't expect a development team who's working on this feature and it's a five person team and they work in a squad, for example, say they're trying to do what sp. I don't expect each individual member of that team to know as much about performance of their code as I would. Or maybe there's one individual in that team that should be designated. As you check performance, we'll write the code and as we do code checks between us, we'll check, you check for that. I would prefer to be sort of like that coach or mentor that tells them, here's your scope of performance. It's relegated to this function or feature you're working on. But I don't expect you to deal with the infrastructure issues that Operations is having to deal with. Focus on your area, get the feedback as early as you can to prevent it ever going downstream from you. And then downstream we have another area that will handle that. So that way, yes, it is everybody's responsibility for performance, but the scope is completely different. It's not necessarily everybody's responsibility to know everything that could go wrong performance wise, right?

Absolutely. So Scott, you know we're heading into the holiday season so this probably released after Black Friday, but you know you'll be released at the holiday season, right, where performance is usually all of a sudden there's a, there's a spike in interest and what's going on. You know, everyone listens to this. Anything they can do to prepare for holiday or unexpected spikes in their. The traffic that you've seen over the years caused some of these outages.

You know, it's the same thing every year. I know in 2022 we are going to have news of outages and our buddy James Pulley is going to report on his show on Black Friday that he does every year and he's going to tell everybody name and shame. I'm sure it's not even Black Friday yet. We've already seen this as we're recording this. We are fresh off of a new story where the new Taylor Swift concert tour crashed Ticketmaster. And that's not a new thing and it's not necessarily a simple thing that has to be done. Ticketmasters had problems for years going down because of high demand in a small period of time and it takes somebody like Taylor Swift to just, you know, expose that problem again. So when you got these point load situations, again, they're not new, you deal with them when they come. But if that's like Ticketmaster, every time that's down, they're losing money by the minute. I mean lots of money by the minute. So I would expect their team to be more focused on that than say some birthday announcement website or something that's, that's not relegated to, it's all about, you know, the money that's coming in. So yeah, they should be preparing for that. They should be preparing for, scaling for that. Any retail company, you know, if they're, if they're not concerned about compression and making sure that everything that can be as close to the user is as close as it can be with CDN and caching as much as they can cache and all the things that we know are the best practices that they're not working on that actively and haven't been for months. They're pretty much behind at this point.

I definitely agree. So, you know, I guess then if we're heading to 2023, people maybe don't have everything in place for this year. Maybe they get ready for next year, I guess before we get into that, are there any common themes or trends you've been seeing going into 2023 that maybe people need to know about before, you know, new technologies or techniques that you think can really help them with the performance efforts.

I think you're seeing a lot more activity on the front end performance side. There's a lot of, I would say developer slash developer advocates that are out there. You see them a lot on Twitter, talking about Google Analytics, Lighthouse metrics, the page load times, they're focused on that. I think that's an area that needs to be focused on right now. So anything that you can do to make that page render faster, whether it's, you know, compression or the lowest cost, you can make that front end webpage and making it as small as possible, working within a budget, performance budget for your webpage. Those are the things that you should be focusing on. Those are low hanging fruit and those are things you can see right in the browser. Anybody can open up a Chrome browser, click on Developer Tools and see immediately, you know, how many things are pulling down, how large is it, how much expensive is this webpage. Whereas the back office operations, they should be testing for, you know, the database, the network calls, all of those typical things. They would be. And I think if you focus on both of those, you know, that's a winning combination. But the front end piece right now I think has been so neglected. I'm happy to see some of these people reporting that. And Tim Verike is one out of Akamai who's always talking about it. You'll see the folks from WebpageTest talking about it and you know, Catch Point owns WebpageTest. You'll see a lot of outages and things that happen from Catch Point. They've got a website called webc, I believe. So you can use these tools right now to find out how you're doing and as you're going through it. Gtmetrics M E T R I X gtmetrics.com Open up your front page on that site and look at it. It'll tell you everything that it sees that's wrong with it. Right. And gives you a score. Not that you have to work for 100% score, but just understand where you are in things. Are your images way too big? Are you making way too many calls? These are things that are real simple best practices, but they're not necessarily talked about. When you're learning to develop the website, learning React and JavaScript frameworks and things, you're not thinking about that you're building something awesome and you want to think about that when you're coding.

Absolutely. Front end. I've been seeing that as well. Another thing I've been seeing is more like infrastructure, like Kubernetes Optimization. I think you gave me the name of Stefano.

Is it Akimus Akamas? Yeah, Akamas and also stormforge. Both of those companies do something very similar, have a different approach to both of it. But one thing, just being back from Kubecon 2022 in Detroit, a huge theme is around efficiency. So we talk about performance efficiency, now we're hearing about energy efficiency. Hey, your code might be fast, but if you're, you know, consuming the energy of the sun every day in your data center, that's not a good thing either. We have countries with power issues, so there's an efficiency and then there's cost efficiency as well. So yeah, your stuff may be really, really fast, but it cost us three times as much as our competitor does to do the same thing. Why is that? So when you are thinking, I guess as a performance engineer, it's not just how fast something is but how efficiently can I get that speed and am I breaking the bank? In other words, to get good performance, that's not a good situation to be in.

So a lot of these companies, when I speak to them as well, they're almost changing their language to be more site reliability engineering centric. Don't know if it's my imagination. Is that something you're seeing as well? Maybe the morphing of titles or roles or responsibilities?

Whatever Google says is the cool thing to be. Most people, we're not going to fight that and they want to be that. I don't know if that's because they think they're going to get a better salary out of it or whatever, but when you say site reliability engineer, some people know exactly what that means. There's a great percentage of people who have no idea what that means. What does a reliability engineer mean? We're no longer hearing about quality and performance. We're hearing about resilience and reliability. What do those terms mean to people who have been in the industry for like you and I, 15, 20, 25, 30 years versus somebody who's in the market, you know, five years and are completely cloud native from day one, completely containerized, you know, applications that are, you know, kubernetes first, it's the mind shift is changing. And this is something I'll mention too, also from Kubecon, realizing that as a classic performance engineer, the world is changing around me. It is changing around me. When you talk about changing terms, you hear about these title changes. You also hear about technology terms changing Everything used to be APM application performance monitoring. Now it's observability. There's a reason for that, right? It's not just we're changing the name to change the name. It's. We have different things to worry about now. When we were just monitoring something, it was always there. Well, now it might be there 15 minutes and it's something else. It's, it's morphing. So we have to make changes to what we look at and how we look at it. And so performance engineers really need to be thinking about this. If they're not keeping pace with how this, how fast this is changing. I think everybody would agree that was at Kubecon. The adoption of kubernetes is faster than anybody ever thought it was going to be. The maturity that's bringing is faster than anybody thought. And so the vendors are having to step up to provide solutions faster than they did for previous, if you want to call movements of technologies. That's what I saw.

So I must be out of the loop. This was actually the first year I've heard of Kubecon, but it seems like everyone I talked to was, went there and they were going to it and looked huge. So. And I guess that's one of the reasons why, right? What's that?

That's what all the cool kids are doing.

I have it on my calendar for next year. Is that the reason why you think? Because a lot of people are going cloud native and like this kind of shake up in the industry where you need to stay ahead and that's why maybe it's something that's coming bigger.

Well, it's the way we're building. Software is changing.

Right.

And we're going to build applications that are as flexible and hopefully as resilient as possible. So when you think about Kubernetes being able to create an environment and replicate an environment across the cloud, across pods, across clusters, up, you know, being expandable at will, that there's some complexity to it, obviously, but it's the way we're building software is changing. We started out with bare metal. We went to virtual machines. The cloud vendors said, we'll take that burden away from you. Then everybody said, okay, we'll just go serverless. We'll continue to take as much as we can away from you that you don't have to worry about managing. Just write your code and put it out here. Well, then we did that. And the next phase of that is, oh, and you know what? We've been relying on one cloud provider to provide this for us. And that's not good enough either, because every time AWS east coast goes down, it takes 40 major companies down with it. We can't afford to have that risk. So to make it more reliable, we're going to have to deploy on all the clouds everywhere, all the time, multi, hybrid, meta, all that stuff. More complexity. So the way that we build and deploy software is continuing to change.

Awesome. All right, Scott. Okay. Definitely want to touch on 2023. Before we do, let's wrap up 2022. Biggest bloopers that people didn't see in 2022.

Biggest bloopers THAT people didn't see. It really hasn't changed from any of the years before that. Still seeing the same sort of cloud dependencies. This massive shift to the cloud and the push over the last two years to the cloud has caused so many people to wake up and realize things weren't as good as they thought they were. They weren't as resilient as they thought. VPNs weren't as resilient as we thought. All of these things had to be rethought of. And I think that's the big thing. And also cryptocurrency and blockchain is right now just kind of turned into a sort of a nothing burger. And there will obviously be uses for it in the future, and it will come. But the big push for. For blockchain to be the solution for everything, I don't see it. We hear a lot about Web3, but Web3 is sort of built on top of Web2. And there's just a little bit of something that we haven't dealt with before, but a lot of it has turned into. It's still kind of smoke and mirror. So I don't see that right now as being the next great move in it.

Absolutely. And best barbecue place you went to this year?

Oh, by far. It was in Austin, Texas, and it was called Jebby's, and it was actually a food truck. I think it was G E B B Y S Best brisket. I know that Texas brisket is legendary, but when you actually have the bomb brisket, you never forget it. It's a whole different food group.

Awesome. All right, so, Scott, 2023, what are your plans? Are you still going to do the perf tour? Any new info? How could people, you know, tell us?

So let's talk about the performance tour. Okay, so for this year, this year of 2022 was the first year that I went from doing stuff on the side, like doing it as A fun project creating content or blogs or whatever to being a content creator. Like, I haven't done any performance testing or engineering engagements this whole year. I've been blocked off because the performance tour, once a month, the podcast that I do, the SMC Journal, comes out twice a week. I'm actually having to reduce that because everybody's like, scott, you're coming out with so much stuff, we can't keep up with it. Slow down. So we're going to go to once a week next year if the sponsors renew. I'm working on that right now, but between those two shows, that keeps me completely busy. But we've raised the quality of the show as well. We're trying to make the performance tour almost like something you would see on, you know, broadcast television, you know, major stations or cable tv. And we're getting there. The purpose of that creating all this content is to try to bring a wider audience to performance because, let's admit it, it's kind of a dry topic, unless you're geeks like us and we love this stuff. But to try to talk to a group of people who maybe are in it, or maybe they run a company, but they still don't understand the value. Performance engineers haven't done a great job of showing, being able to tell the story of what. Why they're so valuable to companies. So I'm trying to do that with, yeah, we have sort of a technical interview. We have somebody on that's an expert. We talk about a topic, but then we try to provide that entertainment there. So at least they. Somebody who would typically just turn the thing off would go, hold on a second. I don't really understand everything here. I'm sort of getting it. But this guy's doing some crazy stuff. I want to see what he does next. And I think that's resonating with people. And then performance engineers that aren't the best at telling that story would say, hey, just watch this guy, what he's doing. Yeah, he's singing a song here, but look at the lyrics. This is actually true. This is my life. You need to know this. It's like, wow, I didn't realize that we. It was affecting us that much. So this year has been like a total transformation for me. I'm Mr. Content working with video editing software and sound and all that, cameras and things. Lighting. I have to worry about lighting. The good news, and this is, I'm going to announce it right here on your show. You'll be the first one. This is the major announcement is 2023, the Performance Tour show has been fully funded for filming. Next year. We will have two software sponsors for this and they are. They do something very similar but different. They kind of cross over. That's as much as I can say right now, but I'll be talking about that by mid December, before the end of the year, I'll reveal who the sponsors are and it's going to be great. We're not just going to do just, you know, music videos. We're actually going to expand into some television shows. And that you're familiar with movie references and things still in the form of parody keeps us out of legal trouble that way. But still, we're going to be talking about really cool topics about back to the end user experience. But we might also talk about quantum computing. We might talk about, you know, the future of where we're going. So I'm excited. I hope that with the podcast, if I get the renewals on the sponsors there, you'll be seeing a lot more of me talking about performance stuff for 2023. And I'll be getting out, having these meetups, hopefully hanging out with you at conferences and talking about the things I'm learning.

Absolutely. It's much needed and I highly recommend people do check it out. I actually caught it in Nashville with Dew and Scott and James Pulley and it was off the hook. So I highly recommend. And we had a barbecue for free. So if you haven't been to a Perf Tour event, you definitely need to see Scott in 2023. So Scott, when are you going to roll out your schedule then so people can stop booking their dates?

Probably late December or 1st of January and I'm actually formulating those plans right now. And if someone is out there watching that is interested in hosting a meetup somewhere. Right now I'm sort of limited to the United States, but I'm not opposed to going somewhere else. I would love to take Perf Tour global. I'd love to go over to Europe, for example. There's things I haven't. I have never even seen places I've never been that I want to go to. And I have friends over there as well. They love performance and they love barbecue and good food.

So I'd love to see you in a Bollywood parody that would be so awesome in India if you were to.

Don't put it past me.

Okay, Scott, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find contact you or learn more about the Perf Tour to.

Find out about me theperformancetour.com smcjournal.com or Scottmore Consulting any of those sites will take you to me. Also very active on LinkedIn and Twitter and social media. I'm pretty easy to find best content out there. You know, not to toot my horn, but my buddies at Perfbytes, you know, Mark Tomlinson, James Pulley, Henrik, those guys are great. Anything that you can go back to and look at and listen to that perfbytes library, I actually do that myself and I learn a lot from that library. There are others obviously I'm putting together. There actually is a webpage out there. I've got a spreadsheet. These are a list of all the companies, the people that I follow, for example, on Twitter and social media where I get my information from and I share that with others. And I'm happy to give you the link for that, Joe, so that you can share it. But those are some places that I would start, unfortunately. I wish that I could say there's course, there's a college, there's these books, there's a few books I guess I could recommend or a chapter. But this is one of those things where you got to kind of do your own research and you got to find people like me and people that are connected to me and your show, Joe, to find out this information because there's just not enough out there.

Thanks again for your performance Testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P10.5 and.

While you're there, make sure to click.

On the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes?

Reviews really do matter in the rankings of the show and I read each.

And every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe.

My mission is to help you succeed.

With creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening.

If you're not already part of our.

Awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community. I'd love to hear from you. Head ON over to testguild.info and let's make it happen.