Performance Testing in 2020 with James Pulley

Five, four, three, two, one.

Drop dead.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with the one and only James Pulley, all about performance trends in 2020 and beyond. James has focused for the past decade on assisting clients and people around the world to help them find answers to the most complex questions related to load performance, response time and scalability. James began his career working for Microsoft and he's also worked for one of my favorite test tool vendors ever, back in the day, Mercury Interactive. So he has a bunch of experience around a bunch of things, helping executives, consultants. He's been an advisor and evangelist from numerous companies, big and small. He's also the practice manager and performance performance engineer and test at Tech Systems. And he's also the host of my favorite podcast ever, PerfBytes. So listen up, you don't want to miss this episode.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code or almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, James. Welcome to the guild.

Good morning, Vietnam. Oh, I'm sorry, Wrong. Wrong show.

Good morning, Joe.

Awesome. It's an honor to have you on the show, James. Before we get into it, I probably messed up your bio. I'll re edit it. But is there anything in the bio that I missed that you want to share with folks?

No, actually, I was impressed by the bio. It's more impressive than my actual life. I think.

I've been very lucky. I've had some extraordinary opportunities. I have found a job which is a very high match to my natural inclinations on pattern matching and things of that nature. And as a result.

I have this joy every day of coming to work, solving complex questions. And yeah, I've also been fortunate to be able to give back to my own industry and have some sort of small influence in helping people become more productive. So.

It'S a good life.

So, James, I'll be honest, I don't speak with many people that actually really say the Company they work for is a high match. So I guess I'm already off script, but any tips to help someone in the new year to maybe find the perfect match for them in their employment?

You really have to know how you like to work. Very early on, I understood that I did not want a job where I did the same thing every single day and I was not constantly learning and expanding how I viewed the universe. Originally, my intention was to go to law school, if you can imagine that. Can you imagine me spaz up in front of some judge saying, sir, I think your resources are over? No, that just would not work there.

And I just happened. My undergraduate degree was in computer science, not in history. Like many people going to law school, I wound up in an internship in computer science with the municipal government and the work model that I desired out of a legal career that is mostly project based, long and short term, constant learning, constant innovation. I actually found that in it and I thought at the time, hey, this kind of matches how I like to work and how I like to learn and I don't have to deal with so many criminals. Little did I know that criminals exist across all industries in kind of a uniform distribution. So.

That'S where I was very fortunate very early on to understand what I wanted out of a career. Not necessarily the exact profession, but I knew how I wanted to work and live and learn. And then having identified that, moving towards a model or a profession which allowed for that has really.

Been a good thing.

Awesome. All right, so, James, let's get into performance testing. You've been around performance testing for a while. You speak with a lot of folks on perfbytes. Just trying to get a pulse on. Since this is going to be the first episode for Test Guild performance podcast in 2020, go over some trends maybe you may see in 2020 and beyond. I thought, you know, I spoke to you a while ago, we didn't record it and you gave me like a good framework to go on with, you know, front end, back end, CICD and apm. So I thought maybe we'd touch each of those and get your thoughts on maybe trends for 2020 around them. So the first one is front end. One thing I've been hearing a lot about is people reutilizing functional tests as performance tests. And I'm not sure if that's a trend you see happening in 2020 or something you see happening yourself.

Well, managers love that trend.

Yeah.

Oh, we can answer two questions at once.

Not really. You kind of have to know, does it work for One before you can early ask the question, does it work for many. Now there are some things you can do with functional testing tools and manual testing for that matter is you can answer the question of does it scale for one, does it meet your performance requirements for a single user? But you have to ask the question. Just about every functional testing tool out there has a timer structure built in. So take advantage of that and actually ask the question. How fast is the page? How fast is the response? Track it and report it. Odds are you will find a lot earlier in the test cycle that your code is not necessarily performant. Someone has a desirement, it's not really written down. This is kind of the Scott barbarism in this case that oh, it should be five seconds or less, or if it's a service call, it should be 100 milliseconds or less. And in functional testing you discover or reveal simply because you're reporting the data that hey, it's 15 seconds for one user. Yeah, we should performance test that with a lot more users. Yeah, that's really going to work. At that point, finding it earlier is key to being able to fix it earlier on a more cost effective basis. So those are things that you can do. Now what is driving the use of functional testing tools outside of this model is these thick client side frameworks, react and the like. It makes it very difficult to reproduce at a protocol level, which is kind of the traditional way since about 1996 that we do performance testing. People forget that prior to 1996 we used functional testing tools to do performance testing. We used X Runner, we used winrunner, we used Rational Robot, we used Cilk Test. And the reason why we used that is we didn't have an effective way to get to the protocol level for one. And client server applications in many cases were thick, they were cross platform. We could use a Unix workstation and spin up hundreds of instances. It was very resource inefficient. It's still resource inefficient. What's really happened is the cost of resources has dropped through the floor with cloud. It's made it much more cost effective. It still doesn't answer the question of how much is inside the front end, how much is inside the back end.

There are a lot of great innovations coming from the W3C, the nav timing metrics inside the clients to help reveal how much time is actually spent inside the browser versus inside the back end. All of that is very useful.

And we find that web clients are every bit as thick and resource inefficient. As the old two tier client server applications of old going to the database, it's not unusual to find a 8 or 16 megabyte react front end running inside of a browser. And then the browser itself takes up a lot of resources. You could even argue that this combination is more resource inefficient than writing a classical two tier client server app.

Do you see any value? Because I do see folks because of that, that very problem is that they tell me, well, our real clients use a browser and because our application is browser front end heavy, we need to use a solution that scales up browsers for our performance type tests.

No, what you're actually saying is your server does not know or care whether it's running a real browser on the front end. All it knows is it's responding to a request. If, if you make the request at the protocol level and give it the proper headers and everything else, it looks to the server exactly like it does coming from a browser. If you don't know how to do that, and there is a lot of people who don't know how to do that, then the natural inclination is to go up the stack. Now there is the trade off there. You do not get a pure measurement of the server response time. You can get that measure using other methods than sampling at the client front end. That is, you can use other W3C metrics at the server time taken field to actually collect those response times on individual requests. You can see those things you have to know to go to look for them, but you lose the granular ability to see what's going on purely at the server level from a scalability. And if you've ever looked in the performance tab of developer tools under Chrome, you'll see that little narrow blue slice which represents server and the whole rest of the thing which represents JavaScript, idle client rendering, things of that nature that are really driving the response time to the client. You need to ask the question what are you trying to measure? Are you trying to measure client or are you trying to measure server? If you're trying to measure server, you want the most efficient method for generating server load and measuring server response. If you want to measure the client during that, we have this concept called a GUI virtual user. It's been around for, I don't know, since early 1990s where you could run five full browsers, one for each business process. And you could, you could get login_ GUI and compare that to login_ HTTP and get a better measurement of what's going on inside the client. But there are an awful lot of people out there who do not understand the mechanics of HTTP, nor are they being trained in understanding the mechanics of that. They're being thrown into the engineering dog pit and say, produce a performance test. Okay, if I know Selenium and I'm being told to produce a performance test, and I have this set of vendors over here saying I can reuse my Selenium scripts to do it, well, that's, that's about the easiest and fastest path I have.

So, James, do you think that's a part of another trend though, where folks want to get performance testing in their CI CD pipelines and therefore they're making their agile teams, regardless of the skills, handle everything, including performance tests that may require someone with a little more expertise in that area.

I am seeing that trend. You know, every time we see a shift in development methodology, people claim, oh, you know, it's QA friendly. And then you see QA groups being disbanded and the QA responsibility given to developers. Now we in the QA profession are responsible for some of that happening. There are quite a few vendors out there that their only value proposition to the client is we are cheaper than everyone else. That is their first discussion point. Not value. What are we going to deliver? How are we going to make you better? It's, here's our price proposition. Well, if that's the only way a client has to measure you, you know, developers are pretty poor testers. I can say that as a reform developer. You know, we test how we build and design. That's, that's kind of the natural form of how humans work. Testers, on the other hand, are supposed to take that secondary view, measured against a different set of requirements, or hopefully it should be the same set of requirements, but oftentimes it's, it's a larger, more comprehensive set of requirements and testing a wider path through the code and in unexpected ways that the developer did not anticipate to generate value and understanding of how that code performs when it gets to production? Joe, I don't know about you, I just don't see that in a lot of the, here's my price proposition vendors. And in the end, in that case, if you're getting such low value out of your vendor, if you cancel it and give it back to developers who have a known value proposition and the value of your outsourced testers are not any greater than that of your developers. Well, yeah, we've created this situation. So as we have shifted from waterfall to Agile, From Agile to DevOps, I do see this trend continuing of, well, you know, automation. Yeah, we'll kill the testing team, we'll give it to the developers. They're more sophisticated and better equipped to handle programming for automation. Anyway. I absolutely see that argument and I see it happening.

So, James, I guess another thing that kind of throws teams off as well is backend. And by backend I mean I've been hearing more and more about hybrid cloud implementations and once again, I don't know if that adds more complexity. Do you see that as a trend where folks don't even know how to handle maybe backend performance monitoring or when they're running a performance test because it may be running in the cloud and they just think they could scale it up and not worry about it?

Absolutely. In fact, some of the core tools that are targeted in the open source universe, they discount the whole issue of monitoring and analysis and reporting altogether.

They view performance testing, whether it's a load test, a stress test or any other type of test, as a matter of production of load. And there is no monitoring capability. You look at JMeter and say, okay, I want to monitor my database. Good luck with that. You know, that's, that's another tool that you need to integrate with. Oh, and you want to do integrated reporting. Oh, that's another tool. You know, we're, we're just responsible for production of load and reporting response times and that's it. Once again, this, this is kind of an industry failure that has driven this, is that you look at the distribution and you can probably see this in automated functional testers as well. We certainly see this in performance testers. There's about one in, somewhere between one in 10 and one in 20 performance testers who have actually been through training, who have had product training, who have had a mentor and of that group that's available, you know, that's the highly effective group. At any given point, there may be 1% of those who are available on the market to actually help other companies who become a mentor or something of that nature. The rest of it is I give you a tool, I'm going to tell you to go to YouTube and watch a bunch of videos and you're going to be our performance person tomorrow. That is a guaranteed path to low value. It also doesn't recognize the fact that performance testing and performance engineering, which it's part of, requires a super set of skills from that of a automated functional tester. Yeah, there's some programming skills that are shared, but there's an architectural Knowledge of systems which is required for this issue of how do I monitor and how do I collect information on time? What's often misunderstood for performance testing is that time is a symptom. It's not the actual root value that you're looking for. Time is driven by how a system uses resources. So if you're only collecting time, you're collecting evidence or a symptom of something that's going on. But if you're not collecting cpu, disk, memory, network across all your operating systems and on your major system components, the equivalent of these, like on a database, this might be something related to cache hits, table scans, lock weights. You see how that translates to, say, one layer up from the core operating system. If you're not collecting that data, you have no idea what drives that symptom, that response time, and where to make recommendations on tuning for the system. And this is something that you very, very deliberately see in the Perfbytes podcast is we emphasize monitoring very, very heavily where to look, how to determine. And on the news of the DAM podcast, we look at patterns. A lot of these patterns also involve resource utilization. We look at public tools like GT Metrics and Google Page speed and things of that nature to see how resources are being used, because that is what drives that response time. And if you're not collecting measurements of time and collecting measurements of resource, you could make the argument, are you really testing?

Absolutely. And James, I guess it's another trend I've been seeing, maybe because folks don't know or want to take the time to do that, that they invest in APM solutions like Overkill. So they have like all these solutions running in their environments and it's almost like bloatware at some point. So is there a trend you see with apm? Is that something you think people definitely can utilize? And I guess another misconception is these tools will just tell you what's wrong without you having to do any sort of investigation.

Boy, you just, you just hit a real sore spot in that last comment. The tools will tell you we can roll it back to and we can blame Mercury for this. And we can do that because they no longer exist.

Who remembers this sales pitch with Tool X? Any business analyst can be effective in Y tester role.

Beautifully delivered, consistently delivered. Removes a sales barrier, a complete and total fabrication. Completely ignores the fact that the universe will engineer a less effective business analyst. So you're chasing a singularity forever down a black hole. And it ignores the fact that there are actually some skills that are required in order to answer questions related to quality. And so we've been suffering from the after effects of that for a while. Getting back to your question on apm, we can see this kind of APM overkill in many cases with marketing beacons on websites for e commerce. I don't know the last time you went and looked at the developer tools to see how many marketing beacons are running, but it's not uncommon to see Google Analytics, Facebook Analytics, Adobe analytics, add four or five others in the mix, all collecting about 85 to 90% of the exact same thing, including response time. And then there are also so many of them embedded that it's actually slowing down the user experience to the client. If you can imagine an e commerce website that if you removed one beacon, you could drop response time by say, four seconds per page. But the marketing department says that beacon is essential. I have to have it yet. Dropping 4 seconds a page may be worth something on the order of $2 million during a black Friday. Who wins in that case? Usually it's the marketing department. Isn't that frightening? So there is value in understanding, particularly response times and abandonment rates. And there's some really cool ways to visualize this data I saw recently doing a site visit to a client. If you can think of complete end user response time, which for the W3C nav timing metrics, that's probably DOM complete for 99% of cases. Then there is DOM interactive, when you can actually do something on the site and get a response back. And then there's this concept of something called first user event. What happens when your first user event is earlier than DOM Interactive?

This is when you get people just clicking, clicking, clicking. Nothing is happening. You're clicking, clicking and frustration ensues. You're going to get a higher level of abandonment in that case. On the other hand, if that first user event is after DOM Interactive.

You'Re going to get some instance of user delight because you're going to get a response back that I think has meaning to understand whether you are effective in delivering your site to your end users. And that's internal applications as well as externally facing applications. Tracking that delta between your first user event and DOM Interactive I think is critical and I hope that we see more of that type of analysis in 2020. Maybe it's out into 2021 before that really catches fire. But looking at that, yeah, that difference between.

System behavior and response and user behavior and response I think is very telling.

Awesome. So James, I got your views on the front end, the back end CI CD and apm and I can't believe we haven't talked about it yet as we talk about 2020 and beyond and that's AI and machine learning. Near the end of this year, I've been hearing more and more about AIOps and I was speaking to Jonathan Wright and he's off the wall and he was telling me about how somehow when you run in production, there's a method that you can monitor your logs and then use that to create a model of how your users are using your, your application, then use AI to create almost scripts for you that then mimic exactly what's happened in production. Is this something you see actually being realistic in 2020 and beyond or something you've heard of?

Well, I think there is a great play for AI. If you look at what we do inside of performance testing and performance engineering in general, it is very pattern based. And the AI that is being used today is less expert system based and more AI based. You know, it doesn't really ask questions and assign weights. It says, does this pattern I'm, I'm observing match condition normal or condition exception? And if it's condition exception, does it match other exception patterns that go along? So when you talk about resource utilization and things of that nature, I think AI can have a great play. In fact, at aws Reinvent, Amazon announced a number of items associated with AI, particularly code checking and things of that nature. I reached out to them and said, hey, I have some patterns, can I write my own rules? And furthermore, I said, hey, can I monetize those rules? Can I make them available to a whole bunch of people and charge a penny per execution? And Amazon's answer was, we have not considered that business case. So they're clearly looking at their own system to look at code, develop patterns and apply those patterns, not necessarily bring those patterns in from established industry experts. Be it yourself, myself, Mr. Tomlinson, Leandro Melendez, Mr. Padelko. There's a wealth of patterns out here. The first company as a testing tool vendor or an APM vendor that's able to leverage that pattern library that has been developed and is able to make it easy to incorporate those.

I think it's going to be extraordinarily successful in bringing value to their customers. So getting to the issue of production of load, what you're talking about and the example you use is the equivalent of like a DB replay for Oracle. And you know, so we have web replay. This kind of discounts the nature that there is some sort of dynamic element that does need to be Handled, sure. Maybe AI can pick it out. We've seen in tools thus far their inability to really identify all of the dynamic components and handle them effectively. There is still a little bit of an art, just like there is with programming to building a performance test. There is certainly a play for AI and looking at the behavior of systems going through the logs and saying, hey, here's what 90% of our people are doing, here's how they're traversing, here's the most common paths and building a model for what that should look like. And I think there's maybe a subset of those that AI can have a play in actually generating the scripts. And that's for those items which are very, very substantially similar bots. This is an area where representation in load models is very underutilized today. Think of your classical e commerce site. You might have 50% of the load on the system actually coming from bots. But how we test primarily today is we represent user behavior. If we could just get AI to recognize in the logs those bot patterns and then have it reproduce those bot patterns while we're actually reproducing the user patterns, that in itself will make our testing much higher quality than it is today and much better as a predictive measure than what it is today as well.

Great advice, James. So is there any trends you see that I didn't mention? Those are the four main areas. But is there anything you see definitely happening in 2020 you think that performance testers need to be aware of?

I would say this, and you've probably heard this as a theme across many years for me, is make sure what you are delivering has measurable business value. Your stakeholder is not developers, despite what the developers tell you. Developers will tell you they are the center of the universe. Your stakeholder is the business owner. So make sure your reporting has tangible value to the business and is reported in a way associated with business risk. Now, you might have a deeper detailed document that is tailored to development or tailored to ops, but your primary audience on reporting always should be the person who writes the check for the entire project, that is the business owner. Making yourself valuable as a trusted advisor to that person or that group of individuals will always have benefits to you and the organization in the end. Because there will be times when you have to report that somebody's code baby is ugly. And that code baby may be the child of a very influential person who does not want to admit that their child is ugly. They will claim that your test is wrong. You need glasses. Everybody else says, my code child is beautiful and you're going to need the support of the person you become a trusted advisor to to say hey, there is business risk here if we go forward and to have that person act as a check against that out of control but highly influential developer. So just keep that in mind. If you are a part of your development organization, if you are part of a separate QA organization, make yourself valuable to the business ownership. Make sure they know your name, make sure they trust your judgment and do not just report response times. Do not simply bring a sheet of response times. Bring to them valuable business analysis associated with risk. This this is something that was the norm in the early day of performance testing. We moved away from it as we got closer to Y2K in through the first decade and the majority of the second decade into the 2000s. As we enter the 2020s we're starting to come around and you're seeing more articles associated with performance and risk. Even with the cloud I can turn on auto scaling, but bad code will grow to fill the available port pool size. If I just give it a bigger pool size, I've just simply written a larger check and I'm still going to fail. It's just going to take a little longer. So just keep that in mind. Other trends for 2020 I think you've hit a lot of them. It's like hey, how do we reuse functional testing scripts effectively? How do we reuse these individuals effectively? This migration away from dedicated teams more to mixed engineering functions. With Agile and DevOps there is a pronounced trend away from commercial tools to open source tools which have their own idiosyncrasies and the ability to deliver value. I picked on JMeter earlier. There are a whole bunch of other ones Grindr things of that nature that really de emphasize the areas where value is delivered which is in the analysis of all of the data area. Let's see, anything else? Oh yes, number one trend in 2020 explosion of QA based podcasts.

Awesome. All right James, before we go, what's the best way folks can find or contact you?

Oh well there are a lot of different places where people can go to get me but so let's just make it easy. You can just send an email to podcastsamespulley.com.

That way I know kind of where you came from and that will route to me. Of course you can always reach out to me on perfbytes. There's jameserfbytes.com you can hit me on the Twitter. I have two handles over there. There's irfpully and there's ewsofthedamned associated with the news of the damned podcast.

I'm sure there are half a dozen other places from LinkedIn, SQA forums to Google Groups, but I think I've hit the top ones there.

Thank you James for your performance testing awesomeness. For links of the value we covered in this episode, head on over to testguild.com pplikeautomation17 and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome new performance test tool, Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.