AiOps in Performance with Jonathon Wright

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Jonathan wright, all about AIOps and performance testing and site reliability, as well as his take on trends for performance testing in 2020. If you're looking to plan your performance testing strategy in the new year and beyond, you don't want to miss this episode. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, Jonathan, welcome to the guild.

Thanks, Joe. It's great to be here in 2020, opening up a new future of performance engineering.

Awesome. So, Jonathan, it's awesome to have you on the show. You've been on my automation podcast multiple times and I thought I have to get you on the performance podcast because not only do you do functional automation, but you are a full stack automation engineer. So you handle performance, security, all of it. And so today I thought we'd dive into AIOps and also digital transformations. But before we get into Jonta, for the folks that don't know, could you just tell us a little bit more about yourself?

Absolutely, Joe. So, you know, I've been doing test automation for the last 20 years and you know, I was lucky enough to be able to contribute to the Test Automate Experiences in Test Automation book with Dorothy Graham, who retired in 2019 after doing her 250th keynote. And she was the grandfather or the grandmother of test automation. And I think, you know, part of that is it's moved over to performance engineering and I've. Christmas has just gone, you know, my Christmas day message from Wilson Ma, who's a functional guru. You know, he, you know, he's kind of the grandfather. He won't like me calling him grandfather. He's still quite young of automation. And I talked to him about this podcast and you know, and the new year and really what does it mean to be a performance engineer now? And you know, Wilson started his career in the LoadRunner space and then he's now he's a, he's a guru when it comes to what we call digital experience analytics or AIOps, you know, the monitoring tools. And that's why I thought it'd be really important to kind of focus on some of the basics but also talk about the Future as it's 2020 and it feels like the future. So, you know, I'm going to hopefully bring loads of top tips, but also some really relevant stuff which has happened literally this morning.

Awesome. So Jonathan, you did tweet out something today about a bank site that has experienced performance issues. So I definitely want to get into that as part of maybe a Black Friday or holiday type issues that may have came up with performance and reliability problems that people experience. But before we even get into that or AIOps, you know, I want to set the stage. Can we talk a little bit about the basics of performance testing maybe and then go systematically through to, we get, you know, to new age AIOps so we don't overwhelm folks.

Absolutely. And that's really where I want to start. So, you know, I think, you know, when I started doing non functional testing and it was, it was the 90s and I was using Loadrunner and I was lucky to be working with IBM at the time who were masters of the arts of performance testing or load testing. And in those days we used to roll in big controllers on servers and plug them into their networks and do load testing, which was really excellent. But at the same time, what were we trying to do? Well, I think what we were trying to do is we were trying to improve the user experience. Right. In those days it was building enterprise platforms, websites which internally would be used. And I think, you know, part of it was how many seconds and how many users before potentially people are going to start hitting refresh. Right? And that's not, that's what we didn't want to happen. And back in those days it was a Java stack. And I remember it was literally I was lucky. I was working with a guy called Giles Airy, who was the first load runner certified engineer back in Europe, who was working at IBM at the time and I know he's now at Deutsche bank working with those guys doing performance engineering 20 years on. And part of it was there was conversations around, well, what would one gig more of memory look like on the JBoss server to solve this problem and would these performant problems go away and how much longer would we need the load controller for? And this was a really interesting one because we were talking about throwing more memory, throwing more cpu, which was kind of the old way of looking at things. It was also the way that we were looking at things for building scenarios. I think I've heard you say when he was talking to people like Scott and other gurus in this area, that part of it is what are we looking for? We don't want to run 100 scripts all logging in at the same time, because what is it actually proving? It's showing OAuth is working or your LDAP servers are up, but it's not realistic, right? And I remember sitting down with Scott and Dawn at StarWest about 10, 10 years ago and they do fantastic workshops and you know, partly what they were saying was, well, how do you get the basics right? How do you understand what a transactional mix would look like for a day, right? And they were kind of saying, well, yeah, it's the non functional requirements, right? How do you define how many users are going to be using it at peak, at the end, month end and start building realistic scenarios? That was a difficult art. I think that's why there's been so many specialists in the subject is how do you build a realistic load pattern? You have to speak to the business. But this is where we're going to come on to things like digital experience, analytics and AIOps is they take away the need to understand what the transactional mix would look for. But actually I remember sitting and doing a webinar with Wilson Marr about 10 years ago and Wilson said to me, it talked all about PAL performance, Application Lifecycle, and these were a number of free tools that anyone could have access to to literally take the logs from the example we've just given JBoss and build a scenario mix based on all the requests that come through to the app server. That was fantastic. I wanted to really understand, and this is what I want to unlock for you today, is why haven't we moved on from those days? Why haven't we still using these tools to create realistic scenarios instead of just looking at load testing and hitting something quite hard? That's where I want to explore today.

It's a great point. And I believe you actually monitor some of the top retail sites and you produce a report. And I'm just always curious to know that, you know.

Companies know that performance is important. And I've just been told a lot of them just say, oh, we'll just scale up. So a lot of these companies are scaling up with their service so they don't think they're going to have performance issues yet. I believe every year still we have these big issues. So can we dive in a little bit into the report you have and maybe some key insights into what companies may be doing wrong and not focusing on the right things about how the users are interacting with their websites?

Absolutely. And you know, I was putting the turkey in and then jumping onto my performance machine to start doing performance testing on Christmas Day. It was actually the highlight of my Christmas. And also, as you said, Black Friday, Cyber Monday, doesn't matter how many times people and I know the other guys from who do the Perf podcast, you know, they also talked about, you know, they did a Black Friday special and, you know, part of it was why is this still happening? Right? And why do we still see, you know, Costco go down and them lose 14 million over that period when you know it can be prevented? Right. And you know what we've seen, we've been doing an independent report for the last four years now. Uh, it's the World Digital report. So it's worlddigitalreport.com all one word, completely free, completely independent. And what we do is we look at the top 10,000 retailers which cover EMEA region. So France, UK, everywhere in the across EMEA APAC. So we cover the Chinese areas. We talked Singapore, Australia and the rest of Australasia. And that's been done by a good friend of mine, a guy called Stuart Moncrief who runs the, you know, myloadtest.com which again is a completely free help people start with their Loadrunner and JMeter journeys. So I go and check that out. And then also my good friend Todd DiCapria, who's based out in Americas, covers the American region. And I did have a call with him on Christmas Day as he was kicking the stuff off in the States. And he's been working for a major, probably the biggest investment bank on the planet out in New York. And he's been working on AIOps and monitoring his code and site reliability for the last two years. And he's got some incredible insight. You should definitely get him on the show as well. But part of it is he wrote this great book called Effective Performance Engineering from O'Reilly, completely free book to download. And it literally both Todd and a guy called Shane Evans, who used to be one of the product owners for the Micro Focus tool sets, got a signed copy in front of me at the moment and it gives you every tip on how to set off the basics. So what is performance engineering what it means to your brand, what it means from your business needs. It's also got, which I think is probably the most comprehensive section, is it's got a whole stack of questions you can ask to your operational guys. So it literally says things like, there's a section around external systems, so where's your cds? And these are the big problems that we see is the CDS is how many times do you jump to different websites to get different content, Right? So if you've got Google Analytics, which I'm actually looking at my screen at the moment, at the Google Google Analytics app for a big retailer in the uk, I can see at the moment what people are looking at. So people are looking at the Boxing Day page. I've got six users that are sat on the shopping bag. But each, even though I can see that there's 256 users on my website right now, and 78% of those are mobile, coming through a mobile channel that has an overhead on the website because it has to, each time a page loads, has to send that information in a JavaScript library over to Google, who then take that information. So the more of these tools that you've got, I'm not going to call it bloatware, but these APM tools, these quite heavy apps, suddenly start bloating down how long your page takes to load. So this is where I'd like to start this journey, because this is one of the things we focus on from the World Digital Report, is we look at what is the actual time it takes to load your website. This is the Pepsi Challenge that I throw to every single person in the performance engineering landscape. How long does it take your webpage, your desktop webpage, not your mobile app, because that's just sending a whole stack of APIs, quite lightweight. You've downloaded the app for 60 gig, so you don't need to download that information. How long does it take your website to load and does it do it? Less than six seconds. And I can tell you, out of the 10,000 retailers I'm looking at them right now, there's very few that managed to get under six seconds. Especially when you take in things like network virtualization, so you can do packet loss.

Time to retry. And also things like you're on a 4G network or a 5G network or poor 3G network as you're going through, you know, some traffic. And I can tell you all the top 10 websites that came out this year in the States. So this is people like Bloomingdale's, this people like Footlocker, all of them are over six seconds to launch their website. And that is poor because by, you know, the old benchmark used to be anything over 3.4 seconds, people would move away. Now that's moved up to six, which is helps with this bloatware. But still six seconds. Would you wait for six seconds on Bloomingdale before you just jump to a different website to buy your product? You know, part of it now is we're in that kind of economy. And that's why, you know, when I tweeted you this morning around this Danish bank going down, I was kind of saying, with open banking APIs, what percentage of those customers are going to say, you know what, I want to use my bank, it's gone down today, actually. I'm just going to switch my accounts to somewhere else. And that's something that the people in the US do like they're, you know, switching their washing powder. Right. Whereas in the uk, we're quite loyal to our banks and we've literally had some of the biggest banks go down over Christmas, right? We've had tsp, we've had hsbc, all experience downtime. So I'm going to give you my first top tip is there's a little application you can download on your phone or you can go to the website and you can see if people, if your website, your favorite website's gone down. It's called Down Detector, right? Literally I can look at it now and I can look at every single website and which ones are going down. Uber Eats is at the top. PlayStation networks having problems, Amazon's having problems, Netflix is having problems, the Call of Duty servers are down. These are just what's happening as I look at the app right now. Is that acceptable? And the answer probably is, no, it's not. And scaling to the cloud, moving it to somebody else's data center, adding more memory, doesn't actually mean it's going to fix the problem. So that's what I want to explore with you, Joe, right now.

How do we fix the problem then?

Well, what I'm going to give you as this is, I'm going to give you a partly live podcast session right here. You give me any website you can think of, your favorite website I'd recommend, just to avoid the disappointment of giving me your website. But you give me a website which you would like to know how it's.

Going, let's say USA Today.

I put that into my load. I'm starting to run. All I'm doing is I'm running A browser proxy. In this case I've got nv, which is the network virtualization enabled and I've also got.

Just a headless proxy. What it's doing is it's. It's looking at.

When the DOM has finished rendering, when all the JavaScript libraries have finished. When you're opening up USC today, you're seeing the page as it loads up. Actually if you look at the bar at the top of USA Today, that bar until it gets to the end is when all the different libraries like the Google Analytics, like everything else is actually loading up. I'm looking at it in real time now and I'm getting a quite good response time. But its performance grade is a D. That's what it's been given. It's just 6.21 seconds. It's come back, it's just over that six, but it's okay. The page set size is 2.1 meg. It's got 385 requests to get that page up. That gives you an idea of all the CDN content it has to wait on. Even just using the normal API, which I would recommend to any new starter is things like the Google speed APIs. It's already telling me the reason why it's failing is a whole stack of URL redirects. There's a whole stack of requests, as we know about that already, Compressed components with gzip that's not acceptable in these days. Images that have come in from third party websites that haven't gone through some kind of content management and components impression means that 90% of that traffic is bloated. Even worse, I'm looking at the response codes and actually there's two 404s not found, there's a few forbidden, which means it's probably in some restricted site that EV tries to go through. And the majority of that content, which is quite good, 1.3 megabit is just JavaScript libraries. They're going off to things like Google Syndication, they're going to an ad site called Moat Ads, they're going to Tobola.com, they're going to connect.facebook.net so if Facebook goes down, which it has done in back in March and then you've got things like doubleclick.net and all sorts of other stuff. There's a huge amount of requests that are going on there and there's also a huge amount of time. So it's actually waiting over 3, 400 milliseconds just for this user script to execute. And that's just one script out of hundreds of scripts that's gotten the page straight away. That would to me would set off a whole stack of concerns around that website. Right. But what I'm going to do, which is a bit of magic and so you can use to get that kind of information, you could look at something like Pingdom, which is just a pingdom.com. takes you 30 seconds just to do that report. I used Loadrunner 2020 with the network virtualization. I use the community version which is completely free. Right. So you've not paid a penny. But now I'm using a bit more of some of our special source and I can look that they've got site cast list by Adobe. They're using Dynatrace, right. So that I can tell by looking in that, that those headers that actually they've got Dynatrace enabled and I know Dynatrace full AI powered, you know they're top of their class so you know that's great but they should be doing more of it. They've got live ramps, they've got a ramp for, they've got net mining on there, they've got.

Visual Iq, they've got a whole stock of Facebook domain insights, they've got new relic on some of their site site components as well. You know, they've got some Twitter analytics, they've got Google Analytics, they've got conversion tracking by Facebook, they've got web analytics from Yahoo, they've got Telium that's being plugged into there. They've actually got what we're using right now. They've got Zoom video conferencing built onto their site as a link. So you must be able to chat to them using that. Then the list goes on. They underlining tech they built with ASP Net. So I'm resolving what type of app server that's on there. They're actually using some of the Google APIs to check their speed. They must know about it. They've got an iPhone mobile compatible widget that converts it into a website. So that might reduce the overall sight of it. They're using the SDK for Facebook using Jquery. You know, the list goes on and on, right? And so that's within 30 seconds. I can look at that website and I can pretty much tell you what you know, they've their devs and their ops have gone to town on every single plugin they can possibly get to get insight and ads and make revenue. But this is all coming at a cost, which is how long you have to wait. For this webpage to load.

It sounds like they're doing everything right. Yet because of that, it's almost bloated because you could put these JavaScript things in your headers and go to town like you said, but yet it seems to be detrimental almost to some of their goals.

It is.

Like I said, a lot of these give you great insight. And this is why I really wanted to talk about Digital Experience analytics. Right. So this is kind of customer experience analytics on steroids. So this is talking about digital systems, talking to each other. So mobile is talking about your Alexa app, it's talking about everything what goes on behind the hood. Right. And I'm looking at the digital experience analytics tool which we use from, from Eggplant. Right. And I can see, I'm just looking at the website we looked at, right. And I. It's saying that if they reduced 1.67 seconds off that six second response, that they would increase their revenue, if this was a revenue generating page by 25%. Right. They'd increase the amount of conversions, they'd have less bounces. You know, they're saying that you'd get an extra 10% based on what they've got at the moment. 10% of bounces. That's people navigating away from their website because they, they just can't wait for that content. So, you know, yes, News Today isn't a revenue generating site, but it just shows you the potential impact of maybe one to two seconds is going to make on how many customers you get and how many customers navigate away. So, you know, this is why, you know, I think this is a kind of, it's not an art. But you know, there's plenty of tools that will help you and I've named a few there, but where you can get information that can start your journey off. And that's why I kind of wanted to talk to you really quickly about this website that went down today. So one of the things that we do with the guys who actually referred me onto this, when James from Curiosity Software gave me a shout out on your last podcast around the AIOps kind of thing and how you shift, right, and you build the models from the, the different behavior that's happening with your real users, part of that referral was how do we keep an eye on websites and make sure that these realistic scenarios are happening so when they're actually real world scenarios do happen in Cyber Monday, Boxing Day, January sales, your site doesn't go down. And so this example which I tweeted you this morning was A Danish bank. Their website went down and there was about four or five people on Twitter straight away jumping on saying, you know, I need to access my bank account and I can't get into it. So I'm not going to name the bank which I normally would name and shame based on what we do on the World Digital Report. But this bank, you know, I'm just looking at the report, I've run it again, so I've just literally gone to pingdom.com so go and have a look. The website in question has got a degrade and again over a six second load time which is not great. Then straight away I've run some additional tests on it. I've run some security which looks good. We give it an independent score. So we run security performance usability. We use sentiment analysis to look at social analytics. We look at what people are saying. Now this particular bank was Danish so we were actually asking to use the Google Translator to say what people had been historically saying. So they got 7,000 people following them on Twitter and some of those people have reported some issues and one of them is saying that they've recently started supporting Apple Pay and Google Pay support. But their website has got, the new website seems to be loading slower than the old site. So someone literally, a guy called Daniel has tweeted to them and said my old app used to take eight seconds to launch. My new app takes 25.32 seconds. And he says it's crazy, it's like being 2009, right? So this guy has literally somehow kind of got an idea, maybe sat there with a stopwatch which is the way they used to do it, right? But based on those social analytics I pulled out that this was a negative response. So that's the, just the MIT libraries you can do for sentiment analysis which has flagged me as social media for this bank who went down today. And this guy's saying that the new app is slower than the old app, right? And you know, I'm sat here looking at new relic at the moment, right? And there's a, there's a mobile plugin to mo to the new relic. So I'm just going through the mobile app analytics at the moment. One of the, if I go down on the left hand side, one of the things I see is versions. It can tell me between releases has my crash analytics increased as my exceptions increased as my overall feedback on the app, I can look at that version trends against OS version. So I can look at the different devices, I can look at the form factors which is really important to see if it's an iPhone search versus a latest iPhone 11 or 12 or whatever the latest is. I can look at maps, I can look at my HTTP requests, I can do my request response pairs, I can look at geography, I can look at carriers, I'm on say at and T and I'm getting more problems because maybe the DNS are not resolving. There's all these problems. This is me looking at it as we talk on this podcast. There's no excuse for bad performance and the tools are out there to actually, you know, recommend so that pingdom for that website, you know, straight away it's already giving it five or six big recommendations just looking at page speed, load and why slow. Yslow is a great little tool. You can add it in both of those apps you can add in as a into your chrome so every page that loads up, it can make recommendations about, you know, so go off and have a quick look at those tools and that will give you an idea of, you know, some of the bloat that's in there. And you know, we shouldn't be dealing with these kind of problems. You know, I think people have forgot about the desktop and they've, you know, focus on mobile, they focus on mobile apps and you know, I'm here sat building an app for a big retailer and one of the reasons you're doing it is because that bloatware is included in the actual initial package. So actually what you're doing with request and response pairs is very low. But I'm still using, you know, certain vendors.

I've got logging, I've got plugins that are going to give me the more debug information so I can create my shift right aspect. So in my case I'm building this act, this app on React native which was what Facebook actually is their code base. Right. So Facebook use React Native. But in actual fact you need to be able to support their particular framework. What React native does is it creates iOS version and Android from the same code base. The great thing about that is you only have to maintain one code base. But it also means that the framework needs to or the SDK needs to integrate directly into that particular platform to give me the digital experience analytics that I need to be able to tell me about user journeys. So this is literally if you, if you've ever looked at Google Analytics and you can go and get a udemy course on Google Analytics for, you know, $6 or something in the new year, you can tell what your user is, where they're going down the journey, what data they're using, you know, what they're buying, how long they go on each page. And that blueprint, that digital DNA blueprint of what that user is doing and the behavior of that user can then be used to create a model which is what James and the Curiosity guys were talking about, is that you, it gives you everything you possibly need. So that's the value of using these analytic tools or these digital experience analytic tools. But I think this is where most companies are going wrong. And I know I mentioned Todd DiCaprio before, he's been doing this with a lot major world's biggest investment bank and he's doing it so that they can build the models of the behavior of them using the mobile app, using Alexa to transfer bank money across or do investments. And each one of these platforms he can monitor and reverse engineer the actual synthetic tests to run in production to make sure they get a heads up.

From a self healing perspective. If a open banking API goes down, if a particular one is starting to get errors and it needs investigating and they don't wait for the kind of incident that we've had today where ops and dev are on holiday over the Christmas period, the website goes down, the app goes down, no one can use their mobile banking. And then suddenly the brand damage associated with that and people going, well actually I don't want to bank with the banks that go down so much because, you know, I can't get my money. And we saw this on Black Friday with NatWest in the UK. So NatWest went down on the biggest day of the year when people wanted to buy stuff, they couldn't even access their online banking so they didn't know how much money they've got to spend. And you know, this, you know, is not acceptable. And you know, going into 2020, let's do better performance testing or performance engineering as a, as a New Year's resolution.

So how do we do that though? How does, where does AI come into play? And do you think that's a real strategy for folks in 2020 to help them with these issues?

We all know AI is kind of the buzzword at the moment and this is why I was kind of saying, yes, AIOps is, you know, your flagship APM tools on steroids like the one I'm looking at the moment, you know, you know, your dynatrace, you know, having AI built into it, you know, I'm looking at New Relic 1 at the moment and I've literally got a distributed tracing for complex systems. I'VE got my Kube Cluster Explorer, I've got my AWS.

Integration with it and I can see from a single view. I can do analytics and work out where the issues are, where should I invest my time. I can create synthetic tests to kind of COVID that off to keep an eye on those things. I can do predictions based on historical information and that's great. But at the end of the day we need to start off with let's build better, more realistic performance tests, right? And that means we need to be able to just have access to the basic data that we need. And you know, I'm just going to give you Google Analytics, right? As an easy way straight away. I look at the reports, what I'm looking at the moment, I can tell you where the traffic is coming from. I can do a few scenarios that are opening up in a web page, opening up in a browser from your email client. I'm going to give you this is a big tip. This is an issue we've just had which we've just fixed, which was if you are on Facebook, which is the eighth biggest traffic generator at the moment and five users have actually just come from Facebook. So I'm looking at those real stats right now, right. The Facebook Organic is what it's called the medium. But I know that when Facebook loads this retail page there's an issue and the issue is it loads Safari, but it loads it within a cider window. And if you've noticed that when you're in Facebook that it's not opened your Chrome, it's not open the full fledged browsers, it's a cut down Mozilla kind of engine and when you're running in that cut down functionality, you don't get the same functionality as your full browser. That we've got an issue where if you're using a particular Android on a particular version of Android 6 on a particular form factor, it won't load. Now this is a bit of a challenge because you've got emulators. So I can load Android Studio up and I can get an emulator off an Android 6 running on.

A Google product, then I can recreate that scenario. I can load Facebook, I can click on the link and get the same issue. I can go off and use AWS Labs, I can go off and use Perfecto Mobile and use real devices and then realize that actually the emulator doesn't show me everything. I need a real physical device, which is why actually I'm sat here with a stack of Huawei Nokia, I've got an iPhone se, I've got, I've got an Nvidia tablet, I've got a whole stack in a rack. So I can actually validate these kind of issues. But I think this is where we're starting to see this transition between functional so me using a real device and automation and maybe Appium using the different techniques because obviously you can't use the Same automation for iOS as you can for Android. So this is where we're starting to see AI come in for functional tools like the UFT1 platform which allows it to understand what a login is and it doesn't care really if one is login or log off.

It will understand and learn that part of the Android rendering is the homepage icon. The three burger will look different to what it does on Apple. And because of just the permutations of how the the SDKs work, it has to learn how to navigate through. One of the bars might be on the right side, which I've got at the moment. I've got an issue where if I open it on Android, the bars at the top, if I Open it on iOS the bars at the bottom, these are all problems. Where iOS you'd have problems with your automation. These are the same functional problems you've got with performance. But in the old days the idea was non functional, was we were not checking for functionality issues. So there was a lot of in the old days culture where if they found all these issues which were functional issues like the Facebook opening micro browser to open the page but then not rendering, they wouldn't report that because that was just noise to them. So you know, when I run, which I did on Christmas day, I expect 1% of those transactions to actually fail. But I'm not going to investigate all that 1%. But within that 1% I'm going to probably find a whole stack of functional issues that I could fix. But in actual fact, to me as a performance engineer, I'm only caring about my cascading KPIs, my 95 percentiles, my, you know what, all those response times, looking I'm looking at the raw data. I don't care about the average response time because that hides some of the information. I have to look at how I'm polling to see what sample size am I taking. Because you can find that Even if the 99 percentile is one person taking 45 seconds to load that page up, then straight away that person will have navigated away and we really need to care about those people. But using Google Analytics, using These other digital experience analytics tools, you can look at where your users are using the application, how the conversions are getting blocked, at what point in the pipeline. I'm looking at the conversion rates at the moment, I can see at the shopping basket, I can see when people, the bounce rate increases and I can start thinking to myself, why is that bounce rate over a certain percentage on Android compared to iOS? Is there a functional problem in that journey? This is where you've got to work with the functional guys, the automation guys, to make sure they're creating scenarios that try those different combinations and you can then leverage what they've done as far as the models they've created, the automation that they've created, the Selenium scripts that they've created. And you can convert Selenium, you can use something like Taurus. You can literally throw Selenium scripts at it. You can use Taurus. It's an open source platform built on JMeter and it will convert anything to anything. So I can give it a Selenium script for my functional guys and it will generate a JMeter script out of the other side without me writing a single code. And I can run all their functional tests to create my non functional tests. And I think this is where we need to harmonize together. We need to work together with the functional guys and build better tests.

Okay, Jonathan, before we go, is there one piece of actionable advice you can give someone to help improve their performance testing efforts in 2020 and let us know the best way to find or contact you.

You know, I'm going to finish with two kind of big tips on there. You know, there was a few incidents this, this, this, this Christmas period, you know, Whirlpool was a great example where their website went down and they've just had the biggest recall that's ever happened. And then the number that they gave in the packaging was an incorrect number so no one could get through to the customer services. And you know, I think things like this where you've got a faulty product or a defect product and you can't get to the website, you can't get to the phones.

You can't reach out to people, these are real problems. I think part of that and I'm going to give just two people here also to check out. James Whittaker's come back into the industry. He's just left Microsoft before. He wrote the book on how Google tests. And I remember 15 years ago he said there will be no apps in 10 years time, which has kind of gone past. But I think relying on building an app to do something is probably not the best way of doing it. Neither is a website. We need to get smarter on how we're moving in 2020. My friend Theo Priestley, Priestley, who writes for Forbes magazine, he said last year, he said where there was an app for that, there's now an AI for that. Now testing AI apps are even more complicated. Putting messages on Kafka, seeing how long it takes for test flow to recognize the image and the misclarification rates. Suddenly we've got a whole stack of next generation technology coming through and people need to be able to start preparing for that. I think digital experience analytics is incredibly important, but we need to stop focusing on a webpage or a mobile app. We need to think about the entire customer journey. You know, from buying the washing machine to finding out the washing machine's being recalled, to being able to, you know, book it in to get picked up, to get it dropped off and the notifications coming to say that you've got your new one replaced with the new fixing it. That whole journey is what's important. The whole digital experience is important, not just a single user interface or a particular omnichannel. So that's my big tip and hopefully we're going to get some really good people on the Perform, you know, Performance Guild, the automation Guild who can help you achieve that automation greatness and performance greatness and also security greatness. You've got to check out that podcast so www.LinkedIn.com in automation easiest way to get to me My twitter is Jonathan Wright. I'm incredibly easy to get hold off, reach out, more than happy to help and.

Keep doing the good fight and let's see if we can improve performance engineering for 2020.

Thank you Jonathan once again for your performance testing awesomeness. The links to everything of value we covered in this episode. Head on over to Test Guild.

And while you're there, make sure to click on the Try it Today link under the exclusive Sponsor section to learn all about SmartBear's awesome performance testing solution. Load Ninja so that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey this has been a Joe Calantonio production.