UX Performance in Continuous Testing with Guy Arieli

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Guy Arielli all about UX performance. Guy Guy is the CTO of Experitest and he has a really strong record in technology innovation with more than 18 years of development and hands on experience in test automation. He actually has prior experience spending time in top management positions at hp, formerly Mercury, Cisco, three Comm, a whole bunch of different companies. In addition, he's founded and sold the largest local test automation service company, TopQ, to a leading publicly trading technology group, TopGroup. Guy also leads the largest test automation forum online and is a keynote speaker at events worldwide. Really excited to have him on the show to talk to you about this really hot topic. You don't want to miss it. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work even for simple playback load Ninja cuts out the dirty work by using AI to inspect and debug your code or almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Guy, welcome to the guild.

Thank you. Thank you Joe. I'm so excited to be here. It's really great, awesome.

It's great to have you. Before we get into it, is there that I missed in your intro that you like to share with the guild?

No, no, I think it was a very good intro. I'm a father for four and I like to like it's not really like I'm running marathons so this is a hobby.

Very cool. So marathons to me remind me of performance being in peak, peak shape. So that's an awesome topic that we'll be diving into today. How do you define UX Performance testing to understand ux.

So first of all I think that if I'm taking an application, a mobile application that is rating in the app store and out of the five stars that you will get from your customer, between one and two stars are performance based. So if you have a very lousy performance and the application is great, you will get between 3 and 4 point. If you want to get the 5 or 4.5 points the application will have a great user experience in the sense of the performance. I'm not talking about the user experience in the sense of how many clicks you need to perform in order to do some operation or how easy the application is to understand. And then within the user experience performance, the response time for a certain page is the number one metrics. And with that speed index is the most important metrics. And after that I think that the battery consumption and the CPU consumption that goes end in end is very important because if you will consume too much battery and CPU out of the device, people will remove it very quickly.

You know, it seems like most of your background is around functional test automation. Why do you think user performance is important?

You know, it is super important if you are in a competitive market. So if you are in the banking industry, retail, it is super important. We see that. And it is super important for mobile. I think that the mobile performance or mobile user experience performance is more significant than web and the mobile and in general the digital transformation. Starting with companies like Google, Apple, Facebook, those companies are excellent in providing excellent user experience. And then a follower are coming and they are trying to replicate the same user experience.

People are used to this level of user experience and this is becoming de facto standard. So if 5 years back 5 second response for a page was decent, today the expectation will be to have two second response.

Absolutely. And you just brought up a point about mobile performance being more important. I've heard a term about dirty networks. I don't know if. Why is mobile performance important, do you think more important that people may not think about when they're thinking about the performance of their applications?

Yes, because usually when you work with your browser on your desktop, you are connected to a wi fi and the bandwidth that you will be using is relatively high compared to anything that you will find in 3G or even 4G, specifically in changing network conditions and where the reception is not very good. So in mobile the networking part is much more challenging. You have packet loss, you have latency and you have other measurements that disrupt the networking part.

So, you know, I think most people may think about performance, they just think about user speed and they may forget about things like latency. Are there other critical factors that actually go into a user experience when they're trying to perceive performance?

So one of the measurement I think today are the most important one. Obviously duration is important, the time it takes for your page to fully loaded. But I think that people should be aware of speed index and I think that the speed index is much more Reliable and important than duration because it takes into account the actual user experience, how quickly the user interface responds to the user. And so this is a metric that people should be aware of.

So how have you seen performance change over the years? Just curious to get your take on it where it was before. You have a lot of experience where it was before and where you see it now. Maybe where we're going so people get a head start on it maybe.

So first of all, the expectation is rising. So people have higher expectation from their application. This is one thing and this is something that I see all the time, the expectation.

First of all, the application should work on the functional level. But then the performance is the next most important part. If we are looking specifically on the QA process, I see that the performance is shifting left into the pipeline. So the need in a QA organization is to be able to measure performance as early as possible. What we see usually in many cases that the performance bug are found in production and then the cost around them is huge. So I see a huge demand to be able to have performance feedback right in the same time that you have functional feedback to your developers.

So how do you get that feedback? A lot of times you only see performance issues when it's actually in the wild. So how can you get people feedback and not give them false, false issues? Maybe where they'll just start ignoring performance. Like they ignore test that fail consistently that are automated.

Right. So I think that it's not so difficult to get to test performance in your lab. And I do remember that I heard an interview with one of the developer of YouTube and he said, you know, I was is a British guy and he said I was driving in the tube in London and I was verifying that my videos working fine when I'm underground. And this is a very good example. So you need to run your functional test in bad network condition. I see that 3G should be the default configuration of all of your functional testing. This is how we do our testing here in Experitus. I don't care how the application performs in WI FI or in Ethernet network condition. It's not interesting because it doesn't match the actual condition that the application is going to meet. So just instead of using, if you are doing mobile testing, instead of testing it when the application is connected to Wi Fi, use network virtualization and just simulate a 3G network condition. You don't need to be too fancy, doesn't need to go up and down, just constant 3G will do the work.

So besides the tips you just gave there. I know there's a trend towards folks using containers is a big trend. Does that impact UX performance at all?

When I started a while back, all the application was a monolithic application and the focus of performance was on the backend. The bottleneck of performance was the databases, what the application server, the file system. As people are using kubernetes and the dockerized environment, the backend is not anymore the bottleneck. And the bottleneck is shifting into the client side. So the bottleneck is in the network and in the device and the application itself.

If it's in the network and the device itself, there are a lot of configurations. Is this where a cloud provider can help organizations? Because obviously it takes a lot of work to set up your own labs and then have all these configurations for experit test for example. Is there anything special that allows people to get this type of data that they need?

Yeah. So first of all the important part is to collect those statistics continuously and over every test that you perform. And the ability to know to separate your application into what we call transaction. Transaction is something that starts with the user interaction on the device and it ends when the device finish rendering the screen in the middle. There is a request to the server, a response and then the application render the results. So we collect this information all the time over all of your testing and then we are able to benchmark a new version against the oldest one. So we don't know whether 4 second is good or bad for your login transaction, but we can tell you, you know what it changes from 2 seconds to 4 second then it's probably a problem. You should look into it. And measuring one transaction is not. Is meaningless. You need to. In order to get a certainty you need to measure it hundreds and thousands of times and then we can tell you we are 98% sure that there is a problem in the login transaction.

So what are some key maybe UX performance metrics people should be following then across the lifecycle to help them with this?

Yeah, so duration and speed index are the first and most common. Then there is the CPU max and average during this transaction, the memory.

Of the application max and average, the battery consumption during the transaction, and the data that was uploaded and downloaded during the transaction. So I see those as the metrics you should follow and monitor and monitor any change over versions.

So are there any lightweight tools maybe developers can use earlier in the lifecycle to help them gather these metrics? Like do you think Google Lighthouse or something like that can help?

I'M not too familiar with different tool. I know that for mobile it's quite challenging to get those metrics and it cannot be collected over emulator and simulator. You need to use real devices.

That's an important point. Real devices critical that you're emulating in the real world for sure?

Yes, absolutely. I'm not absolutely sure that that it need to be in the exact region because it's relatively simple to simulate the network for in experiments. We have 10 data centers and we have devices in Singapore, Australia, UK, Israel, the US and Canada. But I'm not sure that it is very important for performance to test it in the exact region. What you do want to be able to simulate is the network condition.

Especially the latency that is being derived by different location.

Are there any requirements you think teams need to have in order to be successful, say with continuous UX performance testing?

Yes.

So one of the things I'm seeing is that performance, it's not just shifting left, it's also shifting into the functionality team. So the requirements today. So we have the load engineers that will bombard the backends and test the bottlenecks around the connection to the database and all of that. But all of the user experience today is shifting into the functional team. So even if you are a functional engineer, you need to have a sense on how performance work and what impact your application performance.

So that was actually one of my next questions and I think you touched on it. How does functional testing fit in then? How do they get involved in performance?

Right. So I think that functional engineer can relatively easy today get into the performance testing.

Not on the load side, but on the user experience side. So learning the different metrics, that is important for user and it's quite easy to understand what those metrics we said that speed, index, duration, battery, CPU upload and download. And also they need to have a sense in networking, things like DNS resolving and then SSL handshake and things like the amount of data that need to be downloaded. All of that impact the performance of your application. It's not too complex. If you using Chrome and you open the debug tab of Chrome, you can go to network and it will show you the waterfall view. And this is the essence of what happened in the network. It will show you how much it takes to resolve the DNS host and how much time it took to download the actual data. And you can learn a lot about that. But to begin with, I think that there are two level for that. First of all, you need to monitor the different transaction and find that there is an issue, then the next step is, okay, I understand that there is an issue in my search transaction. Now I want to understand what is the problem. Maybe the developer changed the content file, maybe we have a problem with one of the services and the API is slow. Maybe, maybe the DNS is problematic in this specific environment. So there could be a different reason that you can categorize performance, user experience, performance problems.

Great. Now I'm not belittling manual testing. It sounds like a lot of manual effort, but as we're trying to release software quicker, we're shifting left, we're also shifting right. How do folks fit this into their pipelines, the modern pipelines, where they're getting this information on an automated basis? I mean, manual is an important component of it, of course, but is there any way to get this information as well during a release of software to the pipeline, the different stages along the way?

Specifically in mobile, what we are also seeing is that it's not just shifting left, it's also shifting right. So we see more and more customers that will use their APM system like Splunk, Datadog, Elastic and they will have devices in different location running the same script over and over again, once every five minutes or so, monitoring different transactions within their application and then be able to identify performance issues very quickly. So it is what is called synthetic data that can trigger alerts when your application is performing bad in production.

So I guess my question a little more detail was developer checks in code and they have static code that runs, runs a unit test and then probably kicks off their functional automated test. Does performance test also include. Is UX performance test also included in this as well in the pipeline?

Yeah, absolutely. So in the end it is just, you know, if you collect all the information about each transaction, all you want to do in the end of your functional testing is ask the following question. Did one of my transactions perform worse than the last free version in more than 5%. If this is the case, I want to fail my regression.

So how does this work? In experit test, do you have functionality specific to UX performance? Like can you walk through an example of how maybe functionality in your infrastructure, so on. If they release code, they would know how it all works.

Yeah. So very common use case is that you will take your Appium test and within your Appium test we can identify the transaction for you or you can add two lines of code in the beginning and in the end of the transaction you just need to name the transaction and you need to provide the network condition. You want this transaction to be executed and then everything is getting collected. So the video of each transaction, the HAR file, all of the metrics that we have talked about. Then in the end of the execution you have an API that can failure build based on regression in those transactions. A very common example you can say if my login transaction a performance change in the matrix of TPU memory duration speed index in more than 5% compared to the last 2, 3, 4, 10 last build, then I consider it something that should fail my build.

So it's great to know that performance has gotten worse or better. But any tips for analyzing and debugging UX issues when they are flagged?

First of all, I think that the most important part is to identify it in an early stage. Once you have identified we can show you some worst and good example. We can identify some anomalies. So anomalies is not just that we see that it goes to the worst. Sometimes an anomaly will be that suddenly it change to the better. Instead of four seconds it takes one second. This is also a problem probably either you have created a very big improvement, but in most cases something is going wrong and the response is not being handled at all. And we are able to guide you on where is the problem. And I mentioned that it might be in the DNS, it might be in the SSL hand check, it might be in changing the amount of data that is download every transaction. It can be related to caching and the distribution of your files in Amazon. So there are a lot of guidance that you can get on where is the problem.

Okay Guy, before we go, is there one piece of actionable advice you can give to someone to help them with their performance UX testing efforts and what's the best way to find contact you or learn more about Experitas.

So I think that I really encourage any functional engineer, functional automation engineer to learn more about performance. I think that this is a highly needed requirement for functional engineers and this is going to grow over time. I think in order to to get in touch with me guy guiseperitest.com this is my email. This is the best way to contact me.

Thank you Guy for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to test guild.com P29 and while you're there make sure to click on the Try it Today link under the Exclusive Sponsor section to learn all about SmartBear's load ninja performance testing tool. And if you haven't heard already, opening for the third Annual Online Performance Guild Conference is now open. Perf Guild is an online conference dedicated 100% to helping you succeed with creating end to end full stack performance testing awesomeness. So our goal is you learn a tip, tool, technique or best practice that's going to help accelerate your performance testing efforts, whether it's a new project or an existing performance testing project. And best of all, we have 100% guarantee that if for some reason you found no value in the conference whatsoever, you get a full refund. I think with the only conference actually has a money back guarantee so you have nothing to lose. Register today. Head on over to perfguild.com so that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.