The Prestige User Performance Method with Ankur Jain

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Ankar Jain all about how to measure UI without APM tools and other performance topics. Ankara has a degree in Information Technology and has over 5 years of experience in performance testing and engineering. He has an extensive experience in tuning client side UI performance of web pages. Currently he's working on a storage and backup domain for a SaaS AWS based company. So he's taking care of a lot of backup performance, capacity planning. He does a lot of things with Python, Go code level performance, bottleneck identification and tuning and and more. Really excited to talk to him today about all these things. You don't want to miss this episode. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work. Even for simple playback Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Anka, welcome to the Guild.

Hey Joe, thanks for having me over here. I've been following your podcast for quite some time so it was a really pleasant surprise to get an invite here. So thanks for inviting me here.

Awesome. Thank you so much. I'm really excited to talk to you like like I said, you have a lot of experience and a lot of things that I haven't even touched on yet in this podcast. Before we get into it, is anything I missed in your intro that you want to tell us a little bit more about yourself?

So you introduced me better than I can introduce myself.

So thanks for that. Just basic about myself so I am based out of Pune, India. I have around five plus years of experience in performance testing and engineering. So I have been doing a lot of engineering of banking domains, telecom domain and currently working for a SaaS based company, storage which works on storage and backup. So my day to day tasks include taking care of backup, performance and restore performances and doing capacity planning and as well as deep diving into the Python and Go level code. Beside this, I like traveling a lot and playing a lot. But you know, these days, these Things are not possible. So I'm just relaxing at home.

Very cool. So Ankar, I thought we'd just dive right into it. I want to go over the presentation you gave at the Performance Advisory Council this year on how to measure UI element page load times in production without using APM tools. So the first point in your presentation was user perceived timing. Can you just talk a little bit about what is user perceived timing?

So there are various studies and and various research going on and have already been taken up. So in one of them, one of those study conducted by Speed Matters, it says like 75% of the people actually think about the page at the speed at which the page loads. So yes, the speed matters perfect matters for a UI experience to have a better user experience. So I have a little story to tell you. Maybe you might have heard about it. So it's about Houston Airport conveyor belt. Have you heard about this?

No, I haven't.

Oh great. Then I'm going to ask you a question after the story and maybe you can answer it. So it's a very simple question. You will have two options and I will give you a life and ask a friend so you can ask me if you don't know the answer. So here we go. So the story goes like this. So to clean your baggage after your flight has landed, you have devoted the plane. It takes around 10 minutes for a passenger to get his hand on the baggage. So people were really very frustrated. So there were a lot of complaints regarding this lag before they get their baggage. So people at Houston airport, they were very.

Concerned about this. So they had a meeting. So two things come up in that meeting. So these are the two options. The first one was that they let it be because people don't have any other option. They cannot take an airplane from their backyard. So let everything happen how they are happening right now. And the second option was that they can automate this process and increase the staffing or increase the resources and bring this time down to maybe six seconds or six minute or eight minutes from the existing 10 minutes. So what do you think should be the answer?

So I would say two, but I think it's a trick question. Probably one, because I don't think people can perceive between 10 to 8 second 2 second difference.

Okay, so this is why you should never go to who Wants to Be a Millionaire.

That's not correct answer. Should have used your lifeline.

Oh, got it.

So the correct answer is. So the correct answer is neither of them. So they come up with a better option. So they played with the user perception. So what they did is they moved that conveyor belt to the other end of the airport. So now after a person deboads the plane, they walk and it takes them around eight minutes to walk to the conveyor belt. And then in another two minutes they get their back. So they are happy because they are doing something, they are walking along, so they are engaged. So this is where the user perception comes in and this is where it is very important to get this thing. So how I am relating this to our website performances, it is possible that you are downloading all your resources in two seconds. In the next eight seconds you are rendering it and people and the user does not have anything to do. So they are looking at a blank screen or something flashing on the screen that they cannot use. So what you can actually do is you can play with their user perception. So you can display most important and most critical stuff over the screen and then in the background render the other information. So let me take an example of E commerce site because it is very common and people love shopping. So one of the example that can fit in here is like you are going to a product page. The most critical information on a product page is the product image or its description. But there are other information there as well, like recommendation reviews and maybe some Google Ads. So you don't need all the other information, but you need to see the product image so you can prioritize this. You can render them as soon as possible and then in the background you can render the reviews, the Google Ads or other recommendations over there. So this is how you can play with the user perception. But to play with this you need to have a measurement or.

Like quantifiable.

Thing to get this timing.

You know, I think I've seen this before. Is this. So it seems so hard though to really make measure this though. Like how do you know that your UI efforts actually are making a difference? Do you use like heat maps or something to track your users to see how they're, how they're behaving on that page?

Okay, so there are like n numbers of ways that we have captured it in the past and how we can capture it today. So when Web 1.0 was released, there were only images and text. So it was very straightforward. So your Windows onload was your user perceived time. But now with Web 2.0 there are web pages with heavy reliance on JS, CSS, Dynamic HTML and XHR calls. So that onload event does not actually translate to user experience. So we need something which can actually tell us when the hero image or the most important image is displayed. So for that there are many tools in the synthetic monitoring. So few of them that I actually enjoy are webpagetest Wise Flow developed by Yahoo, and then there are Google Lighthouse and all. So what they does, they have a, like a way of getting how much.

Like your viewport has been rendered. So what they does is they take a video or they take multiple screenshots and then have some algorithms going around and they do the image processing and get how much they have changed from the previous frame. So now if the page is 75% rendered, so they say like it is 75% visible at this particular time, and if it is 80% at this particular time. So they get this thing from their great computing power. But the issue with this approach is you cannot use it for the real user. This can be easily done for a synthetic test, but not in the real world because these are very compute intensive. So you need something lightweight, which is accurate, which is standard across browsers. So this is where custom metrics comes in. So one of those custom metrics are user timing APIs. So what it does is they have two main APIs. The first one is called the mark and the second one is the measure. So mark gives you, it stores the time taken since navigation starts. And so it is the time when you first hit the page and then the time it took to actually hit that mark in your JavaScript code. And the second one is measures. So measure, it records the delta between two marks. So if you have two marks in your page, you can easily get the difference between them. So these are the two main APIs which we can actually add it into our code and get the particular time of a rendered image or a particular text in our code.

So is this an API that you create or is that part of like where do you hook into that API?

Yes. So these are W3C specification, so these are standards and we're always there. So I was listening to one of your podcasts on front end, front end performance. So the guest over there, he said that they uses date.now function, which is actually. So these actually does the same part, but they are more precise and they are precise, they are around thousand time more precise than your date dot now because it gives you time in microseconds, not in millisecond. And also they are more reliable. And these are part of your browser. So these are supported by almost every leading browser. And if your browser is not mentioned there, you, you can easily polyfill it using polyfill created by Pat Mina. So yes, it was always there but we always overlooked it.

How do you hook into them though? Like if you have a. Like, say I don't know what tool you use? Say I don't know, say loadrunner. Do you use. Do you import it and then you do performance mark or like how do you. How do you capture it?

Yeah, so how do we capture is. It's the right question. So what happens? You can actually add these markers in your JavaScripts and or you can add them, let's say for example, all your css. After all your css you can add this marker. So this will tell you when these CSS have done blocking the rendering and you can also add them after your synchronous js. So it will tell you after how much time that JS have.

Done rendering. These can be hooked into our web page actually and to retrieve them there are a few performance timeline APIs that we can actually use.

There are basically two main APIs. So first one is Performance get entries by type. The other is performance that get entry by name. Name gives you a particular mark or a measure value that you have put into your JavaScript code and the type performance getentries by type will give you all the markers in your page or all the measures in your page. So this is how you can actually get this.

So dumb question. Is this a developer activity? Then you say you need to add this JavaScript and here are the performance marks and measures you need to add.

Yes. So it can be done both ways. So if you are confident and you know about your web page end to end, you can easily add it. But it is better to sit with a developer or for a developer to add this because they know which image is going to render first and which is going to render and when they are going to render. So yes, you need developer help as well as you need product managers help. Because you need to know which are the critical images because it may happen that you added these markers at a very not a very relevant place.

Cool. So say someone's doing like a traditional load test where they're putting a load on the back end and then you have these markers added as well from your developers and then at the end you have a report. Is there a way to bring that front end information into your backend report so you do some sort of correlation?

Yes, it can easily be done. So what I did for this new test pack for this year is I created a photo gallery app. So it is basically 10 images so you click on one image and it renders on your viewport. So it's a very basic application. So I added these markers all over my webpage, all over my images. So once I click on image it tells me how much time that image is taking to load. So now what I did is I just wrote a 10 liner JavaScript code and I inserted into my web page and what it does is it takes fetches this value using those performance getentries by type or get entries by name and then send it over to my backend system. So in the backend I was having an influx DB which was running on an EC2 instance.

T3 large. So it costed me around 50 to $60 to have around 100 to 150 hits per second. If you are an organization who is low on budget and who is having hits per second around 100 to 150. So you can capture these value for around 50 to $60 per month. And if you are an organization who is having more hits per second, so you can always upgrade your hardware and maybe use clustering and all to get higher hits per second.

So interesting. So would you release code in the wild with these API performance monitors in there and then you collect them over time in production?

Exactly. So this is not my idea or this is not a only a POC kind of stuff. So if you go to famous sites like LinkedIn or maybe Google or Microsoft, their homepages, you will see these markers embedded into them everywhere. So there are, there might be ten hundred or maybe two hundred markers over there. So they are loading this and getting these minute times which is difficult to measure otherwise Very cool.

So this solves an issue. I was speaking to someone else and they were saying they were looking at a website like USA Today and they use like new relic. They use, they use like three different monitoring vendors that are supposed to capture all this data. But this sounds like a lighter weight kind of solution rather than. Because I guess the more JavaScript you add, the more you actually could slow down the website. So how much overhead does this actually add on a website if you do release it in production like that?

Yeah, so the JavaScript that I have embedded is an async JavaScript so it loads in the background, so it is loaded at the bottom of your page. So it doesn't really block anything from rendering. So it will stay there in the background and it will just send this data to your backend server. So consumer or the end user does not realize any lag due to this script.

Is there any Way to report this data saying Google Analytics. I think a lot of people overlook Google Analytics.

Yeah. So there is a option of custom metrics over there. I haven't tried this but I have read about it. So you can actually embed these timings into Google Analytics, send API and send this data to your Google Analytics. So now you have an advantage there because you can correlate this with your conversion rates, your bounce rate and even your revenues.

I assume a lot of people listening to this podcast are kind of new to performance testing or newer engineers. So I'll probably edit this and put it in the front of the interview. But I want to go over the difference between a synthetic monitoring and real user monitoring.

So yeah, so synthetic monitoring, as the name says it is done in a controlled pre production environment so you have a full control on the test that you are going to run. So you can run it in a scheduled interval. This can be done to get early detection of your performance issues. Like you can integrate this in your CI CD cycles and easily get the results. And this can also help in benchmarking your competitors or monitoring your third party services like your apps, how they are performing. But on the other hand real user monitoring is like saying you are sent into a jungle without any protection so you don't know what to expect next. So it's a real chaotic environment. So you gather real data from different devices, browsers and also geolocations and then you can correlate these data with your key business performance indicators, user engagement stats and then you have with both of these you get 360 degree view of the application. So both of them are kind of important to have better performance of your front end application.

Are there any downside to using user.

Timing APIs so you can always load 10 to 12 or maybe more. I haven't seen anything slowing down in my poc so I'm not aware if they are going to block down your application. But I have read that adding 10 to 15 won't hurt.

And how do you know what to measure? I guess, I guess I mean that's obvious but sometimes how do you know where to wrap these timings around?

So again going to my E Commerce example. So if it's a product page, you know the most important thing is image. So you wrap this markers around your image so once it is downloaded and when it is rendered you and then you add the ending marker over there and if it's a description so you add it over there in the starting and in the end so you know when the description is loaded, then you get a timeline over there before a particular page which is coming first and which is coming second for a particular real user, not your synthetic user. And then you can have this data and manipulate this data to come to a conclusion like what you need to put forward first and then the second so you can rearrange your priorities using these steps.

Very cool. So in your day to day job, how much is user perceived performance? Actually part of your day to day job but you focus mostly on the back end type of storage performance activities.

Yeah, so in my day to day activity it is not much of a part because we have a back end solution but we do have a portal. So.

It depends on many factors. So one of the factors that I will like to quote over here is.

Your user performance depends on your geolocations as well and.

On the hardware specification of your device. So let's say you developed an application for US customers so you have your storage over there for us. Let's say we have a storage in AWS cloud in US East North Virginia region. And now if I want to upload something from US east region it will be very fast. But now let's say I want some customer from India where I am here in the Mumbai region and I don't deploy any S3 buckets in Mumbai region. So what I will do is I will try to upload this data to my bucket in North Virginia. So there will going to be lag around 100 to 200 millisecond and this can impact your performance by maybe four or five times. So all this information you can gather if you have these timings over there. So you may know like it took more time for a particular user sitting in mobile location to upload his data. But why this time is not like why it is increased time is not seen for a person sitting in North Virginia. So you know like there is some issue with the geolocation. So you try to put a S3 bucket over Mumbai locations so that the person sitting in Mumbai region can easily upload it and faster as well.

Are there any other common things you've noticed like in capacity planning? I don't hear as much about capacity planning as I used to because I think people just assume they can just scale in infinity in the cloud. But I'm sure there's a cost associated with that. So is your capacity planning mostly like hey we need to get this down because even if you can scale your cost is going to increase and be outrageous.

So being a SaaS based AWS based company. The cost is the most important factor that plays a real good part in your revenue. So if your costs are low, your revenue will eventually grow. But if you are having higher cost, your revenues will be less and, and.

Maybe you won't profit much out of it. You have to increase the cost of your product and all. And that's a turn off for most of the customers. So my capacity planning, by capacity planning I mean we run these backups in our test environment and see how much CPU they are actually consuming and where these CPUs are actually being consumed in our code. And then we go to that code, optimize that code and try to bring that.

Like the CPU cores and your memory consumption lower so that we can.

Decrease this cost in our production as well. And this is how the capacity planning works. So if you require, let's say 2x C5, 2x large machine in your cloud, but you have provision C5, 4x large machine, so your costs are going to be more. So this is where we come in. So we tell them no, this work can be done in 2x and we don't need a 4x large machine. So this is where we come in and this is our duty to report.

So going back to the performance API, I really love this approach just because I'm an old guy. And the way we used to do this before is we'd run a winrunner script to capture, try to capture front end performance, but it was never accurate. This seems more accurate than that. So seems like a solution that anyone can implement.

Exactly. It is a solution anyone can implement. And most importantly, what you can have a benefit of this is when your functional testing is going on, you can actually get these timing in your database and you can measure it. So this can help you in one user performance as well. I have a very great analogy. It's from the movie Prestige. I think you must have heard about and watched that movie. So it's a movie about two magicians directed by Christopher Nolan. And I'm not going into the story, but there was a very popular dialogue in the movie. So it says like every magic trick has three part or three acts. The first one is called the pledge. This is the part where the magician shows you something ordinary like a pen or a book or maybe a pigeon. So for us this the pledge is to create a website and give it to our customers. And the second act of the magic is called the turn where the magician takes something ordinary and turns it into something extraordinary. What it implies to us is we take this website and turn it into monitoring tools. So we add these markers and measures all over our website and makes it into an extraordinary website. But the dialogue continues. But the audience won't clap just now because disappearing something isn't enough. You have to bring it back. That is why every magic trick consists of three acts. The hardest part, the part we call the prestige. So for us, the prestige would be to gather this data and send it to our backend system or our database and then get something out of this data, some relevant information that we can actually use and improve our user experience. So yeah, so that's the analogy that I really like and I would like.

To tell this okay Anka, before we go, is there one piece of actual advice you can give someone to help them with their performance testing efforts? And what's the best way to find or contact you?

Okay, so one advice I would give is like there is nothing like too much data. So gather your real user data, use it, analyze it. Maybe you can find some issues that you never knew existed. It is important to monitor your real user behavior to have a better user experience and better understanding of your end users. And you can contact me. I'm very active on LinkedIn so you can find me over there. And for my email address it is uncle jan9292mail.com send me a hi. Send me a text question or query that you may have. I'm happy to help.

Thank you AKA for your performance testing awesomeness. Please have net of value we covered in this episode. Hand on over to test guild.com P28. And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Smart Bears Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.