Mobile Performance Testing with Sofía Palamarchuk

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Sophia all about mobile performance testing. If you don't know, Sophia is the director and board member of Abstracta and co founder of I think a new company. We'll talk about it, but CEO of Aptum. After beginning her career as a performance engineer in 2015, she has led Abstracta's expansion to the United States, heading up the business development and working with companies like Shutterfly, QA Technologies and Singularity University. After seeing the challenges that mobile development teams face in 2019, she embarked on a mission to transform the way global mobile teams create quality apps. When she's not busy, she's leading a startup and you can find her kite surfing or supporting projects that promote more women in technology. Really excited to have her on the show, especially as we get into the holiday season as people start using their mobile phones for e commerce, performance always tends to be an issue. So really excited to dive into this issue with her today. You don't want to miss it. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Sophia, welcome to the Guild.

Thank you, Joe. I'm pretty excited to be here. Thanks for the invitation.

Very cool. So Sophia, before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

It was pretty good. I do like kite surfing in the Bay Area. In case we have any listeners, they want to join us. And I'm also vegan, so I love cooking and I try to spend as much time as I as I can with my friends and my family. But you know, running a startup means that you have to put a lot of effort in the first years and that's the state we are. But yeah, I'm really happy to be here and sharing my knowledge and this topic that I'm very passionate about, which is mobile performance.

Awesome. Yeah. I've had almost 400 interviews. I don't think anyone ever mentioned being into kite surfing. So is that literally getting a kite and you just. Is that what it is? I don't even know what it is.

Oh yeah, that's a great question. I'm happy I'm the first one, actually. I think those four became really popular in the last, let's say five to four or five years ago. And basically, yeah, you have a kite that you need to push yourself in the water. So instead of only like surfing, you just need a board. You have both a board and a kite, meaning that you actually can go faster. You use the wind to push it. So really we go out to the water when it's really windy and then you have different boards but very exciting sports. A lot of women that are also getting to know it more. And yeah, that's something that I also enjoy doing on my free time.

Very cool. Sounds very scary to me, but sounds fun for sure. Cool. So seviy, you know, I like to define things before we dive into it. A lot of times people hear like load testing and performance testing, but they don't necessarily maybe associate it with mobile performance testing. So how would you explain what mobile performance testing is compared to, say, performance testing a web app? Are they the same thing?

Not really. But yeah. The truth is when you talk about performance testing, typically everyone thinks about load testing. Actually, you can sometimes use the same word to talk about load testing, but actually it's more than that. When we talk about performance testing in general, we can talk about performance of our backends, of our client apps, and then you can dig deeper into if your app is a web app or mobile app, then what are the differences between these types of systems and the infrastructure below? And what are the type of performance tests that you can run? So it's more than just one area. But typically and historically, I think load testing has got most of the attention and for good reasons because there were a lot of performance issues in that part, in a way of our systems. But in the last years, from my own experience being a performance engineer myself, a practitioner and a computer engineer, I actually moved towards more the front end of apps and specifically mobile, where, you know, a lot of new people are jumping into the Internet using their mobile phones, even before desktop. So we saw that there was a huge need there for tools and ways to helping mobile teams test their apps and being more efficient in that. And we started with mobile performance as Thinking about the app on the device and testing on the client side. So we don't when we refer to mobile performance or you can actually think about it mobile app performance. Because people when they search for that is how can improve the user experience and the client side performance. So it's a bit different from what we are most of us, what we are used to, which is load testing and stressing our backends or web testing, which means how to deliver your content faster through the Internet through typically a client server architecture. So mobile has a lot more challenges and I think that's why it is pretty interesting and required to have specific tools and methodologies to test this. We can get deeper into that later. But basically there's difference between client and server side. And if your client side is mobile native app and it's not a web app, there are three things that you have to take into consideration were when testing performance.

Great. So we're definitely going to dive in. So you just mentioned something about native apps compared to a browser app. So when we're talking about mobile performance testing, are we just talking about native apps to like iPhone or Android? Are we talking about even web apps that scale when they're on a browser, on a mobile device?

Both when we talk about mobile, is any content that you will deliver mobile and can have web apps, meaning I'm accessing a website that's mobile friendly, hopefully then you have hybrid apps which are actually apps that contain web views inside the native app. So it's kind of a mix in a way of both worlds. And then you have the pure native apps that are the Android and the iOS today in the market. So all of them, there are different things that you can test when it comes to how they perform in mobile. My experience and lately even my own startup, we've been focusing first on the native apps and testing the performance of them because we saw that there was a huge need for testers to access information about how their native apps are behaving. And typically this is information that is available in profiling tools or in other parts of the development cycle that not every tester has access to or knowledge. So trying in a way to democratize this information is what we're doing right now. And then for web, the truth is that there are already a lot of tools available to know if your website is mobile friendly or what optimizations you can do to deliver the content faster to your mobile users. So we didn't see that there was a huge issue today and we actually recommend when I give some talks about this, tools that are built by Google and other companies that have already experience in web performance testing and I will just guide you there. But for native apps, more challenges that come into place and that's where we put our focus right now.

Nice. So you did mention also a lot of the times you are focusing more on the user front end experience, not necessarily a traditional load on the system. So the lack of tooling or what people may be not familiar with tooling, what kind of tooling should be in place when you're doing mobile type performance testing?

Yeah, that's a great question. And typically, even before getting to the tooling, the main question that I think as assessors we should ask ourselves like what are we trying to find or what are we trying to solve here? Again, we have a lot of companies that come to us asking can we virtualize hundreds of users for our mobile app with Apptium, which is one of our first tools. And truly I go back to them and say there are so many tools in the market today for load testing, which is what you need and you can simulate traffic as if it is in your mobile app. And you don't really need a specific tool for that today by the way. What we're trying to do is simulate a real world scenario when you have hundreds or thousands of users accessing your app. You need a load test and there are specific tools for that. Open source, you know, everyone knows about JMeter and Gatling and many others are out there. So that specific, like if that's your need, then you should use those tools. You can actually complement them with front end performance testing tools. And answering your question, what's available today? So let's say I have, I'm testing my app and it's slow. That's what I see as a tester, I'm trying to load or moving from one screen to the other and the experience is really slow. The next question is, okay, I need to find out where and why is it getting slow. And for that you need to understand that there are three main factors that actually will affect that experience of the user. One is of course the server response times, backend database, every service that we're using and how long those requests take to get back to the actual app. Then is the network, which again, thinking about mobile, we are usually using our phones outside connecting to LTE 3G or Wi Fi. And also that affects of course the user experience and how fast certain data can be load. And then we also have the front end, which is the app itself and how it uses the resources from the device where it's installed. So if you are thinking about a smartphone, it is a computer, but with less resources available than any PC or personal computer today. So we can actually think about these three main factors first and say, okay, if I'm having a slow experience while I'm using the app in my phone, then I need to start using tools to understand where the problem might be, what's taking more time. And that's when I think not only the back end, but also testing the front end can help us that tools available. As I mentioned before, profiling tools that help you in a way debug how your app is using the resources from the device where it's installed. And by resources, I mean CPU usage, memory usage on the device, power usage, everything that you can think about that is actually shared with other apps. So we have to be conscious about that, that we are not going to release an app that is high consuming CPU because we're probably going to drain the battery from our user's phone and they're going to be really ma. So these are things that typical developers will use. For Android and iOS you have both native profiling tools provided by Android Studio in the case of Android apps, and in the case of iOS Inside Xcode, which is the IDE used to develop iOS apps, you have instruments, which is kind of a module provided by Apple to access this information. So this is typically what you will use. And of course you can complement this with proxies when you want to capture traffic and understand how that factor impacts the user experience. And it's a lot of investigation and research around what could be causing this. But typically you will be thinking about both profiling your apps and as well looking at the network to understand at least have the main root cause of where this loneliness might be caused. And then you go deeper into one specific area.

So there's a lot there. I think what makes it even more complicated to me is environment, like you said, the network. How do you emulate a network? How do you emulate a certain OS on a certain device? If you don't have a lab, what do you need in place, environment wise, in order to set up a proper mobile performance testing environment?

Exactly. Yeah. And that's also something that is tricky and I will say today in general mobile testing as a practice, if we don't even think that our performance has evolved a lot in the past years, the automation frameworks are available and it's actually getting better. Think with the latest releases from Appium, which is the open Source framework to automate tests on mobile. And that evolution came also with the fact that you can today rent devices in a cloud provider where you can access those devices. And these are real devices located somewhere in the world. And you can in a way configure real or not, but you can configure the network where you want that device to be connected. And usually I will suggest, of course, depending on your business needs and what specifically you want to test to keep the lows, the causes as low as possible. And you have ways to simulate different networks from the devices you're connecting to by looking at the bandwidth and the latency of those networks, which are two factors of the network, but is the ones that when they change, they might affect how fast the data is being transferred. So you can simulate that as you're doing a load test. You simulate virtual users. You can simulate that on real devices, both if you connect to some services in the cloud that today they provide that they provide you with devices and the ability to simulate network. And also you can do that even locally if you have your own phone and you can plug it in to your computer with a proxy can control and change these factors of the network. So you can simulate connecting to 3G or 4G or WiFi and then see how that also affects the performance of your app.

Have you ever seen anyone implement crowd testing using performance testing so you can monitor what real users are doing on their devices as kind of a trying to emulate what would happen once it is officially released?

It's funny that he asked, because they're currently actually working on a partnership with the crowd testing platform so they can use our solution and they actually saw the same idea. They have thousands of testers in different locations in the world testing locally with their real networks and devices, but they lack the tooling to capture all this data automatically about how the app is behaving on the performance. So this is something that we're just now looking into as a way to not thinking about automating the test, which you can also do. But also what if an actual tester that is located in Asia or another tester there is in the Americas, if they can actually, if they're already testing in the local network, but they might not be very savvy with even like profiling tools, but they are testing a lot. Let's capture those that data and put that as part of the test that they're doing today, which might be functional test and have that view on how the user experience is and how the app is Behaving on the devices they are testing.

So going back to, like you said, three key components to mobile app performance you mentioned backend, the network and the client. If someone has a web app as well, and they also have an app when they're doing a traditional say performance test, should they also incorporate mobile? Because can mobile have an impact on the backend or can web have an impact on the backend that you may not see if you're just testing a mobile device in isolation for performance issues?

Yeah, exactly. I will actually recommend that for web as well. Same happens if you're testing a web app. You can actually, while you're doing your load test, you can be running Selenium script. You don't do it manually, but you can have something the client side testing the browser and you can be measuring what are the response times for the user during the load test. And there are things that will impact both. And I usually say this is kind of advanced. First start with your load test, focus on the server and what you can optimize there. Then you can look at the front end and then you can run both in parallel to see the end to end experience. So the same for web applies to mobile. You might want to be running a load test, simulating mobile users and then having a way to run a test on the front end and capture everything that's happening there and comparing results as well. There are some things that can be correlated, other things that are completely separate, but I think there is a use case there as well. Do an end to end performance test covering both your back end and the front end.

So Safir, who normally does the performance testing of mobile devices, it sounds like if it's native, it's already in the the developer's IDE for things like CPU and memory consumption. Are they responsible or do you still need a tester or a performance engineer involved in that case?

Yeah, our experience is mostly this is something that performance engineers are getting into more and more. So have companies that have centralized team of performance engineers and they might even have a specific team focus on front end performance, which are the ones that test both the websites and they are getting into mobile and they are also accessing profiling tools that developer use today. That's the ones we see that have an actual structure for performance testing then the ones that are developers. It really depends on how much time basically developer has and how important it is for them. Of course, if they have any issue that they need to solve. But typically the most successful teams that we've worked with they have dedicated person that is usually from a performance engineering team or a test engineer that is getting to performance and those are the ones who start doing this test. What we are seeing is there is a technical barrier for any tester, even manual testers that are testing their apps with their phones to access this data. If we only rely on the profiling tools that involves installing the ide, having access to the code of the app, and many things that usually testers don't always have. And that's where we find that if we can provide a way that without the need to access the source code of the app or install heavy tools or analyze a lot of data and in an easy way to show this information for the tester and developer also as well. Because when your reporting bug or something from the user experience perspective, you want to give as much information as possible to developers as well. So I think that's where we found that there was at least one specific need that we can solve with our solution that basically captures everything that's happening during a manual or an automated test on the mobile side performance, and then generate reports that you can share with anyone. So we are trying to democratize this so it's not only those teams that have the dedicated performance engineer looking to this, but basically our vision is that anyone developing mobile app should be able to access this data as fast as they can.

That's great segue. That was one of my next questions was what does your company Optum focus on? So to me, I haven't seen it yet, I haven't used it yet. Sounds like it's a, not a dashboard, but a place where everyone can go to find out the status of the performance of their applications. I guess data collection or what does it do? I guess before I make guesses about it.

That's a very good guess. Yeah. We have a lot of clients that call like this is one place for our client side performance. And these are typically mobile teams that are working both the developers and testers together. And our solution, in a way what it does is it monitors the app while you're running the test and it works both with manual testing or automated tests. So in our case we have users that are functional testers. This crowdsource testing platform that we talked before, they don't have any automation, they don't have any background performance, they're just running their test cases and they have a tool that we provide them, which is a desktop app that they should use to monitor and record everything that happens in the test. And they also have Evidence not only performance, but they have a video of everything they did, they have logs, we capture crashes. We do more than just performance. But the way it's used by this functional testers is very easy and friendly UI so they don't have to dig deeper, be delayed by installing a lot of dependencies. So we try to make it super smooth for them. And that's the first product that we launch. It's free actually. So we want everyone to use it and just get the voice out there that performance something that you can start testing early on. You don't need to get very sophisticated to gather this data. And actually any tester in the team could use it. And that's one of the solutions that we have. The other one is mostly used by test engineers or QA automation team that they already have a lot of tests automated for functional testing. They're probably running in the cloud in CI CD pipeline, and that's where we have a solution that they can add to their current test. And instead of only getting the results from the functional test, they also get all the insights and performance data that Apptium collects. So in a way we are adding a layer of data to all the regression tests that they're running so they can also capture and in time see if there's any change in performance and might be due to a new change in the App Pro version that was released. So we have both options today. One is more for local kind of manual testing and the other one runs in the cloud with real devices. And you can automate most of the process and just get as an output all the reports and then see the trends on the performance of your app. Which is something that it's very interesting as well, seeing how we might be affecting also certain aspects of our app while we add more features as well. And of course as any application or website as well, if you don't test it often, you don't know until someone complains. So that's exactly what we're trying to do with automating the performance testing testing on different devices because that also affects the performance because they have different hardware and resources are not the same. So this is something that we are looking into improving that part as well, kind of continuous performance testing.

I love this idea. I used to work for a very large enterprise and it was very hard to know what was going on with other teams and a lot of people were like, I'm just responsible for this piece. But it sounds like this type of solution helps give insight and visibility to everyone about performance. So it could maybe help the culture to be more performant oriented.

I guess that's exactly what we're trying to do. And we have a lot of clients that they love that as well. Someone in the team is trying to push for more of this type of test and what we realize is there is a lot of information that is not being captured both on the manual testing team, the automation team, if there's any tests that's being automated. And we provide kind of like one platform when with two different solutions that we have today, we can see everything in a centralized place. So QA manager or even developer can access all this data that's being collected by Apptium in each area of the testing process. And that's where we want to go. Being able to do flat testing both with automation and even without automation. As soon as you have your first version of your app, you're not going to start automating tests because it's probably going to change a lot. But you might look, you might be interested in knowing the user experience and the performance of your app and that's where I think we can help early on. And then if you start getting more tests automated and increasing your coverage, I think also we can provide you feedback on how your performance is evolving with time.

Great. So I did first hear about Sachini Neotus on it was on the user performance type of angle. So I'm just curious to know how to improve user experience. Does your tool or in general are there certain areas people should be looking at to help with user experience?

Well, yeah, there are different areas when we think about testing for user experience and how more performance is, I would say is one of the main factors that affects the user experience. So we can identify there's something not working properly. Let's say we're using our app, it freezes at some point or as I mentioned before, it might be very slow. And there are different areas where we can look at. One of the things that we started doing besides what I mentioned, that is resource usage in general is all the drawing times and rendering times. This is very important for games. So any game that you have, game apps that you have installed, they need certain rendering levels to provide that experience with the user. And other apps are also turning into measuring how much time it takes for them to render screens while the frames per second on their apps and they're putting targets as KPIs in performance. And I think this is an area that not always everyone is aware of and it's something that you can incorporate things to see that really affect the user experience. High rendering times might provoke disrupting the animation of your app. So things like glitches or screen freezes, as I mentioned, can be consequence of this. So in a way we are also going bottoms up. We are showing you the raw data below and then usually you can correlate that to this is where my screen freezed and then I can go and find the root cause. Well, at least know that there was probably a rendering issue and then go to the root cause with the developer. But kind of tracking this information can help you directly identify issues that a user experiences. Of course, anything related to crashes, exceptions that happen in the app, things that are caught, things that are not caught, things that are known by the team or not, and everything that happens on the front end should be also something that we need to be aware of. And then the last but not least, which is actually an area that we're really interested in at apptium, in going further and exploring and researching ways that we can help teams to capture this data is response times, but from the user experience. Meaning I click on a login in an app that I'm using and then I want to measure exactly how much time it took for the app to be available on the next screen and how long it took for the screen to be active for the user to interact. And this is not as easy as web, because web, you have the DOM and there's ways you can know when the page finished loading and when the content appeared to the user. And pretty sure you're familiar with a lot of tools that show you specifically when the content is available and things that you can optimize. For mobile native, it's a bit trickier. And of course it's completely different for iOS and Android because there are two different platforms and even how the apps are built. So Android have activities and you can measure activity times, but it's really hard sometimes, specifically, if you want to measure this in an automated fashion, let's say a logging time from a user perspective, you have to either try to do with the scripts or as we are trying to research more now, finding ways to provide our testers and our users with measuring specific transactions that they want and then comparing them. So let's say I don't want my logging time to be higher than three seconds, which is typically kind of what users will allow the most. So you want to be measuring that in every new iteration and every new release of your app in an easy way. You don't want to do it manually. And then with the test Automation that happens today. Usually there's an overhead added by the tools that are running the automation. So it's not as true to the real user experience as it should be. So everything related to response times, measuring from point A to point B in your app, how much time it takes, and then replicating that on different devices and then seeing over the time how that evolves is something that a lot of companies are looking for solutions and I think there might be a way to help there and provide something new that it's not available today. And the other thing that is also specific metrics for the user experience is how much time it takes for the app to start, which is the app startup time. It means when you click on the app and you open it, you can measure that the first time that it opens and also when it goes to the back and it stays open in the back of the back stack in Andre and then you push it again to the front. And there are specific technicalities and differences between those two, but you want to measure that as well. That is actually one of the first steps that the user has to go through opening the app. So if it takes too much time to launch, they might just not even use it. So it is something that impacts directly the user and you can start measuring this right away.

So, Sophia, you speak with a lot of clients and does anyone have asked you like, what's the difference or should they even have a mobile app compared to a web app? Because a lot of times I've been hearing recently that mobile apps are dead, but I'm not sure that's the case. But do you have thoughts?

Yeah, a lot of users and people I talk to in the community, they usually ask me, should I have a mobile app or web app, what's better? I don't want to get into detail. I will say it's really different. It really depends on your business and what's the goal? I'll say usually the mobile apps provide a way to actually deliver faster content and a better experience the native mobile apps to the user. And also today there are new technologies coming out that I think anyone can try. When you're deciding what you're going to be developing, if you need a website that you want to access a lot of, I know a huge amount of people anywhere in the world, you can deliver content faster and it has its benefits. But if you really want to look into more personalization, having more intuitive interface, being able to work offline as mobile apps can do, I think that's a better option. And I will think about performance in early stages when deciding what to develop and knowing that for each even if it's a website, you need to start earlier testing for mobile performance, meaning what's the performance of your website or app on the mobile device and there are tools for this. So this is something that it's really available today and from our position we're trying to help today the native app developers and testers to get this information faster.

Okay Sophia, before we go, is there one piece of actionable advice you can give to someone to help them with their mobile performance testing efforts? And what's the best way to find contact you or learn more about Aptum?

So one piece of advice is I think I already said it, but start as soon as you have the first version of your app. You don't need to wait to go live and have users complaining or reporting bugs or issues. If you're developing iOS I think you have an easier life because there's way less models of iOS and also the version of the operating system in mobile changes a lot how it performs. So say start as soon as possible. There are profiling tools I mentioned if you're a developer, Up Team is a tool that is built for testers so you can get started for free. Just go to our website, download it, get familiarize yourself with everything that we talked today about metrics and what things to look for and then in the sense of how to contact me you can go right to Twitter so Palamarchuk, which is my surname and then also through email it's sophia my name ptim.com and the easiest, easiest way if you're already on our website is actually just go to the chat on the right bottom corner and I'm actually going to be there as well.

Thank you Sophia for your performance testing awesomeness. Links to update and value we covered in this episode. Head on over to testguild.com P52 and while you're there make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing solutions, Load UI Ninja and Load UI Pro. And if you like this episode, why not rate us in itunes ratings, really help in the rankings and help spread performance testing to the masses. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and site reliability podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.