Performance Testing FOMO with Nicole van der Hoeven

Welcome to the Test Guild Performance and Site Reliability Podcast, where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. I'm really excited for this episode today because we'll be speaking with Nicole all about performance testing and how fear of missing out affects performance outcomes, as well as probably other performance testing topics. If you don't know. Nicole is a performance testing advocate at Flood IO, where she helps make the Internet faster. She has over 8 years of experience as a consultant in the Netherlands and Australia. She came to the conclusion, though, that the hardest part about web performance is learning how to deal with the human element. If you haven't seen any of Nicole's videos, you should go to Flood IO. She does a really cool style and really professional. Really love the way she does tutorials, so you definitely need to check her out and you don't want to miss this episode. So check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey, Nicole. Welcome to the Guild.

Hi, Joe. Thank you for having me.

Great to have you. As I mentioned, I really love the way you teach and you're really technical, but you also are able to talk about what you're doing as well in a way that people can understand. So, really excited to have you on the show. Before we get into it, though, is anything I missed in your bio that you want the Guild to know more about?

No, I think you did a great job.

Awesome. All right, so let's just dive in. I think when people hear the title of this episode, Fear of Missing out, they're like, what does that have to do with performance testing? So I guess for the folks that don't know what fear of Missing out is, what is FOMO a Fear of Missing Out?

Well, I think that it's a bit of Internet slang that's developed recently which talks about a specific phenomenon where because of social media and just how easy it is to get information these days over the Internet that people are more keenly aware of things that they feel they should be capitalizing on. And I guess that's just been defined as fear of missing out. So, as an example, social media shows us all of these really rosy pictures of people's lives. But of course, that's heavily filtered and edited. But the mistake here, this is where FOMO comes in, is that other people can see that and imagine that their lives are a lot more mundane than everybody else's, which isn't really true. I think this has always been around. It can be like trying to keep up with the Joneses. But I think it's been especially prevalent recently.

Absolutely. And I don't know if this is another kind of part of it is I know a lot of companies, a lot of people that work for other companies look at other companies like, say, in the Silicon Valley area for, like, Facebook or Twitter. And the way they develop software, do things, they always think, oh, it's always they must do it correctly or they must do it the best. So I should mimic how they do it and not. Rather than focusing on how they develop software and what their needs are. And I think sometimes that might lead into issues as well. Do you see that often with different clients trying to emulate what other people are doing that may not be right for their particular company or needs?

Yeah, absolutely. I think it can go both ways. One other, like you said, companies can look at other companies and try to keep up with them. But I think also from the perspective of the customers of the companies, I think that it's a lot easier now because of the Internet, to just change to another site that sells the same product. If they feel they're missing out on something, it's just a lot easier to do that now.

So one thing people might feel like they're missing out on is like trading apps. And I think this is actually based a lot on a blog post you have on Flood IO where I'll have a link to in the show notes on fair missing out and performance testing why Robinhood went down. So can you talk a little bit about the Robinhood app and how maybe that ties into this fear of missing out concept?

Yeah. So I'm particularly interested in Robinhood because I'm pretty interested in investing personally as well. And Robinhood is an app that I've had my eye on for a while, although unfortunately it hasn't been available in Australia. I think the stock market, it just shows a classic case of fear of missing out, because every time the Stock market is on a rise or when it falls, that doesn't necessarily correlate to changes in the underlying companies or the stocks that people are buying and selling. A lot of it is led by fear. Fear that a rumor is going to be true and the company is going to. The stocks of the company is going to take a nosedive. And it's all based on speculation, really. And it's really exacerbated by the presence of social media where everyone can chime in and speculate. And I think Robinhood really shows that because towards the beginning of COVID 19 and this current pandemic, they actually had a pretty significant outage cost, mainly because of the huge numbers of people. There was droves of people who were logging on to Robinhood to both buy and sell stock. And it didn't help that the stock market, I think, reached a world record or historical record high and then it was followed by a historical low within the span of a week.

Absolutely. And obviously, usually people, when they're doing performance testing, plan for maybe a normal case scenario what they normally see. And this obviously caused probably spikes in the traffic they weren't anticipating. So I guess is that how fear of missing out actually can impact how an application performance can happen in the wild?

Yes, absolutely. And I think that it's not just in the stock market that we can see this. I'm actually working on a blog post now about the Dutch Corona line. So I live in the Netherlands, in Maastricht, close to the border between Belgium and the Netherlands. And in the Netherlands, on June 1st, they actually released a new telephone hotline for corona coronavirus testing. So before that, we couldn't get coronavirus tests unless, you know, you were actually sick in the hospital and really needing it. But I think it was really interesting because on the first day that this hotline opened, they got 323,000 calls. So of course they. Their systems went down. Everything was in chaos. There were reports on Twitter of people waiting up to six hours. And this has real health effects on people. Performance isn't always just a matter of, you know, money that's lost or downtime on an E Commerce application. It can have real effects on people's health. There were people who were potentially sick, who didn't get tested, who didn't then get treated because this coronavirus testing hotline was not perhaps adequately tested to the level of the load that it actually received.

Nice. And like you said, it just doesn't affect financial applications and healthcare applications could be, even if people are listening like Well, I don't have a mission critical app. I think I read something about Epic Games had an outage because they had a freebie on the store and everyone was trying to rush to get the freebie before it went away and the service crashed on everything.

Another example, I guess, yeah, definitely. In gaming we see this a lot because there's always discounts, you know, when you patch. I know that I personally play Pokemon Go. I got started with it because of my nephews. And I know that they have had issues in the past where they've had real events in certain locations like a Pokemon Go fest. And the sheer number of people who logged in to play it was too much for their servers and also the mobile networks. And in that particular city, you know.

We mentioned a few examples of fear of missing out causing performance issues. How would someone know that their application does require fear of missing out? Do you go by risk based or do you have certain criteria you use to say, okay, if you're, if you do any of these things that you may want to look into this type of testing?

Yeah, I think it should be risk based. You should assess this risk just like anything else. I think depending on your application, the risk to FOMO may be higher or lower recently. Well, in the past we at Tricentis Flood have also helped companies load, test or prepare their systems for the Melbourne Cup. And I'm not sure if you know, but the Melbourne cup is a multimillion dollar horse race that's held in Melbourne, Australia every year. And it brings billions of dollars in revenue not just to bookies and other gambling sites, but also just the state of Victoria because a lot of people come from overseas to this day to watch like a two minute race. And I think that there are some applications like that one. Let's take the case of a gambling website where they're actually selling bets for a particular horse or combination of horses for the Melbourne Cup. That's a clear example of a situation where FOMO can really affect your performance outcomes. It's a very public application and it's time sensitive. So I know from testing, load testing the Melbourne cup in previous companies I've worked for before as well, that something close to 80% of their entire year's profit is gained in the last 10 minutes before the race actually starts. So people wait and wait and wait for all the odds to come in and all the information and then just before they go en masse to this, to the site to place a bet because they're just afraid of missing out on placing it.

So that's a great example. I think you also pulled out some other examples as well. Say if you have a customer base that's made of younger tech savvy folks, I guess could that cause an issue? Because I guess you may want to do fair missing out type of testing if for some reason you have a very vocal group that's on social media that could give you negative publicity, I.

Suppose, or I guess yes, in Robinhood's case that was their downfall because majority of their clientele is younger people who are on Twitt and Facebook and other social networks. And because it is a mobile app, Robinhood is predominantly used on a mobile they unfortunately had a lot of their customers going and tweeting and posting about their failures as a company. So it's not just the actual outage or performance issues that we need to think about as performance testers. We also have to weigh the risk of the company losing face and the decrease in reputation against the extra costs of testing for fomo.

So once again, some other parts that you call out also in this post, which I never thought about was recent legal action. I used to work for a large healthcare company and we would get audited by the FDA and sometimes they would get slapped or get a fine or they wouldn't be able to sell something. So how would recent legal action impact how people would test then in that case?

So I think that for any company, if you already have legal actions that have been filed against you in the past, I think that that increases your risk profile because the company's reputation has already been damaged in some way and any other negative publicity is just going to make that worse. I think it's a publicist's nightmare to have that happen. And I think it's so interesting because that's not something a lot of performance testers really take into consideration.

Great point. And also I think the problem with fair missing out or this type of thing is, you know, a lot of times people struggle with how they're going to test things that they don't necessarily know what's going to happen once they're in production. Kind of like Black Swan events. So how does someone go about maybe planning a fair missing out type of scenario for their performance testing.

So I think that workload modeling is an easy place to start. I think that you should look at the most common transactions for your application and go from there. For example, in an E Commerce application, maybe logging in is not as common as perhaps going to the details page of an item. So once you have a list of these common transactions, then you can start thinking about what kind of user level you need to prepare for. And while it's true that there are a lot of these black swan type events that you can't accurately predict, I think that doesn't mean that we shouldn't try to predict it. There are certainly other statistics that are available that we can look back on. So, for instance, for Robinhood, maybe they could not have predicted exactly when the stock market would crash and rise, or exactly how many users would want to rush to their platform to buy or sell. But they do have an idea of how many users they have, or they must have some sort of projection of how many users they will have in the next year or so. Those are maybe just estimates, but they're certainly better than not testing for it at all. So in the case of the Dutch coronavirus testing hotline, maybe the government couldn't have predicted that there would be 323,000 people calling on the first day. But the statistics are available for the number of people who are infected with COVID 19 in the Netherlands. In fact, just less than a month before that, the Dutch minister for the Department of Health had said that he estimated that there were 700,000 people in the Netherlands affected with it. So that's a number to go on. That's something that's public knowledge, and it's also public knowledge. And what also is public knowledge is the percentage of people who are infected with COVID 19 who are showing symptoms. So you can already apply that and come up with some sort of educated guess as to how many users your application can expect. And then I recommend starting with that. So using that estimate as just a starting point and then building on it so you can run, assume that that's your peak load test and make sure that your application can withstand that.

Does it matter how an application is being accessed for a performance test? So say they have a number. Like maybe some applications, for fear of missing out, are more prone to people using their cell phone or their mobile app to interact with the application. Does it matter what kind of traffic you should be testing for fear of missing out? Or is it just as long as you test the amount of users hitting your website, it doesn't necessarily matter how they're coming to the website. I don't know if that makes sense.

Yeah, that makes sense. And I think it does definitely matter. And it really depends on your application. In Robinhood's case, mobile networks and any latencies involved in that certainly played a part. And I think most Open source and commercial tools these days will allow you to simulate bandwidth that way so that you can make your tests a little bit more realistic. But perhaps a government site is not necessarily going to have a mobile app, so maybe the effect of mobile networks is not going to be as pronounced in those cases.

Great. You talked about a little bit how people could stop modeling the number of workloads during a peak time. They have numbers usually ahead of time or based on historically what happened in the past. So once they have maybe a goal of the number of peak users, how would you test up your scenarios? I know. What kind of test would you use and how would you go about the next step of performing or planning out your fear of missing out testing?

So I would start with those numbers that you got from your estimates as like the peak load number. You can run the usual battery of tests for that to make sure that your application can withstand that. And then you can also get into stress testing. So for example, you could run a test that starts with that number and then maybe increases and adds 1,000 users or 100 users, whatever makes sense for your application, and runs that for 30 minutes and then increases again, just so you can see that breaking point when your application cannot handle any more. So if you know that, you might be able to put measures in place to stop that from happening, to stop those performance bottlenecks from happening at those points. And then you can also, sometimes depending on the application, SOAP testing can be really important as well. If there's something that needs to be processed, perhaps the application doesn't fail at the time that the request to process them was sent, but perhaps they fail later on. So it's also important to run soak tests for a longer amount of time. In the case of the Dutch Corona testing line, it was supposed to run for 12 hours, but then the processing of appointments made by callers for tests was supposed to happen overnight. And if there are issues with that, those are things that you would find out with a SOAP test.

Nice. So those are definitely two awesome ways of testing. I think you also called another one resilience testing. I don't know if that's the same as chaos engineering, but can you talk a little bit about how someone would perform a resilience type test?

Yeah, I think they're related and I think chaos engineering is great. I think it may be difficult for companies to have that capability within the team, but definitely it can really show some big issues that's better to be known beforehand before you actually release it in production. So resilience testing in general involves just running a load test and then simulating an outage so you can turn off servers that are part of your stack to see how the application behaves and you can also test the behavior of your load balancing to make sure that that's working well. I really like Netflix's approach on this and in the realm of Chaos engineering where they have this really interesting concept of monkeys set loose in a server room so they have like a chaos monkey and all sorts of different monkeys or just a series of scripts that simulate what would happen if servers were just randomly offline or something didn't work the way that you would expect it to.

You did mention capabilities of teams when doing something like chaos Engineering, but I think we talked about this in the past, especially at previous Perf Guild and that is sometimes you don't have skills for people to really dive into protocol level type scripting. How does that go about into fear of missing out when selecting a tool? Any best practices or things you recommend for teams when they're selecting a tool to perform a type of fear of missing out type of test?

Yeah, I think human factors are important to take into consideration when you're creating any sort of test plan because it doesn't make much sense to force a non technical team, or a bit more of an inexperienced team at least in load testing, to go dive into protocol level load testing, which can be a little daunting. I think that tools aren't nearly as important as the types of tests that you want to run and the reason for running the tests. I would suggest that teams go with whatever tool works for their processes and luckily we live now in a time where browser level load testing is a viable alternative. Browser level load testing is a way to drive the browser so that instead of needing to know specific API requests that are sent by your application, you just need to tell A browser level load testing tool like Puppeteer or our version is Flood Element or Selenium is also really popular to click on a certain button or type something in a text field. That's a lot easier to start with. If your team doesn't have the capability of doing protocol level load tests, great.

And so when someone's actually choosing a tool, should they also be planning on not only the tool but how are we going to scale this? Because obviously when you're doing a type of fair missing out test, it seems like you need a lot of users. So how do you go about running a fair missing out test then?

I definitely think that it's important to keep that in mind, if you know that you're going to be running thousands or sometimes even millions of users. That is a disadvantage of browser level load testing tools because they are more resource intensive. So you're going to require a lot more servers and load generators to generate the same amount of load compared to protocol level load testing tools which are really efficient. But I always like to champion hybrid load testing because I don't think that it's necessary to choose. In some situations it makes a lot of sense to choose both of these different types of tools. So I think you can generate majority of the load with a protocol level load testing tool because they are really built for that and really efficient in generating large amounts of users. But then also get some of the front end performance benefits of a browser level load testing tool so you can run a few users of the browser level tool and then the majority of it with the protocol one.

I'm curious to get your opinion on this approach. This is sometimes what testers will resort to is they'll have maybe say they have a thousand users that they need to emulate, but rather than use an actual thousand virtual users, they would just do say 20 users, but like triple the amount of transaction workload to try to mimic a thousand users. So is that a good or bad approach?

It depends. I think that's the answer to a lot of these questions because you can't say that one size fits all. It really depends on the application and what you're trying to test. So I think that concurrency, the user concurrency, can be really misleading because a lot of performance engineers, myself included, use the number of users as kind of a measure of throughput. And to some extent it is a good kind of ballpark. But at the same time that doesn't accurately describe the load because as you were saying, that you can also just increase how quickly those users are making requests because a user can be technically active and concurrent, but not really doing anything, or perhaps doing it more slowly. So I think that if you're testing something like the number of sessions, like login sessions that are active on your application, then you do need concurrent users to match the number of users that you want to test being logged in. For other situations, maybe that's not important. And it's just a matter of how many requests your server can process. So it can really depend on what you're going for.

Great. So you have your test, you have the plan, the amount of users, you run your test, you get the results. I guess with these type of results could vary Based on run to run. So how do you know what's a good result?

Well, a good result is one where you've been able to successfully isolate certain variables of your test, for one thing. So load testers often say that a test is successful even if it failed. Because the point of load testing is to test your application, it doesn't necessarily mean that it has to be a positive result. I think a negative result is sometimes even more helpful because it can tell you exactly where your application is likely to fail. And that's a lot more useful than a test where nothing went wrong. You actually want it to go wrong before it goes live?

Absolutely. Now, this question is from a different blog post you wrote, but I think it relates to looking at results. I'm just curious to know your opinion on this. Then. How about cognitive biases? When they actually look in the results of these types of tests, does that affect how someone will perceive the results or how they'll move forward?

Yeah, I really love the intersection between technology and human psychology because I think most beginner load testers or performance engineers fall into the trap of only looking at the technical parts. So the code or exactly how to script or with what tool or what protocols and all that is important. But it's also really important to recognize that every part of the process involves humans, and it starts with you. Because as a tester, you make mistakes. You make decisions that are not always based on fact. Sometimes they're based on an idea that you have that you then seek to confirm. Or maybe you've had a previous experience that causes you to jump to that solution over other solutions, even if they're more likely. I think sometimes as developers, we can also we've all experienced the feeling of just being stuck on something seemingly trivial and going outside, having a break, and going for a walk can solve it. Those human factors are not really often discussed in performance testing, but I absolutely think that you should take it into account. I also think that sometimes we focus on just making something faster, but we don't look at the human factors later on in the performance process. As an example, I have previously been involved in load testing an application where we needed to get a certain number of letters out, and we got it really, really fast. Unfortunately, what we didn't realize is that those letters were a little controversial and the customer support department hadn't been adequately trained or even involved in the decision to handle irate calls from customers about those letters. So it can be that the bottleneck isn't even the technical parts of the application. It can be much later, but those factors and those aspects are very real and can really affect your outcomes.

Okay Nicole, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts? And what's the best way to find or contact you or learn more about Flood IO?

Okay, I would say value humans over code because humans make code. Code is there to serve other humans and ignoring that part of it is just going to lead to trouble. And as to where people can contact me, I don't want to make people spell out my last name because it can be difficult. So you can contact me at Nicoleudd IO. Fludd's website is flood IO. And if you want to catch me on other social networks, you can find a list of them@nicolevdh.com thank you Nicole.

For your performance Testing Awesomeness. Links to every value we covered in this episode. Head on over to testguild.com P40 and while you're there, make sure to click on the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance testing tools like Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.