Performance Testing More than Tooling with Harinder Seera

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Horinda all about Performance Test Tools does not equal performance testing and probably touch on a bunch of other topics. He's an expert on things like blockchain, Big Data, performance strategy, and a bunch more. Really excited to have him on the show. Horinda is a consultant based in Melbourne, Australia. He has a blockchain Advisory Board of the Global Skill Development Council. He's also the author of Introduction to Blockchain. He is also the co author of the IBM Red Book on Performance and Capacity Implications for Big Data. He speaks at a bunch of different conferences and workshops and contributes to a bunch of various publications. Hernandez currently leads the performance engineering team for a major financial organization in Australia and part of his role is actually a performance engineer or establishing a performance engineering strategy for the organization's DevOps team. So he has a lot of knowledge like I said, so you don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenec and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and LoadUiPro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try, but head on over to smartbear.com solutions performancetesting to learn more.

Hey Hirunda, welcome to the Guild.

Thanks Joe. Thanks for having me on the Test Guild.

Awesome. So like I said, you have a lot of different experience. We're probably going to be all over the place, but I guess I'd like to start with. I guess before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, I think you covered pretty much in my bio.

Awesome. Awesome. So I know you recently spoke at NeoTest's Pack Online pack. They have a great I think it's almost annually now online performance testing conference and one of the topics I think you touched upon was Performance Test Tools does not equal Performance testing. Could you just give us a little idea what was the meaning of that session? Or what was the Main gist of that session. Sure.

So I'll give you a bit of a background. I started performance testing over 10 years ago. So the first thing basically my background is functional testing. So it's surprised when I tell people I've been functional tester. And then I moved to performance testing. So when I started performance testing it was hey, you don't touch a tool. So I had a mentor. Basically the first day on the training gave me load runner analysis graphs. So I said hey, here you go, you've got these graphs, tell me what they mean. What exactly can you understand? I looked at it like two, three minutes I couldn't make a sense of it and I just said hey, this looks like a spaghetti to me. All different colors, I can't make sense of it.

So that's how I started and went over the years now I noticed there's a lot more to performance testing and what I'm seeing the patterns in last couple of years with DevOps and Agile where the focus is more on performance test tool or the performance test framework rather than what exactly encompasses performance testing as a whole. There's a lot more to performance testing than just a tool. So that was the gist of my presentation at NEO this.

So it's kind of funny, I have kind of similar background. I started probably older than you over 20 years ago, but I started as a functional tester and then I started a job and I got into performance testing and the same thing I learned because someone was there that knew the tool and then and then trained me on what is performance testing. So is that still a gap like this doesn't. There's never seems to be enough education around performance testing. It's always almost like learning on the job or is that how you think people would normally learn? Or the preferred way of learning performance testing is on the job.

I've got different experiences based on on the people I've talked to. So for me I was working for a consultants company so they didn't put me to a client's to work straight away. So I had to go in house training for like few months because performance testing is a critical like a field, it's. There's lots to learn, to understand. So they didn't put me straight up to a client and saying hey, go and do the performance testing. Whereas I've seen other guys that I've met, they were learning as part of the company so they were given hey, you go, you've got your tools, start doing performance testing. So there are different experiences in my case it was more you got to have few months, couple of months of in house training before you head off to the client.

Gotcha. So actually everyone always talks about oh, shift left. It's so great, it's going to be awesome for performance testing and security testing. But as you mentioned, it seems like sometimes we actually create bad practices when we try to force things left. So can you talk on some of the patterns? You notice where people are getting maybe wrong or kind of not doing correctly when they're trying to incorporate performance testing as they shift left?

Few things that I've noticed is one of a major one is the data. So nowadays what I've experienced is you know, developers doing performance testing. So they don't understand the concept of using different data sets. They think having a stub you can one sense they're right basically trying to look at the throughput of the system. But data does play a really critical role as to how system behaves. So for example I was testing an application where you're dealing with customer related information so they could have lots of different accounts. So there's a skewness in their data which most of developers don't think of it, they just think, all right, we'll use Mr. X and Y and that's it, we'll just keep on hitting our system with that. But in reality that's not how your application would work. You might have different kinds of skewness, you'll have different kinds of customer, how they access the system. Or even if you're dealing with APIs you might have third party APIs that you're making call to the delay that you need to add knock on effect of if they slow down, what will happen on your system. So those kinds of practices I don't see when they move to left. So that's the major one. Other one is what happens is they also don't think of how. So I've got like work with 10 different squads that I try to holistically look at the performance testing for them. And one of the things is there's no cross communication between the teams. So even though one team's API is consuming other teams API, their requirements are different so they're testing to different requirements. And when you come to the holistic performance testing, you look at it, you're like this don't make sense. So you've got one person has got a higher spec system compared to the other team. So I noticed that kind of things as well.

So I know you're also part of coming up with the strategy for performance testing. How do you handle issues like that when you do have teams working on different features and then when they finally merge up into the main branch, it's like, well, we didn't know that API was going to interact the way we were testing our API.

So basically one of the things I've got is I've got a couple of guys in my team who looks throughout the whole squads so that you've got an overall team, like a couple of guys sitting at the top that look at holistically understand how the whole landscape looks like of the services or our application. And you've got the performance team within the squads. So their focus is just around what squads are testing. But then you've got these two guys sitting at the top, they look at the holistically and saying, hey, we know from our discussion with the architectures this is how the whole system will look like and why are you guys testing differently or you've got different requirements than these guys over here who have got the same. You will be consuming the services, but they're running at a different sorts of SLAs and throughput kind of thing. So that's how we basically look at the strategy. Another bit that we look at is we try to get engaged as early as possible in the design phase of the our applications services as well as when we are trying to define the SLAs for our services. So at that time we can raise questions saying how these services will be consumed, who will be consuming it. So we've got like this whole. That's how we work.

Nice. So it sounds like you almost have like performance architects and then each squad has a performance team or is it developers doing performance and that's what you're calling a performance.

So we've got a performance person in the team. So we're trying to educate developers to do performance testing so that things like it's really hard to scale performance team and with the whole Covid issue and stuff. So we have to shrink our team. So we're now trying to see what's the best way we can still handle the load that we have in terms of performance testing. So one of the things we're doing is trying to train developers to do performance testing, explain how it works, how performance testing works. So I was trying to understand the requirements I've been given from the solution designer and they've got the requirements, but it doesn't feel like how things fit together. So I was doing a flow diagram with them or sequence diagram as well to understand how these APIs fit together and when I was explaining them, the developers were like, oh, we didn't know. This is how our whole floor looks like. Ah, good question. Ah, we've got a, looks like we've got a throughput over here which is higher than this API, but we can't make call to this API. So like that's how we're training them so they understand what's involved. So in a nutshell, yep, we currently do have performance tester in each team, but with the whole shrinking.

Reducing. So we have to start looking at different ways how we can do performance testing.

Gotcha. So you did mention SLAs before as well and it sounds like you're doing a lot of APIs. I don't know if they're microservices. How do you come up with realistic SLAs where it's not just oh, it should be fast or it should be under two seconds when you know, the whole transaction together takes about five, you know, something like that.

To tell you the truth Joe, it's really hard to come up with SLAs. Now one of the things we have tried to do is just from the experiences of other performance ingenious architects, most of them got 10 years experience and stuff, so they have dealt with microservices. So we use some kind of a benchmark industry benchmark saying hey, most of the APIs should be less than 1 second or 500 milliseconds. So let's say if it's in cloud, we'll look at more around 300 millisecond. If all our APIs are within cloud, AWS or Azure or whatever, we'll make 300 millisecond as our baseline number that we need to start with. If we are trying to access application which is on our on prem stack, which is not in cloud, but we host it, then we add, look at the production data and add a buffer to it. So it's not an easy coming up with sla. But what then we do is once we've got this information, we look at what's happening in production. So once the application goes live in production, we monitor the consumer or the services are consuming these APIs and then revisit our SLAs. And if we see, hey, we thought the sla should be 300 millisecond, but all our APIs are responding within 100 millisecond, we need to bring down our SLA. So now next API that we set up will be 200 milliseconds. So it's like not just shift left but also shift right. So get a feedback coming in.

Absolutely. So that feedback, then how do you let the team know how things are performing? Like you said when you first started, someone hand you a graph and look like spaghetti. Like, is there a better way to communicate performance at a high level to all the whole team? So everyone's on board that know, hey, we released this and the performance is worse than it was previously.

Ah, yes. So we have, over time we have set up dashboards so we use different tools like Splunk, cloudwatch, so we set up the metrics on that. So everyone's got access to that CloudWatch, not because it's an account level access, but for Splunk. Most of our dev teams, even business product owners, have got access to Splunk so they can log in and see what's happening. So over time, what we do is when we're doing performance testing, we'll set up the dashboards, we think these are the metrics we should be capturing. And then once testing is completed and in discussion with devs saying, hey, these are the metrics we'll be capturing and performance testing, we then promote those dashboards into production. So that way we've got same level of monitoring in production and in our non product performance environment. So they married up. So whatever they've learned in performance testing is what they will basically monitor in production. If something comes up in production, then we'll revisit or feedback to our performance monitoring environment or stack and update it. So that's how we work.

Nice. So are you able to capture or find performance issues before it actually goes into production? Where you say developer A and B checked in code and performance got worse. So we're going to reject that. They're checking, are you that sophisticated yet or is that unrealistic?

We haven't gotten to that point yet. It's something we want to do it to get to that point, but it's more around we do have. We use cicd as well. We've set our performance testing to use a Jenkins pipeline so we can monitor it through that, but we haven't gotten to that stage where developers can look at it and saying, hey, I've done something wrong over here, I need to revert the code back. So we haven't gotten to that stage yet, but it's part of our pipeline.

Also, when people think about performance testing, a lot of times they jump right to like a stress test where we're going to hammer the system with 1,000 users and they haven't even really looked at like a one user or two user test. So do you do a lot of baselining or a lot of performance tests from like a one user perspective as well?

Yes, we do that often. So one of the things we trying to promote is even the functional testers because most of our applications are API driven. So before even gets to us, the functional guys are already doing like a single user test with the APIs. So we would try to promote it to the functional test to say, hey, when you're just writing your API because they used to like Postman, it gives you the detail of how long the request took to respond. So they can look at it and say, and we've already told you like we need roughly 300 milliseconds for this API. If you're already seeing more than 300 millisecond, probably a good idea to have a chat with dev now before it comes to the performance team to start performance testing. And sometimes they get the defects early on, even before it arrives to us. And then what we do is we do run with a single user but with different data set and different skewness everything else and different, different flows as well, but with a single user to understand what's happening before we even start running a proper test.

How critical is performance on the list of things developers need to get done though? I don't even know what kind of application is, I guess, is it like, is a financial application so performance is like critical or is it. I'm just thinking like when I was going through the definition of done, like are they really thinking about performance or is it just almost still an afterthought? Is it still a struggle to get them to think about performance?

Well, you've tightened a good point. Yes, yes. There are different flavors that you find. So there are some team members who are really, really passionate about performance. So as they're developing, they're thinking about performance and others they are developing, but they might not be as focused on performance. So there are different challenges we face with that, shall I say?

Yeah, absolutely. So is anything you could put into your performance engineering strategy that helps maybe get everyone on board earlier with performance? So it's not like, oh, here, okay, now I need to do performance. So I guess that's a culture thing. But any like really firm thing you found that helps kind of spread the performance of culture, the culture of performance into teams?

Yes. So one of the things is in the requirement state definition of done for a feature. So we're trying to promote, hey, as part of your done statement you need to have all these requirements, performance requirements completed before you even move it to next state or next gate of a deployment. So they are aware of it and it's, it's been a journey for us. And other thing is like if you're a big company, you've got lots of people coming in, going out so you have to have that continuous training happening. So we have even written some documents on basic understanding of what performance testing encompasses. Like what is throughput, what is response time, what is error rate, how do you capture these metrics? So we give that as like what you call the onboarding documents for them to read so they have basic understanding of what exactly we're trying to achieve.

Nice. So you did mention it's not about tooling but when I started over 20 years ago, oh my gosh. But LoadRunner was the tool and I think it still is a really great enterprise tool because it has everything built into it. So it's not only just tool to use to create the test, but you also analyze the test and see the results. It seems like a lot of the newer tools are focused maybe on just putting load on but not necessarily looking at the results. You have to put together a whole solution using all these open source tools. So I guess my question is, I know it's not about tooling, but are there any tools that you think have really helped your team with performance testing when they're paired together?

We've got thing was like one of the challenges we had. I've got a background in Load Runner so I've used more commercial tools previously. But as our culture was more of a DevOps agile and we wanted to promote performance testing to the developers and even to extent to get functional tester to do it. They didn't understand how LoadRunner worked or the language. Let's say you talking at C, writing in C, writing code in C, capturing network traffic. But most of the guys had experience with JMeter Gatling. So thing was for us, okay, if we need to get these guys onboarded or work with us, then we need to look at the tools that they are comfortable with and then start adding other tools. So we used, we basically use JMeter but for monitoring analysis we use Grafana, InfluxDB and then you've got your APM tools and plus you've got like tools, log analyzer tools like Splunk, so we use that. So it's like you've gone from one tool that gives you everything to multiple tools and you're trying to Set them up, connect them together so that you got one big framework.

Just to me, I just find that frustrating. But I know now everyone has like million dollar budgets for enterprise tool, but yeah, just curious to know how much of a struggle that is for teams.

I think it's because I'm just one part of the bank. So we've got other teams that use loadrunner.

But they're dealing with system like Siebel.

Right, Right.

So they've got different protocols that they need to deal with. So which like tools like JMeter and other ones might not be able to deal with it. So they still use loadrunner, but whereas our one, it's more API and the culture that we're trying to promote is DevOps Edge also. And their background's been they've used these tools previously like JMeter, Gatling, so it's easier for them to just start using them.

Nice. So I know a lot of times this is a thing with automation as well, is you choose the language your developers know, but then you end up you just have a tester still writing it because the developers still don't contribute to it. So is there a way that teams could try to go too far left where they use the tool like K6 and they're using actual Java to create their performance test, but then the developers still ignore it, or are your teams. Because the ad developers really are using Gatling and Scala and they're coding up the performance test.

Yeah, so we do see that it's like different pockets in our teams. So there are guys who will basically go and use it and they want to learn and they want to. They're curious about performance, so they're like, yes, I want to improve my code, whereas we've got others who might not be basically that concerned. So it's a bit hard. So it's. Yeah, it's hard. It's just a journey for us and we keep on pushing them little bit by little bit and they start learning over time. But like all other colleagues that I've met, they've got the similar challenges they're dealing with as well.

Absolutely. So to switch gears just a little bit, I know you also wrote a book on blockchain. I don't get to speak to many people that actually talk about blockchain, especially from a performance perspective. Now, I was hearing a lot about blockchain a few years ago and now not so much. So is blockchain still a thing and are you still a believer in it?

I am still a believer in it, but I have Got the view that basically blockchain needs the right use case for it means if you look at things, you can use database, you can use any other mechanism, but there's no such use case that I've come across other than cryptocurrency, where you can use blockchain properly. It's like the right fit for blockchain. We've got other companies looking at your supply chain, which is probably another use case. But then what are you getting out of that? I don't see much of a value other than. Yep, we know we. This is the supply that we previously. What? Like I think in America, which there's a company, big grocery company, they're trying to get the supply chain set up with blockchain. Don't remember their name, but it's like I'm still battling with that saying hey, there is blockchain can be used, but we haven't found the right use case for it other than the cryptocurrency. That's my view.

Gotcha. So as a performance tester, is there anything different with blockchain that they need to know about if they were getting on a project that actually used blockchain from a performance perspective?

Yes, yes, definitely. So it's more around depending what kind of blockchain they are dealing or are they talking about the decentralized system or are they talking about centralized system? So if it's decentralized, you've got lots of different peers that needs to sync the blockchain data. So what kind of bandwidth they're on, how long? Even the consensus that they need to deal with. So what happens if one of the peer doesn't receive the data and they're trying to come to a consensus? So there's lots of things around that bandwidth. How do we deal with that? How do you performance test? How do you make sure you've got enough in your block? How much data can you have? How much even disk space because you've got all peers copying all the data onto disk. Disk is cheap. But most of the guys, let's say if I'm the user, I'm not going to put an aws, the whole blockchain. I might put it on my laptop and trying to do certain things with it, but I don't have enough disk space. Maybe one terabyte. But as blockchain grows, sooner or later one terabyte won't be enough. So you're looking at performance issues now with your laptop, where it needs to write the data, how quickly it can write, even trying to download the whole blockchain. It will take time and until it stinks, you might not be able to even do anything with it. So you've got all these different performance challenges that you're dealing with.

So another topic I know you're also focusing on is big data at really high level, I guess for big data, before we wrap it up, is any performance things people need to focus on with big Data as well?

Not really because we're pretty much covering all of it. It's like it's just a data.

It'S different kind of data you'll have, you'll have structured, unstructured data. But the challenges that we see with the data already, it's pretty much same with big data. Maybe using different tools like Hadoop, trying to do the map reduce stuff, which has its own challenges, performance challenges, how do you do map and reduce kind of thing. But other than that, it's similar challenges that I see with all your real time applications or even batch testing kind of thing. It's just data dealing with it.

Before we go, is there anything I missed or any question you want me to ask before I ask my very last question?

Something that I normally advocate is called three eyed Monster, which we're starting to forget.

Okay.

It's an old concept even I'm sure you also aware of it. I just give it a different name. It's like when you're doing performance testing, three different perspectives that you need to look at. Like you've got your customer view, you've got your IT ops and you've got your business view. I'm starting to see that's people don't put too much focus on it. As we're moving into DevOps, agile, that whole thing of hey, how does customer perceive what exactly is important for customer, what is important for business or what is important for IT ops? I'm starting to see that follow a little bit.

So why is that? I mean, so can we talk a little bit about the 3i monster and why shifting left or trying to create software quicker and faster. We may be losing sight of the bigger picture, I guess.

Yeah, because one of the things is as you've got different squads doing their own API, own functionality, performance testing of their own functionality, they don't understand a bigger picture as to what exactly even the business is looking for. What is business requirement? Even for example, nowadays you've got finops, we're talking about cost optimization, they don't think on that side of things as well. It's like, hey, we've got this new tools, new things, we need to use them, we'll put it in the cloud. But they don't understand end of day someone's got to pay for those services. If it's not the developer or the architect, it's the business. So if we can optimize a little bit, it will cost us over time. But that's, that's not what they're thinking. It's more like let's push this thing into production as quickly as possible, let's move it every week we'll do a deployment without even knowing what is the from business point of view is that important or not? And does it cost them or not? Same thing with end user, it's more like, yep, our APIs work. But what about end user? How does the end user sees perceives that performance of your application? APIs on their own, they work, they're meeting your SLAs but end user might be complaining it's slow and you don't know.

That's a great point because I would think there's pros and cons for doing API microservice level testing. But it's true. I guess you just see gets focused then on that small piece and not necessarily how the API is going to be consumed or how they're going to be chained together to create end flow. So you may not necessarily know that this, this particular API service is actually going to be used in this customer purchase flow or something.

Yeah, and that's right because as I was mentioning, each team might be focusing on their own APIs, but end of day they combine together to give you a proper or let's say one single page app. So it's one team doing their own API, another team doing own. But end of day when they stitch together, that's what the end user customer looks at it. And if any one of them is slow and you're making a third party calls, it will have its own impact which the teams don't look at it from that point of view. So that whole holistic view of customer, they starting to lose that with microservices concepts.

I don't know how you get around that though. How do you address that other than having okay, before we ship we're putting it all together and then once again it's almost like waterfall. Let's test it and then let it go once we test it all together.

Yeah. So in this client currently, because we are microservices, we don't have an end user per se, we've got a system which is consumer our services but it's a single system and it doesn't have any concept of end user for them. So that's where I haven't gotten much exposure here. But in the previous place where I was working, yes I was working for a retail company and over there we had to deal with APS and we were using tools like WebpageTest so we had set up our own webpage test in house and each API so as they were getting built we were using WebpageTest with different bandwidth emulations and seeing how what's the impact and even using like circuit breakers to understand if the third party APIs are not responding, what's their impact on our website, the rendering of the page. So we used that mechanism.

Very cool. Okay Arunda, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Best advice is start looking at how to analyze data because you can learn tools pretty easily, but if you are in performance testing, understanding how to analyze the data, even slicing and dicing the data is really, really important. There are other things I won't go into details like architecture, understanding the architecture requirements, but if you can understand how to analyze the data, it makes your life easier. That's where the crack, that's where everything lies in performance testing, the analysis side of things. If you can learn that, you're on your way to becoming a performance tester and if you need to contact me I am available on LinkedIn so I'll share my LinkedIn details and you can also find me on Twitter. I've just recently started Twitter so I'm still trying to get comfortable learning Twitter. But you can find me on Twitter.

Thank you Horinda for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com.

And while you're there make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Performance Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the guild to continue your testing journey.