Continuous Integration Performance Testing with Paola Rossaro

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Here it is, Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. One thing I've seen teams struggle with is including performance testing earlier and earlier in the software development lifecycle. This is getting even harder for teams as more and more companies are looking for ways to also incorporate their performance testing into their continuous testing, continuous delivery pipelines. In this episode we'll be talking with Paula Rosario, CTO and co founder of Novella, all about how performance testing can be part of your continuous deployment lifecycle, helping you achieve faster deployment of performance scalable applications. This interview actually was released a few years ago on my other podcasts, but I noticed I don't have any episodes yet really on incorporating performance into DevOps, so I wanted to share it with you as well. You don't want to miss this episode. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Paula, welcome to the guild.

Thank you. Thanks for inviting me.

Okay Paola, before we get into it, could you just tell us a bit more about yourself?

Yeah, sure. Yes. So my background, as you said, it is in the performance and is in the technical field. I have a PhD in computer science and the topic of performance and multiprocessing. At that time I worked in the distributed system and then I moved to the industry where I also cover different roles as.

The architect, designer and then managing also several projects. Some related to the performance specific and others related to other aspects like security and other topics. But interest has always been in that area. I worked in different companies, startups and none and then at some point I met longtime friend of mine, another Paola who also.

Was. We worked together in the past briefly and then we always kept in touch and she had an interesting problem and so we started to talk about that together and we thought it was an opportunity to start a company. About that.

So what's cool about your company is not only is it true women co founders, but. But you both have the same first name.

Yeah. So the other Paola. Paola Moretto, yeah. She also has a technical background. In fact, we met at UC Berkeley first doing some research together. And initially we just started because we felt there was a synergy in terms of interest and skills that we had. We weren't close to.

Male participants just as a principle, so we would, you know, we would accept anybody. It just in a way just happened if it was just the two of us, two women. And we thought that that was though, a great opportunity. And because probably we have known each other also for a long time, there was a lot of trust, you know, and confidence on each other that was very helpful in building the company.

Cool. So let's get into some DevOps at a high level. Does performance testing fit into DevOps? Do you see it as actually part of a DevOps activity? I guess we'll start there.

Yes, I do see testing and performance testing as well as part of the DevOps activities.

First of all, we can also decide there are different ways that people define the DevOps. I actually was talking to someone recently who said, yeah, well, our DevOps team doesn't communicate well with development. They said, well then that is not a DevOps. You just said contradiction on the term.

But performance testing. DevOps is really.

Complete in overall review of the software development life cycle and as such includes also all the testing, including performance testing. So the key, or one of the key components of DevOps approach to development and to production is really the ability of being able to automate and make the whole process fast in execution. So performance tested has also to fit in this picture. And so in order to fit, has to comply to these requirements. So being fast and.

Agile.

Awesome. So a lot of kickback I get from teams is that they say even if they did performance testing in DevOps or in the development environments is that it's never the same as when we roll it out to our customers because our database, our data isn't as large. So when you talk about performance, are there any tips you give teams to say, even if you don't have the same size database as your customers, there are still other key indicators you can look for.

Yeah, so there are definitely. And first of all, if you have a smaller production in a way of your production environment and you already have issues there, that's a key, clear indicator that you're going to have problems in a larger environment.

But also you can, I mean, there are also things that you can look in the real traffic, obviously performance testing is not something in isolation. I think it's probably one of the key concepts of DevOps is like everything, there is basically a synergy in every single step of the software lifecycle. And that's true also for the performance testing. So there are some parts that you can also monitor once you have a production environment on the real traffic. And then you can also sometimes avoid some issues just even in a smaller scale. Like you were talking to some customers, they were saying, I don't know, they have Java virtual machines and one of their concerns also for performance is memory leaks or things like that. They said they can reproduce a memory leak even in one instance. So they don't need to have 2,000 instances scattered all over the world to be able to figure out, oh, there is a memory leak. There are things that you can also test in isolation in a way or in a smaller scale. And so you should take advantage of that.

So how do we get teams more valid with performance when they have a requirement? Do you think each requirement should have a performance metric tied to it or when do you actually do the performance? I guess when does it come in? When you're actually developing the software and you're trying to get it into your pipeline?

Yeah, that's a good question. So we've seen people using performance testing at different stages.

During the development and the pipeline.

There's the general tendency that we see where majority, also the customers we have fall in. It's when you're getting closer to the final pushes. So you have a pre production environment and all the pieces in a way are already there together.

So you can test them that way. But we have seen other situations where actually this is becoming even more interesting I think where for instance, when you make a large use of microservices, we've seen also microservices tested in a way in isolation and then because of often the ability of joining them together in more complex services, also then the ability to build up on that initial set of testing and then create more complex. And so in a way covering almost every single step from a little small microservice to a whole application.

There isn't a fixed recipe in a way. What I can say is in general it's a pre production.

But we see more and more uses where There are specific APIs or microservices that are also tested also earlier on in the phase. And you can always look if you use it methodically and regularly, like having regressions every time you do a push, you could Run at least some small test or specific maybe to that area and they can give you already a feeling of possible regressions, issues to complete. Where can we use it? We also do see some use in the production environment after the push in two cases. One is when there isn't really an environment available before for different reasons and the production environment has peak times and slow times so that you can have almost an isolated environment. So. So that's acceptable too if someone doesn't have other options. And also we do see some of the performance testing used as a real time monitoring in a way, so that it's like what we call the synthetic monitoring, like a very more elaborate way of pinging resources. And it's not just a ping, but you get a full set of key performance indicators every time you do that. So in this case, usually they're smaller tests obviously that want to overwhelm the existing scenarios, but they are very useful as well to see if there are changes throughout the time.

So I think that's a great point. I think when people think performance, they automatically think like a stress test or a load test. But it almost sounds like even just running maybe a few users, adding a little load and just monitoring it is going to give you some really informative information. From Sprint to Sprint.

Yes, exactly. Yeah. And we do see people that they have, you know, other synthetic monitoring, you know, they said, yeah, we can do ping or pingdom or things like that. I mean, nothing against these products obviously, but they were saying it's nice sometimes to have something more complex, you know, we can build in those full scenarios, you know, that we can use also to monitor, you know, the real product.

So another thing that people struggle with though is monitoring, because you can monitor anything and everything. So, so how do you know what's important? What metrics do you look at or what counters do you think people can look at to help troubleshoot maybe performance issues?

That's $1 million.

Yeah, right.

It's difficult also because every. There are now tons of monitoring solutions and.

A lot of them are becoming very complex and elaborate and they're really rich in terms of results and problems, honestly. Yeah, problems can happen anywhere. For the monitoring. We see that one of the trends is really having sort of like a monolithic view of the whole system where you can go into details at the same time on each component of your application from just the network traffic. So we see performance also the level, there's several solution also hardware solution to continue to monitor and then you see that at the system level and then you see it all over along the stack until the end to end. So problems can be anywhere. And having information though it's important because you might have some alarm at the top level when you see really response time. The first maybe signs or concerns could be either high utilization of a system or so the high congestions and slow response times. This could be kind of the classical.

Sort of red flags, I would say. But then it's important to have all the other pieces of information because it's easy then to blame the system, blame the network, blame the infrastructure, blame someone. But if you have data that you can trace, it makes it so much easier to exclude things and narrow down to the real causes.

I used to be more of a performance tester. Now I'm more. I don't know what, I'm more like an architect now, but when I was doing performance testing, we never really tested in production or when we did, that's when we got the best results. So do you see more and more companies actually doing performance testing in production or do you see monitoring in production being more of a something that more and more companies are doing?

I do see more the monitoring, honestly. The performance in production, yeah. Like I said in one of the previous questions, I think.

We see occasionally some production testing, but so if it is in production mostly, I mean, as a good rule we have this synthetic monitoring, so the smaller tests in general, but maybe can be rather complex in terms of scenarios so that you can not just ping your homepage, but you can get more information about general workflows. And then at the same time we do see occasionally testing in production, like with real load or stress testing in situation where it's not possible to test.

Earlier. Sometimes it's because maybe the product is part of a more complex situation and by itself it wouldn't make much sense to test it. Or it could be that it's hard to recreate a testing environment that has the same capacity that.

The production environment. So these are kind of like the scenarios we see. But I wouldn't say I see a trend of doing a heavy load testing on production only like smaller tests, but that yes, I do see.

So are there any trends that you do see often? I guess you speak with a lot of different companies when you're trying to implement your solution. Do you see a same problem companies do all over and over again that you think is an issue or a trend you see more and more people moving towards?

Well, one trend in general, but I'm not sure if it's really the question you have but it's something that we do see is that people moving away from older tools and all their solutions that are kind of slowing down their approach to DevOps. So that is definitely a trend that we see.

We do see occasionally people that say they don't use Agile but they are saying we are moving to it or people that are not using it. Doesn't have to be a public cloud, but even any sort of cloud.

It'S becoming more and more unusual to see people not using the clouds. So these are kind of trends that we see in general and as kind of like a side effect in a way of these trends I think there is more interest in solution. There are.

Covering making easier to build the full pipelines, make it easier to integrate testing and all the pieces that are needed to be in the pipeline. So I definitely see that trend and also both in people wanting to use these solutions and in companies that are offering these type of solutions. So that obviously it's also in the case of performance testing we do see people definitely trying to move away from more kind of old style solution that would take days or weeks to set up tests and run the test and set up infrastructure and so on, but like to have this information within minutes.

So that's a great point. That's another thing that a lot of teams complain about or they have issues with is they don't have the expertise with performance testing. Creating the script. So how is Nouvella different than these other tools? Because I guess it's newer. So of course it was, it was created with DevOps in mind. But how do you see it different than other players in the performance testing space?

Yeah, I think that, yeah, one of the major point differentiators is really the ease of use and, and speed, speed of execution as a consequence. We do hear several times customers or people when we go to conferences and so on saying wow, I never seen anything so easy or it's really so intuitive or I could see my issues within.

Seconds.

And that is really, I think one of the key factors. Another thing that I wanted to mention is like when we're talking about, you know, we talk about performance obviously and a performance tool has to provide performance information fast. And so, and that's the other part. It makes it easier to modify and change.

The tests. So that also makes it agile. And so these are all components that I think are very important and differentiate us from almost all the other solutions. The other aspect is the results. So we do provide out of the box a set of results that are very easy to.

Visualize and to understand really, even people that don't have a strong technical background can jump to, oh, there is really a big issue here, or response time is going up or this call is not acting properly and so on. And that takes really just also a few seconds or a few minutes. So I think these are probably the major differentiator.

Awesome. So you've mentioned the ease of use. So a lot of people when they think of performance testing, like I said, is the scripting piece doing parameterization, correlation, things like that. How does your tool handle it differently than another tool? I believe it's is a scriptless technology that you use.

Yes. So we do have a scriptless technology. So we have a way of recording the complex scenarios, the scenarios you would like to then reproduce. And once they are recorded they can be edited internally, but still using ui. We have a concept of variables and meta characters that allow you to randomize and correlate and parameterize different values.

In the workflow.

This is also very simple. We had the learning curve also for a relative complex solution, it's often maybe just talking 10 minutes with us and showing them, oh, that's what you need to do.

Like I said, the ease of use has been something that we heard again and again and again. And we have customers that are using authentication. They're using multiple parallel scenarios and providing different.

Values every time so that it's all possible and really easy to implement.

Awesome. You also mentioned results. And that's another thing I remember struggling with earlier in my career was I did a performance test, but then I needed to get the input from the database guy, the UNIX guy, the network guy. So does your tool act like a dashboard for all these different parties or how does it work now?

So we do have. So our tool has a dashboard with indicators.

More on the end to end like the response time and throughput errors and also critical calls and so on. We do have integrations with the monitoring tools, for instance, where you can also look at the same time at.

How the back end is responding, the database, CPU utilizations and so on. So that is possible to easily integrate with other solutions as well so that you can have all the pieces together. You can also just share our piece of information. We have just.

A link that can be passed around and it has an interactive HTML based interface so that anybody can look and share the results that way.

As I mentioned earlier, they are very easy to use and to identify potential issues in an iBlink.

So whenever I talk to a tool vendor, I'm always curious to know about what they're working on. So do you have any exciting things that are in the pipeline that you think really going to help performance testers in the near months or the upcoming year?

Yeah, yeah, we definitely have of course improvements on what we are doing right now, but also different areas that we are expanding and one part is a larger focus on what we call a client testing where we really use a full browser and can provide type of waterfall results that can be then used in parallel also to load testing or just as separate set of results that will give you really another angle in a way of the performance of your application. Kind of complete the end to end user perspective.

Another piece that we are also working on is the ability of.

Doing a deeper analysis of your results. So being able not only to tell you okay, these calls might have problems and might be slow, but also the ability of going deeper in the stack and tell you maybe this method might be the cause of a slowdown when you have a high load or this set of methods or this square in the database can be causing a lot of issues. So. And that's where we are to. So these are kind of like probably the two top things. In addition to that, we are expanding also regions where it's possible in the world geographies where you can run tests. And we are adding better ways of tweaking and the test executions and so on and adding more to the results. I mean obviously these things, it's never ending, but I think these are the major milestones in the roadmap.

Okay, Paolo, before we go, is there one piece of actionable advice you can give someone to improve their performance testing efforts? And let's know the best way to find you or contact you or learn more about Nuvola.

Well, the advice that I give you people, that I could give to people is like test, test and test. It's not just, you know, performance testing, but any type of testing. So iteration is also important. So you keep testing, improving and you know, and repeat. So that's kind of like one of the major piece of advice. And if you want to contact me or Nuvolites Nuvola website.

Or you can find us on Twitter or LinkedIn. But our website is nuvola.com is n o u v o l a dot com and my name is Paola Rosaro. So it's p-a o l a dot Ross s a r o nuvola.com thank.

You Paula for your performance testing awesomeness. For thanks to everything of value we covered in this episode. Head on over to testguild.com P21. And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Smart bears. Awesome Load Ninja Load Testing Performance Testing Solution so that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you test succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.