Taurus Performance Testing & Modernization with Matthew Andrus

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we're talking with Matthew all about performance testing with Torras, end to end performance testing and much, much more. Matthew has over 20 years of experience with distributed systems and DevOps engineering, from network management systems to satellites to cell phones, including enterprise system testing and DevOps transformations. He's a former Broadcom purchasing and implementation customer and is very experienced with obstacles and opportunities involved with modernization of software development test. He also was one of our speakers at this year's Perf Guild and everyone raved about their session. So really excited to have him on the show. You don't want to miss this. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Matthew. Welcome to the Guild.

Hi Joe. Thanks. Glad to be here.

Awesome to have you. Before we get into it, is anything I missed in your bio that you want the Guild to know more about? No.

I've got a big smile on my face. It's a lot more than normally gets said, so I appreciate that. I did have a few years where I ran some non technical businesses and anybody's interested in that stuff. I have some hobbies and things like that, but from a professional standpoint, no, that was great. I appreciate it. Thank you very much.

Very nice. So as I mentioned, you have been in the game for a while and you've done a lot of different things in the space. So over the years, have you seen anything change within performance in the way people think about performance?

Oh my gosh. So right before we started, the words that were going through my head were Industrial Revolution. And when I started I had, you know, I used to look down my nose at testers and I thought they just kind of got in the way. I thought that they were overpaid keyboard monkeys and you know, they just kind of did the same thing over and over again. And I feel very fortunate to have been able to come over to the other side of things and realize the value not only of the process, but of those people who really care about it and are paying attention to the innovation. When I came into performance testing around 2007, Loadrunner was the premier tool and they really hadn't fully evolved, so I got to watch it evolve a little bit. But because I was a developer prior to testing, I also saw some of the lack and started working on some things and then saw the industry catch up. And I think that was a lot of my fortune, was that I was paying attention to things that were missing. I worked on teams that were open to the ideas that there were gaps that we weren't covering and if we could cover them before they were found in production, then we'd be doing the entire system of service. And at first there was a lot of resistance. That's a lot of extra work. But it only takes one or two things that you catch or things that you've mentioned, and then it does go bad in production before people start realizing, oh, there is some merit to that. And the last 10 years of my career have had the fortune of being around teams that appreciated that, finding a product that was very motivated towards innovation and capitalizing on information when they find it. You know, when, when you see a weakness, when you see an opportunity, not just waiting until the market catches up to it, but going ahead and starting to feel that out. Changes in DevOps, Marty Kagan and the ideas of, you know, breaking things down in smaller chunks, failing fast, learning about your investment path have all been a lot of things I've had the good opportunity and the good fortune to experience. And then at the perfect timing where the blazemeter tool that I was using as a performance tester was picked up by then CA Technologies and then Broadcom, it was kind of that Roman Empire. Perfect time in history to have caught up in the wave and had a company that was interested in helping more people bring me in and invest in my helping more of their customers rather than just going to one of their customers. And that's helped me appreciate and enjoy and even see more of how the industry and the technology is changing, where people tend to get hung up. You know, I had my personal perspectives of why some of the companies I worked for just didn't succeed in moving towards DevOps and having faster release cycles, more dynamic coding. I don't think it's as personal anymore. I think that there are some, some major aspects of this Industrial revolution, that people just don't realize how big the world is and how much it has changed and what kind of investment is necessary to not only catch up, but get ahead. And I love that I have a job where when people are interested in that, I get to help.

Nice. So how do you get companies that may not be aware of the need for these type of performance testing, security testing, testing in general, to really catch up and be faster continuously? Like, what was your mind shift when you were a developer to start embracing this of thought process? And how do you encourage other companies maybe to embrace it as well?

Actually, the initial shift for me was somebody else's vision. Somebody else had been a tester for a large company. That company lost the contract because they did not have the integrity that they needed to have. But my friend did and they kept him on and he needed to build the team out. And when he realized that I was in a position where I could take on that kind of a life change, he asked me to consider it. And six months later I joined his contract with a very large government organization. And even there, it was the combination of being willing to fulfill your role, but also be aware of where you might be helpful elsewhere and having a respectful attitude towards that, that over the course of a year was involved with a project that was in a lot of trouble, had the good benefit of the skills and the interest level to step out of my box and earn the right to be heard and then speak respectfully and bring the knowledge that people needed to hear, but may not have been ready to hear when I first knew it. There's a little bit of maturity, there's a little bit of luck, but there's a lot of keeping your ears open and tuning your skills, paying attention to the industry and building a rapport with your senior, with your leaders and your managers and whatnot, so that you are the person that seems to have out of the box information. But you're not being pushy where you're getting pushed down about it all the time. And then when those leaders hear a buzz or are given a mandate that they need to do some change or maybe they want to do some change, you can be a person that's a bug in their ear or a savior in a sinking boat if it's the case. But the combination of being poised for that opportunity, I guess, is what I would say has happened to me over and over again between the balance between being a little bit more interested but also being patient and helpful rather than a clanging gong or, you know, A lot of us tech people can be a little opinionated or a little pigeonholed into our technology or our experience and our expertise. And so it's a balancing act. But I think being a little bit forward looking and having an attitude to help the organization is what earns you the right to be heard when the time comes.

Nice. So once the organization believes in it, they have the vision, the developers are on board, everyone's on board. Where do you usually start? I know a lot of people are talking shift left and shift right, but with performance testing, where do you see is the best value for people to start with?

Well, that's a tough question. Unfortunately. I see the industry stumbling forward when they trip on things. There are a few organizations that really have worked on the internal cohesion to understand and move together in that direction. But a lot of times there's a lot of natural complexities like management that's not prepared for change, bad history of bad advice and costly efforts to change that make people apprehensive. So I've seen it go both ways. I've seen it go top down, where the top executive management says, in order to be successful in five years, we need to be here. Our research indicates that that's going to need to involve changes in these parts of our industry and in our infrastructure. And so we're investing in that now, top down. But I've also seen where bottom up can do the same thing where an individual can say, I see an opportunity and I'm going to look into it, get a little bit of leeway with their boss to carry their regular workload, but also start showing some additional benefit to the ideas you're seeing. And if you've got the right kind of manager, a manager will often say, well, okay, as long as you're getting your workload done, I'd be happy to look at whatever else you have to say. And Google showed us that model where they only insist that their employees work 32 hours a week because they want eight hours of that employee's work to be innovative and thinking outside of the box and bringing new things to the table. And obviously Google has benefited from that. My work with the product that I represent now, Blazemeter, was much along those lines where I was finding more opportunity in the tool. And I would just listen to other people's problems and I would say, if I could solve that for you, what benefit would it bring? Well, we could go to, we could get this out two months ahead of time. Okay, well, let me work on that. Invite me to a Meeting. Let me see what I can learn about your problem and see what I can help. And worst case, somebody who's not on their team spends a little time on something that doesn't work and they have to listen to it for 10 minutes in meeting. Most people can afford that kind of inconvenience. But if from that bottom up perspective somebody is paying attention and ambitious, but also respectful, you can find a tool out there to solve any problem. If there's anything that I've learned, there's either a tool that does it or tool that's flexible enough to give you the power to do it. And that's what I love about my platform being open source, both in terms of the platform we actually sell is completely open source, but then embracing other open source. It's that idea that we don't think we know everything about next year right now. So let's be open to the changes that are coming and the engineers that want to do that bottom up revolution that can start looking to solve problems that are out there. I don't know if you guys have seen the Pixar movie. Robots see a problem, see a need, fill a need. But from bottom up, that can work as well because if you can save the company a few hundred thousand dollars or get something done a little bit faster because you fill an opportunity, that can start a revolution as well, right?

I think when people think of opportunity though, for some reason they don't think about performance, performance testing. Do you think that's an opportunity for people to focus in on or are there other lower hanging fruit you think is a good start?

You know, there's almost always some limitation or something in every system that is either a bottleneck or a risk or introduces some performance problems. And not every performance testing environment is prepared to catch that. But when I was a developer, one of the first things I found using some Unix tools was that our core engine had a bottleneck. And I was really looking at it from a curiosity standpoint. But when I went to the person who was in charge of coding that and said, what do I not understand? Why does, when I look at this tool, why does it look like this is a bottleneck? And nobody had applied that tool to that part of the system and they couldn't give me an answer, oh, that's necessary because of licensing or because, you know, we need to preserve the data, they looked at it and they were able to improve the efficiency just because somebody asked a question. And there's always a cost. Everything about DevOps is about reducing the Infrastructure, cost of having all the servers, about having to host everything, going to SaaS and be able to spin things up, is all about reducing cost and improving reliability. It can be difficult to shift the design of that stuff ahead of time. But performance testing and exploratory performance testing can be a great place to start building a rapport with the rest of the team. Not everybody's going to have access to the systems and to the tools, but if there is an opportunity to, to start exploring the performance of things, it almost always yields good relationships and starts good conversations.

So it's an interesting term you use, exploratory performance testing. That's something I think people think about when they think of exploratory testing. Can you expand a little bit on it? What are your views on what is exploratory performance testing?

I don't know that it's an industry thing, but for me, the whole point of automation and having more efficient reusable testing code is to then use your resources and your knowledge of the system to go and find areas that you're not testing yet, or areas that things could come together in dynamic situations and find out how they are going to react and interact. So once you have your baseline set up and you've got your core set of like regression type testing, we need to make sure we can do what we know we need to do. That's the opportunity to look at the plans and say, well, okay, we're going to open up in a new country and that country has different interests. They are more interested in their teeth than their fingernails. So let's test the teeth part of our system, heavier than we currently do, and find out if there's any limitations there. So that can be stress testing to find out which parts of your system individually as you ramp them up to 110, 150, 200% of just that portion of your system's load. How does that impact everything else? Maybe you meet your SLA for that individual subsystem, but maybe it taxes another backend system enough that other parts of the system start suffering, they start queuing, they start receiving more errors. And so if you wait until you have a new requirement to explore how your system interacts, then you're going to be playing catch up and you're going to be testing in production, you're going to be responding and it's much more costly. But if you are investing in your testing process that you're testing sooner, so you're automating and running the things that you got to run all the time, run them automatically, run them overnight so that when you come in the morning, number one, you've got. You have test results to know if things are okay. And number two, if everything is okay, you spend your time finding new design opportunities or new demand opportunities, either based on the modeling that's coming out and the planning, or by paying attention to the production information and saying, you know what I'm Every time we cross a time zone or every time daylight savings comes around, we have a hiccup and we have a surge in our problems, in our customer responses. Let's go explore that. Let's see if we can recreate that situation and learn more about it and give some feedback to the design team to accommodate it.

Nice. So Matthew, one thing that came up during your session at Perf Guild this year was mocking and mocking in relation to performance testing. And I'm wondering, can mocking help at all in this early stage of exploratory performance testing? Because I know a lot of times people say, well, I need to wait until I have a complete system before I could do any sort of performance. Right?

So mocking helps, in my opinion, three ways. One, it lets you test interdependent systems sooner and not have to wait for one component, a key component, to get the whole thing going or to get the full picture. Number two, it allows you to take out costly things to test. A third party API or things that take a lot of time. And if you can virtualize them because you don't need to wait for that time, you don't benefit from that time, then you can get your testing done faster. But the third thing that mock services allows you to do is look at your production trends. Look at either anomalies that you can't explain or a direction that you know you're leading towards more usage, more demand, a shift in which part of your system people care about and you can take that trend and move and look forward to what it is likely to look like. Use mock services to change what that performance looks like, to anticipate it and find out the impact on the rest of your system to either document changes in your SLAs. You know, we're getting bigger, things take longer. So we just do want to let you know that over the next month you might see these things take a little longer or more appropriately, more hopefully to begin to engineer to handle those causes. And mock services takes the guesswork out. It lets you actually create the scenario that you need to anticipate to be able to test it more accurately.

Nice and I think another thing people struggle with, this may be controversial. When I started performance testing, it was early, you know, late 1999, 2000, and luckily the company I had had enterprise licenses for LoadRunner and LoadRunner is a complete performance solution. And then open source started coming on. I looked at JMeter, I was like, this is crazy, I don't understand it. But I believe you have a solution that helps make JMeter easier and I think might help encourage this earlier type of testing. It's called Torres. So when you said you found a tool, that blazemeter has a tool, is Torres what you're talking about for open source?

Yeah. So Torus is the engine behind blazemeter. Blazemeter is really an encompassing platform that uses all open source tools to accomplish that goal. Taurus itself is command line only. And so back in the day of development, anybody who used a GUI type IDE was kind of more beginner or needed help. And the guys who could do it all by command line were considered more the experts and the hardcore, you know, grizzly atoms of coding. So Torus has that element and that makes it very efficient. If you think of regular performance testing where you're generating network load against the backend versus trying to load an actual gui, the overhead of loading a browser and rendering things is both very costly. It takes up time and it's hard to get enough of that to hit your backend system. So there's that knowledge that running a GUI is costly. Well, that cost is also valuable. It's good to see that. But when you need to get things done efficiently, it's a lot. It can be a big benefit to be able to just kick things off. Command Line is easier to automate. It makes it easier to go across platforms and that kind of thing. But then to have a powerful GUI that will make that easy to understand, easy to share, easy to compare, is kind of the best of both worlds. So I think blazemeter did a great job embracing both of those aspects.

How does it allow you to write tests, though? Do you use YAML? Do you use like a BDD like syntax on the Torus piece?

Torus is YAML based. And what blazemeter does is it creates a YAML wrapper file that defines the test and is read by Torus at the YAML level. And then it refers to the 21 other testing types, whether it's postman, JMeter, selenium, those things can run in their native state. And Torus simply provides kind of like the command line arguments, sets up the environment, the number of test Users throughput any destination test engines. All of that can be encapsulated in the YAML easy to read format, which is kind of becoming a little bit more of a common industry standard. Almost everybody when they look at YAML can understand what it's saying. And a lot of particularly functional testing is embracing Selenium and is using YAML to define things. And when you define your own testing architecture and framework, the framework itself is often defined in YAML. So it's kind of become a common language and a common interface. Even when you are running Junit or running Selenium separate from each other. I think that having Taurus be based in that offers an opportunity to collaborate between the fiefdoms of the other languages.

So interesting. When I started once again it was all protocol level, but it seems like a lot of people are using Selenium also to help them with performance testing. Is this a shift you've seen as well?

Yeah, I think it's kind of like how Linux unified Unix, right? We do still have different forms of Linux and there is still the other forms of Unix, but Linux has kind of bridged a gap and the commonalities between Linux and then let's say the addition of like Docker things are going more towards the unification rather than the segregation. And I think YAML is definitely following that trend and capitalizing on that historical success. And it seems to me that I'm finding more and more places that are open to it than places said oh, if it's based on YAML, we're not going to do it.

So besides Torres, are there any other solutions in Blazor Meter, that open source that can help folks, I believe you have a recorder as well.

So we have proxy recorder and we have a browser plugin recorder. And then because Blazemeter is very tightly tied to JMeter in terms of the automatic setup and things that we make easy for you. JMeter also has a recorder that has some advantages. So there are those three levels and then there's kind of an extra level when we talk about running not the SAs version of Blazemeter, but we talk about having it in behind your cloud. We have an on premise recorder that's basically just an instantiated version of the SAS Chrome Recorder, at least three different ways that you can capitalize on the platform's recorders.

So you mentioned about this unification of technologies. Another trend I see is more and more companies embracing cloud based providers. Aws, Azure, Google Cloud is any tips for performance testing that like maybe someone that's used to client Server based performance testing now needs to test in the cloud. Any concerns or things they need to be aware of?

Yeah, cloud network segmentation and security is both more flexible than traditional networks and it can be a little bit more complex. So I definitely recommend that people take whatever free level of training or reimbursed level of training they can take for the cloud that they're going towards. For me, that was AWS and the first DevOps company like, and I mean like fully embracing DevOps company that I joined had an optional 23 week AWS course. And people, you know, they had reserved times for people to get together and work together or work quietly in a protected environment where no one's going to come tap them on the shoulder. And it can be involved, but it doesn't necessarily have to be. You can at least go through the, you know, networking for dummies levels of things and ask the people who are setting up your security protocols and procedures, what are the gotchas, you know, what am I allowed to do? What do I need permission to do? And for those things that you have permission to do, go see what you can learn about that. You know, are you allowed to allocate compute engines? Are you allowed to create subnets? Because if you are then going and setting up engines and installing software on them and being able to turn them on, off more cost effectively will very likely open up the doors for you to explore more tools and more opportunities because it doesn't require as much of a request process.

So another thing that comes into play I think is more people using microservices and APIs. So is that a shift you've seen also with performance testing is more than just HTTP protocol, people also starting to utilize APIs for the performance testing and their setup type steps?

Yeah, we've always written APIs. We just called them procedures or subtasks or functions. But we've always had APIs. We just would protect them. Particularly in C and C and Java we'd have these protected classes, but internally we were calling them. What we've learned as an industry is a lot of times the benefits of the security and the encapsulation don't outweigh the opportunities that are available to open those up and let other developers, other parts of the system, access them more directly and make our system operate more efficiently. So opening up the API opportunities and portals has made interoperability more appealing and easier to do. The risks did not outweigh it. You know, there was always this opportunity that something was going to go crazy and call an API too much and bring the system down. But it seems that the industry has mitigated that amongst all the different ways to implement it. And the standard of publishing the API is really at design time, specifying what that interaction is going to be, what limitations there could be, has taken care of that problem. So now the idea of being able to test other people's APIs, in particular, we talked about the mock services that if an important functionality is going to be key to multiple parts of development, standing up a mock service of that API so that people can interact with it more naturally as they're coding their own things, has made code more secure and more efficient. So I think that shift has been very productive for the industry.

So along with that shift, a lot of times now with monitoring in production, a lot of Times People use APIs to make sure that it's still performing over time. Is that a thing you see? And also, does Blazemeter have an option to help you monitor APIs?

I certainly see that in advanced groups, groups that are doing more of a shift left and trying to cut down the cost and time of mitigating problems to see in production blazemeter. And really this Broadcom gets a lot of credit for this. When Broadcom took over CA Technologies and looked over the tool set and evaluated where BlazeMeter was going to fit in that tool set, BlazeMeter has become the hub of the rest of the ESD, the electronic software development platform that all of the other components are being brought into the blazemeter testing platform. So the API monitoring, which is geared not towards testing the development APIs, but monitoring the production APIs, brings in the dashboard to be able to see at the testing level if you're starting there. But if you're shifting left and your developers are using the platform as well, the APIs that you are routinely updating and paying attention to, you just click another tab in the same tool and you can monitor how those APIs are actually doing in production. And then conversely, you can be notified in the same tool when those APIs are starting to approach your concern points. And it doesn't have to be the production level concern point. It can be at the developer level. Each developer can have their own perspective of what they consider important and they want to track. So blazemeter has definitely taken advantage of that and made that a key part of the shifting left process. And then to be able to reuse API testing artifacts with your API monitoring allows you to deduplicate your efforts and be able to say, well, okay, this got me through my design and testing process. I want to slide it over and see how it does in production. And then because it went one way, it'll easily go the other way. When you see it happening in production, it's easier to come right back into the testing process and start working on it when you see a problem.

Great. So Matt, I like to keep these to about a half hour right at the half hour mark. Right now I have one last question I want to ask. But before I ask that question, is anything you definitely want me to touch on before I close my last question?

Sure. Blazemeter has shifted itself towards large scale renovations and industrial revolutions. And we do still offer our free tool. There are a lot of other good tools out there and with the idea of open source and the idea of the industry helping each other out, don't feel like you need to pick one. You know, one of the biggest changes in my career was when I was asked to evaluate and recommend different tools. And it opens your mind and you might need to take some extra time. But think of it like, you know, if you're getting a master's degree at night, you're going to put some extra time in. I really encourage people to find a curiosity in this field. Find some tools, evaluate them, read about them, and blazemeter can help you on that way. It's probably not going to be the right fit for everybody, but that's okay. And as people find the tool that does fit you, be comfortable without growing that as time changes and eventually blazemeter might be the thing that helps you. And if that's the case, give me a call.

Okay, Matthew, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Sure, I think people should find what they're curious about, what they either have a natural gifting for feeling out. Are you the guy who always crashes the system? Are you the person who's always asking the question that nobody understands and then a few months later it becomes relevant? If that's who you are, find ways to explore that. There's open source tools, there's open source forums. Give yourself some extra time, both in your job, to the extent that your job condones that, and some time outside of your job. I think a lot of technical people don't want to too buried in the rabbit hole and they don't want to bring their work home with them. But find a balance and explore performance testing and performance testing tools and trends. Find your curiosity, see if you can go to well right now virtual forums and sessions and meetups and whatnot. But invest in it a bit and be okay with the idea that the things that you used to know might be changing. The things you invest in now are most likely going to change. And I think embracing that and having like an 8020 perspective, spend 80% of your time working on what you are doing and and where you are going, and 20% of your time exploring where things could change. I think that's a healthy perspective. And if blazemeter isn't in your organization's roadmap or you don't fit into that, try some of the other tools. And when blazemeter becomes the solution that fits the size of where you're at, give me a call. You can email me Matthew with two T's Andrus a n--rusrodcom.com or hit me up on LinkedIn. Several of the people from Perf Guild have been communicating by there and I'm always happy to help.

Thank you Matthew for your performance Testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P43. And while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about Smart Bears performance test tools like LoadUI Pro and LoadNinja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.