WebLoad vs LoadRunner vs jMeter (Who Wins?) with Paul Kanaris

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Are you struggling with JMeter's limitations and Loadrunner's overwhelming complexity and price? Wonder if there's a better way? Well, today we're going to break down how webload strikes the perfect balance between ease of use and robust performance testing at an affordable cost. So don't miss out on transforming your testing strategy. In this episode, performance testing guru Paul Kinieris, with over two decades of experience, shares insights on selecting the right performance testing tool and ways to optimize your performance engineering efforts. And if you're looking to move away from JMeter or Loadrunner, definitely try it for yourself. Head on over to TestGuild me LoadTrial and follow along and see how it can help you elevate your performance testing efforts. Don't wanna miss it. Check it out. Hey Paul, welcome to the Guild.

Nice to meet you, Joe.

Great to have you. I guess before we get into maybe a little background, I know you have a lot of experience, but for the folks that don't know a little bit.

More about yourself, I've been involved in software testing for over 20 years, claim that I can do all things testing. I've done functional testing, manually automated. Started out with the Mercury suite of tools, implemented Test Director, Quality Center, ALM, LoadRunner 6.0. So that'll kind of give us a reference of how long it's been. Worked a lot with Loadrunner over the years. As we know, Loadrunner is like the 10,000 pound gorilla that does everything performance testing, but it's still the gold standard. So that's where part of that comes from. I've been an instructor for performance testing as well as the testing tools. I am ISTQB certified, the foundation level as well as the test manager and my current role. I actually lead a global QA team of about 73 people spread around the world. We've got a small contingence for performance testing right now that we're hoping to grow over the next few years.

Nice. So I guess you know, you mentioned Loadrunner and I started off with Lone Runner myself as well, but I've been investigating more and more web load, so I'm just curious to get your idea of, you know, I believe in your current position. Have you made a switch from Loader Load Runner to Webload and if so.

Why Actually we went from what the company was using was JMeter. I looked at lode runner but of course loadrunner, hate to say it, it's still a very pricey set of tools which makes it very difficult to get a company to buy into. But JMeter of course is not a tool. If you're looking at end user experience, many people go, but JMeter's out there, everybody uses it. I always say use the right tool for the right job. If I want to know how this is going to work with thousands of users, they're not interested in how fast the API works. They want to know when can I do my job? So that was one of the things that we did. And basically when I looked at Web Load here is I brought up the Load Runner documentation as a reference to say if it doesn't do these things, it's probably not a tool to use. So looked at something that people think is a very good tool and that was something that I encourage people. Then you take that, you bring that in and then you start going through your checklist of what will this tool do, does it meet that? Then it starts becoming down to if you can get a tool that beats 80, 90% of what's there. I also consider things like how extensible is it? If it's not extensible, how good is the company at what I might need to have added or not. And that's one of the things that I've learned really starts helping you understand what you can do. So that's kind of what I did when I came on board. Now of course we looked at the cost and you need to do a good ROI and you need to start looking at though even if you can buy a very good tool, it may not be as easy to implement or as fast as you want. So don't over promise.

So I think you're right about JMeter. A lot of times people say oh it's free. But you know, from my experience using something like Webload is full functional where JMeter is very, it's not enterprise wide that I'm aware of that can handle what I'm used to with a tool like webload.

Right. JMeter is very good at doing checks on very small activities. It's when you start getting into the more complex like we're a supply chain management system, you're starting to talk about some very big amounts of data that need to be moved around. And again I go back to what am I trying to do JMeter was designed literally to say, are my back end calls working? It was not designed to say how long does that presentation layer take? How much time is that consuming? Where am I really seeing the impact so that I don't get any of those nice phone calls at 2 o' clock in the morning from my CTO screaming that the system is going so slow that he's got the CEOs all screaming at him. So this is what you have to really start looking at is understanding what the tools do, what your goals are versus their limitations. So when you go out there you'll hear, well, you can embed Selenium. Well that's fine. It's still not checking the return and the responses of thousands of users. Whereas with Webload you've got virtual user. These are real virtual users. That's why you can only add about 150 per generator because they start consuming resources. Whereas with JMeter you can spin up 2000 users on one box. It doesn't use that much memory. That would be a big savings. People say, well okay, fine, what am I going to get from it? And that's where I keep stepping back. The other part is how much do I need to do in JMeter to start extending, to connect to things like we're going to be hooking CloudWatch in from AWS because everything we do is up in the cloud. Well, if I've got to do that in JMeter, I either need to rely on somebody building that as a plugin or figure out how to do it myself. When I get something like webload or other third party tools that are more industry based, I can get that already and therefore I save a lot of money and time where I'm not trying to figure it out. Performance engineering shouldn't be focused on building tools and I call it performance engineering, not testing. People don't understand the difference, but one is running scripts to run scripts. Engineering is developing that test, developing the methodologies and understanding what you're getting out of it so that you can come back and say, based on what I've executed, here are my potential bottlenecks that I see and here's where I think we need to start looking and also adjusting those tests. How do I find out if the background servers are chewing up my system or not once I do that, what do all these metrics mean? That's the engineering aspect. If you can't explain what you're really coming out with, you're not going to get anywhere. And that can be really hard with something like JMeter and that's where Loadrunner had a lot of advantages and that's where webload steps up to the plate. I get a lot of data, I can look at it, I can figure out where I can be, I can build new graphs and I can do some things like overlaying my transactions against my server response times. I can go ahead and see those spikes and start looking at what's my relationship. And I'm doing this real time. One of the problems with JMeter is yes, you can bring it together, you can try to pull it all together, but you've got to actually get that data from the server, bring it in and hope the timestamps are right. Whereas a third party tool, if it's designed correctly, it should bring that data in to the data at the same time. So now when I'm looking at that spike at X milliseconds of the run, I should be able to go check my resources and say, where's my problem? When you've got a large system like the company I have and most companies that will benefit from something like Webload, I've got about 45 servers, I've got to keep track of each doing different things. I've got my application layers, I've got my database, I've got elasticsearch servers. All these things are interacting. So where's your bottleneck? Well, if you can't precisely gather that and you can imagine bringing in 49 servers worth of metrics, if you had to pull those from each server, bring them in, it would be extremely challenging. Whereas Webload, it's all pulled in. I can look at what I want, I can remove variables out of there, I can remove measurements and then I can really look at, start focusing what do I think is my problem and present those findings clearly to my development team.

I love that. And like you said, a lot of times roi, people hate hearing that term, but a lot of times people see a free tool and they think it's free. But I think you laid out a lot of good points. Why a free tool like JMeter isn't really going to save you money in the long run at an enterprise level is what I'm hearing.

Most tools that I see that are free, people have to reflect on and see it's not free. There's a lot of overhead. If you want to do something with performance engineering, you got to build a lot of infrastructure around JMETER in order to get it to work within your environment. Now if you're working with some strange environments and I'm working with aws, I could be working with Azure. Webload supports both. I can go ahead and work with the cloud. It's now got some artificial intelligence which I've got to work with them a little bit on. They decided to go with some of the open frameworks out there. Going to have to work with them a little bit on how we get with a restricted cloud. Because that is one of the things that you've got to be looking at too with the tools is what if I have some regulations such as GDP in a lot of other restrictions, how do I use a tool that's open source, how do I get them, the information that I need and even the tools that I do pick, how do I make sure that it's compliant? So that's going to be another layer that you've got to add on top of that. And how do you keep all those libraries in sync? One of the things I like is if I don't have to do all this heavy lifting of building some extra extensions, then I don't have to go ahead and maintain all the security potential risk because all of our stuff is actually scanned, make sure that we're not bleeding anything out to the Internet. So if I was actually doing that all I'd have to have them look at all my libraries, all the stuff I'm compiling. Whereas what we get from someone like Webload, we've already got a security certificate. They don't need to go ahead and start digging into the tool to figure out if I have some vulnerabilities. And I'll say that because I have an open source framework, something that I was kind of forced to build, I would prefer something different. And it has cost us a lot of money and a lot of heartache keeping track of those libraries. Keeping track too of building all that code, building all the integrations. Think about that. When you're dealing with performance engineering, the more things that you have to build to complete your job, the less time you have to do the job. So therefore is the ROI really saving you money? And I have to look at a tool like webload and it was easy for me to do it. It costs a lot less for me to have Webload on my annual purchase than for me to spend my salary plus some others to build all the stuff I would need to with JMeter.

All right, so in this round I'd say webload versus an open source solution. This round goes to Webload. So round two, I'm going to call Webload versus say Loadrunner. We talked a little about Loadrunner already. So you came into the environment, they had JMeter, and I assume you started exploring Loadrun and other solutions and you've used Loadrunner before. And I'll be honest, I haven't used Loadrunner Hardcore since 2006, so I don't know if it's changed at all. So is there any functionality or pain points that you had in loadrunner that when you looked at something like Webload, you said, oh, wow, Webload actually takes care of these pain points as well?

Some of the things that I looked at was, as an example, Webload, you can bring up their dashboard in the browser and you can actually open that up to where others can be looking at it. You can only run it from the original console, but others can be looking at it. Well, if you want that in loadrunner, and I looked at Lode Runner and I actually used loadrunner probably about six, seven years ago, you had to buy that it was an add on. You had to buy different. What type of thing are you testing? You got to buy each of the pieces. Now, of course, you could buy the big bulk package that gives you all the different protocols, but then you spend a lot of money. That's a problem. When I start looking at correlations, are they easier in Webload than loadrunner? I'd have to probably say that's why I like what Webload does. A lot of the features are comparable. So when I start looking at features, the majority of things that I did in Webload, they worked pretty good. But of course I didn't dig into everything that Webload could do because if I didn't need it, why worry about it? Same thing with Webload. I'm sure there's a lot of things I don't need for the current testing project, but those things that you need from a user perspective for performance engineering in general, I can say you're going to get complimentary features. I do think that there's a much cleaner look to web load. As far as the browser piece, as far as the Council, I could tell you, the council in Load Runner, they spent a lot of money building that because originally that was the only way you could see things. So I would have to say the Council was in some ways a little bit nicer. I could bring up more metrics on it. I could make it look like I was basically being an air traffic controller with all the things I could bring up. That doesn't quite happen in Webload, but do I need it? And I think that is what you got to look at is they're nice things, but are they things I had to have or was it really more along the lines? It helped me while it was live to try to do some preliminary analysis because that's what I needed to do for that job. So I would say you've got a pretty consistent one to one. At least Webload does the same thing as Loadrunner. And then I think the browser piece really starts opening up the doors where you can start sharing and building collaboration because the more you can build that collaboration where an architect, especially with my world, it's global. I need to have my architect be able to look at that. In matter of fact, my architect I don't think even lives in the same city I do. So this is really becoming the world that we need to have those tools built in with collaboration and we shouldn't have to be buying extra plugins for collaboration. So I think that's one that really kind of sets Webload aside.

All right, so this I'm having flashbacks because every time I needed to do something at my job I used to go, oh, it's not available, I need to pay to unlock this. Or so it was crazy. And it doesn't sound like Webload has that, which sounds great because you can get your job done saying oh and by the way, we need more budget to buy this other protocol or other add on.

I haven't run into that and it's just like. And I had looked at Web Load with my previous company, they just decided not to. So that was why I was already familiar with Webload when I came to this position. They built things for me that I needed at the previous company and we've had to make some adjustments and I can work with their support people and I even actually work with their develop one of their major developers because most of the stuff I get into, it's not that easy because I've been doing it long enough. I can figure most of the ways around things. So when I get stumped, it usually takes some help. They're there, they're not asking, they're not holding their hand out for more money. They're not saying that's going to be special support. You got to pay X dollars. I had that with web, I should say with loadrunner where they would say, well we can probably do that for you if you, you're willing to pay us. That's not what I experienced with Webload. Now I'm going to ask for like I say, the private AI stuff. They may have to hold off on that because they may not be ready for it or they may say no, and that's okay. But I believe if you can justify it with them, they'll do things. And that's why you don't want to go ahead though and say, I want to work with this special unique little AI tool. They probably will tell you no for sure. What I'm looking at is Azure. That's probably something that they would go with. The only reason why I'm doing that is I've talked with my company and with my automation and my test management tool, they want to implement AI, I've got to use Azure. I'm like, well, if that answers my AI, then I'm going to try to do the same with Webload and I'll talk to them. So I think that's one of the big things, Joey, that you bring out really shows what you want with the tool. Are you paying for every little thing or are you paying to get a tool that does what you need? But also I think the other thing that you got to look at is what is the support like, can you get help? I remember Loadrunner a lot of times I could wait a couple weeks before I got a hold of people and when I first used Loadrunner I was working for a major health company, the largest in the United States, and I had to wait. So I hear you there and I.

Think that's one of the stellar features of Webload and radview from what I hear is their help is top notch and like you said, they don't nickel and dime you. They're willing to work with you right away, which in my experience has been when you're doing performance testing, like you said, you never know you're going to run into. And having that type of help desk or that helps support really is a big deal.

And we've got not just one product. I've actually got about 14 different products that we're going to be bringing over and these are all from different companies that were consumed or purchased. So you can probably understand I've got some that are on DB2, some that are on Oracle, we've got some on MySQL. We got every flavor of databases as the back end, every flavor of application server. We've also got some very unique. As I'm sure you're used to, Joe, developers can be rather creative in how they build things. So having the ability to have a tool that is flexible, but also you're going to run into something, it could be the way that they're returning information you're trying to get the data out so you can feed the next screen or maybe you're trying to do some changing and you just can't find it. I've had that problem and I can submit a ticket and they will get someone online with me if necessary. And that's where I've worked with their major developer because we found out it wasn't that easy. Well, he helped me and within probably 10 minutes he was like, oh yeah, just do this and this. And I'm kind of going, who would think of that? And that's the kind of help you want?

Yeah. And I think it also outlines, you know, enterprise, the need for an enterprise wide solution. I'm used to working at health care companies, banks, you know, financial insurance and you never know what you're dealing, what protocol is going to pop up, what back end you're dealing with. Like you said, what the developer is coded up. So it is a big help to have that help there to be able to get through those type of issues. For sure.

Yes, that's probably the biggest thing when you're at the enterprise level. When you're with a small company you usually can adjust your application to work with the tool. Instead what you want is a tool that will adapt to your environment. And when you've got an environment that eventually will probably be because we're building a consolidated system. So we're going to be looking at more like about 150, 200 systems all talking together and that will be a very massive load. And we've got a lot of unique constraints which you have in the enterprise. We've got gdp, so we got those regulations, we have the regulations here in the states and we have to be very careful because we also as supply chain management support some more unique companies that deliver things to the government that have security clearances. So we've got the whole gambit of all the challenges you can have from a customer base. But remember those customers are the ones that are going to drive what you need to do. So each of those opens another potential risk that that tool will not do what you need. The best mitigation I can tell you is have a tool that is either adaptable for it or a company that is willing to help adapt their tool and. Or adapt the tool.

I totally agree. I hate using a tool. And you have to go to their what they recommend, their format or their flow. It really will mess things up if you have to conform to their, the way they do things rather than it conforming to way you do things. So I love this.

So, and I found Webload does plenty of that. You can use it however you need. We've got some very unique ways that our current, the main product we're working on, which is one of our biggest, the way it pulls data back. Every company is unique. So even though you're searching for, in our case a product, every company brings back a different way way, a different payload, everything. So this is all the things that you're going to want to think about when you're looking at what tool do I want. But I do strongly suggest that if you are looking at tools, go with the one that you know is the best and say, okay, so what do they offer? And then start looking around and say what tools can I get that will offer that and start cutting down your list.

So another thing you mentioned, once again, you're in enterprise, you never know what you're getting back. What your developers are doing is the big thing about correlation. And always when I was a performance engineer, correlation was always something that kind of could struggle with, depending on what you were testing. And from what I hear from Webload is the correlation engine is really superb. So do you have any experience between Webload's correlation compared to loadrunner and which one you prefer?

I actually think the web load correlation was a lot easier once you understood how it was set up. Coming from Loadrunner, I had one preconceived notion about how it was set up. And where I learned it from though, of course was healthcare. So in healthcare we had to do a lot of things with using that correlation to get information out of the headers and the streams of data that really made it unique. Now most of the time when I worked with other products after that, it was pretty easy, pretty straightforward. Coming now back to a supply chain management where we've got a lot of data, we've got a lot of customization, that correlation again has become a little more complex. So when I looked at it, Webload is easier now. There were some things that I've needed some help with, but there again, that's where it's great. If you follow everything they tell you and you still can't get somewhere. I could make a phone call, I can go ahead and send an email, might go through the support desk, then they get someone on board, but it isn't two weeks from now, it's usually either the next day they'll send me a snippet to try. And if that doesn't work, usually we go through a couple tries with that and then we get the appointment set up and we're real time. They're on my system, we're figuring it out. So, yes, the correlation tool is a lot better. And I think then you've got to look at too, though the tool can only do so much of that correlation and sometimes you might just need to find out that there's a syntax that you wouldn't think of and they didn't think to put it in their document because it's there, but it's just not something that you think about. It's kind of like sometimes people ask me about testing certain questions and I don't even remember. But then when they ask it a different way or different context, you remember. And I think it's the same way for the tool vendor is they try to cover everything you need, but they don't. They can't always put it in the document.

Also, I think that's flexibility where you can create your own using code. So, you know, I know what loadrunner back in the day was C. I assume it's still C. And webload is on JavaScript, which I think more developers know and you probably get a lot more help with. So do you see that as an advantage as well? Being using JavaScript over C has been able to make you more flexible or get more help around coding out of the things that are available out of the box?

Probably where I don't pick up on that advantage is I'm also the automation architect, so I write all the code over there. So.

When you ask that question, Joe, I have to go, do I get help? The answer is no. So how much can I pull from? Because we do that all in Java. Groovy. Anyways, so JavaScript does give you a lot of the leverages. And if you go look at the JavaScript documentation, that shows you a lot more information. So by going with JavaScript rather than something C like, because remember, it wasn't really C, so you couldn't really use C either. They didn't go either go C or not. They kind of created their own hodgepodge. This advantage with JavaScript is you can go find the entire JavaScript documentation that'll show you how to use write expressions. And there's a lot of information out there about how JavaScript works that's going to give you the leverage that you can do a lot of those correlations without having to ask someone to help you, because it's there. It's just you need to have a resource that you can access. So I think that that is a Big advantage that they are actually implementing a language that is industry rather than proprietary. And that's probably another good thing. Joe, to add to your checklist, what are they offering in the way of the language? Is it something industry standard or are you going to have to go to their school and learn it? You can always go ahead and do what people did when they wanted to learn Load Runner. They got to come spend a week with me as an instructor and I would show them how to use loadrunner. I have not had to go to Radview to learn webload.

Right, that's true. That's a good point. So another thing you mentioned, I think I glossed over. We shouldn't have glossed over. I always think of performance engineering as a team sport where I used to have my database guy on the mainframe person on looking as I was running the load test. But you mentioned something and I'd have to tell them, hey, I'm seeing something here. Are you seeing it? You mentioned something about webload being more collaborative and the fact that you have a browser front end that the whole team can monitor while you're running the test so they're not all waiting or huddling around your cube. So that sounds like a big benefit as well.

Yes. Being a global company, you really need to build that collaboration a little bit different. One way would be that I had done in the past would be to use something like teams and share my screen. Well, that can cause some limitations, especially when you start getting into bandwidth. This way you can go ahead and set it up where they can actually go view it. They can see that live run. We can have dashboards that are built, they can go back and forth between dashboards. They don't need to say, hey Paul, I see what you're talking about, but can you show me now the system resources? Now that I see what the database is doing, can you show me the app store? They can immediately go through those dashboards we've built and look at them real time. They can bounce through them, they can go ahead and start doing some of their own understanding and if necessary, we can even build specific views for different people. Now that's going to be something long term. We're going to look into. The other part is the way that we've set this up is we're setting it up to where our performance engineering team will be setting up all the tests. But once they're set up and we have a full regression, we should be able to have those executed by others. So you can actually use that Browser front end to give them permissions. Now, if you want to start running 5, 6 performance tests at a time, yes, you do need to buy some extra console license because you'll need them. Does it only lets you run one at a time, but I think that's normally what you would do. Now, huge massive companies like an IBM or someone, they would probably need multiples of those. I still think they could save money and also benefit from the tool as far as its flexibility and the way that it works. Like you mentioned with the JavaScript, it can reduce that specialization. Because I know when I wanted loadrunner I had to find people and they weren't necessarily as cost effective as if I needed someone that understood JavaScript.

That's a great point. That's a big point because I used to have to hire people that knew Selenium, that knew how to do Java and how to do automation and Java is really hard to find. It's almost like a unicorn kind of. Same thing, I guess with performance engineering, right?

Yes. If you want those things, you got to spend a lot more money and that was one of the things that made loadrunner a niche. If you knew how to really use it, you could make a lot of money if you wanted. If you didn't get to do that as a profession, but you were trying to do that as part of your job, it was next to impossible. Whereas something like Webload, I can tell you I've got two contractors that do my scripting. I actually do all the web load running and I also manage, I have six managers that report to me. So you can see a lot of things that are going on. So if you think about that from a QA perspective, not having to immediately dedicate that resource when you start now, what this is proving is it works. We're testing a major database conversion, so it is proving out its point. We've already found a number of performance issues. When we went from one database to another, and this is talking going from DB2 to Postgres, we found those performance issues. They've been identified. That makes a big savings. But we have not had to dedicate a performance engineer now. Now once we get ready to move to the next phase, we've proved out its value. Yes, we need a performance engineering team because now you need people managing a much larger work effort. We're going to be bringing in 10 different products that all need performance testing by themselves and together. So this gets you in the door to start, but it doesn't go away just because now you've went Ahead and improved. You need performance testing. You can now take Webload and keep building it. You can add more generators if you need to. You can say, I need a couple more councils. All this is extensible as you grow. And I think that's another thing that you want to be able to say. Plus you can keep all your results as long as you're willing to spend the money. You can store whatever you want. Now you can also export because we all know with load testing you run a lot of tests that you throw away. Well, I probably want to keep all my results that matter, so I would want to pull those out and I can archive them because I can actually create a zip file of everything from each run that I want to keep for save my release report. I can have six or seven runs, put it all together, say put that in the X release folder. That becomes part of my artifacts. Or what we're getting ready to do is connected to our test management system that actually is a partner with Webload and figure out how to put the results up directly. So this is some of the things you want to think about too is does that tool integrate with other people? Because that may give you another reason why you want this tool over that tool.

Great point. Yeah, because integration capabilities is huge and it sounds like Webload has it. But I don't want once again gloss over this point and it almost sounds like in your experience Webload saved you money because it kept headcount down. I'm not saying it replaces performance engineers, but it seems like it helps alleviate the problem of having to have niche. People that know it just know a tool and that's why you need to spend the money. I know. If that makes sense.

Yes. When I first came on board with Loadrunner many years ago, we had one person that just did scripting, another that knew all about how to create data, another that knew how to do some sort of analysis and understanding of the data. We had to have multiple people and usually the other people didn't have any idea how to do the scripting. Now will we still need someone dedicated to data and how to do data creation? Yes, but I think that they can also have part of their job responsibilities. Beta assist with that scripting because it is JavaScript. So I can now have backups rather than having to have duplicates of say the load runner scripter. Now I can have my database person who, yeah, they can't do the high end, all brand new stuff, but if there's something that goes wrong, they already know how to fix it they can help work on it. They can be the same ones that can run the test and be able to come up with some preliminary reviews. Now it may be that they can't do what I do. I can go look at it and I can start just by looking at it go. I think we need to go here and start drilling in. But that's based on 20 years experience. But this gives you more flexibility of how you can use your team and also another thing where you can locate your team. We're not restricted loadrunner, you had to have a license for the region that you were deploying in. I'm an international organization. I'm going to be testing products around the world. I also have staff that are around the world. I haven't moved any yet. Right now my contractors are in Bolivia. But I'm looking at long term being able to supplement this with people that are in other countries that have the skills I need. Because JavaScript is easier to find in some other countries than Lode Runner and I can actually have them working over there and then they can load it all up on the server which happens to be in our AWS region and everything runs from here. So there's a lot of advantages in, as you mentioned, the staffing. What can I do about it? And a lot of our staffing is going to not be because of the tool complexity. I told you, we're going to be testing 10 to 12 applications. All of it's got to be put together. You can imagine just the orchestration of all that is going to be huge. So you start getting away from the focus of the tool and get to the focus of how am I going to bring all this together to test number one and how am I going to get out of it meaningful data that we can take action on to either correct performance issues or mitigate the risk of performance issues. So there's a lot of things here that we can take and look at with Webload that are going to give you that advantage. The less time you have to spend either scripting or building things for the tool, the more time you have to focus on the job you're here for doing the testing or supporting that testing effort pricing.

Let's try to tackle this head on. I know Load Runner very expensive. I'm not telling you to give me a price as maybe give me like a percentage maybe of like how much of it cost to get loadrunner with all your everything you need to compare it to Web Loader. Is that even a valid question?

I will put it this way. I called it a gorilla. And of course it's also a gorilla. Takes a lot to feed and usually think about how much a gorilla weighs and it figured that out in gold, that is Load Runner. So if you want me to kind of give you an idea, you might be able to go look at a small chimp and put that on the scale for the gold price in comparison.

So Paul, you mentioned when you first got there you had a bunch of JMeter scripts. If someone has a bunch of Load Runner scripts and they want to really get rid of the gorilla and go down to the chimp size type, how hard is it to move from say other tools into say something like webload?

They've got to look at one thing. If you've got a huge investment in Load Runner, you've got to make a decision. What's the maintenance costing me in the staffing to me to continue to grow the gorilla and feed it? Because I am going to have to take whatever I've done and redo it. It's that simple. You can't take a Load Runner script and magically convert it to webload. Sorry, that doesn't out there. Anybody who's heard that you can go from VP script to Java and there's a magical tool out there, I will tell them I've got a very big bridge to sell. Just like going from DB2 to Postgres. All it is is take the data out and stick it in. Hate to tell you no, it doesn't work that way. So if you got a big amount of stuff to move, you have to really think about that. Is it worth the cost now from an ease of use you have to look at, you're probably going to reduce your working time from LoadRunner. The web load probably down about half the work and you don't need as high of a level of expertise. Those are things that I would need to consider if I had a huge accumulation. But I'd also have to say am I going to have to rerecord those anyways? Am I making enough changes in my technology? I know simply that I can probably keep my load runners, I should say my web load scripts around probably for maybe a year, 18 months and I got to redo them anyways because there's been so many changes in the product that they no longer work. They're not like you can do with automation where I can take things in and out easily. Load testing scripts of any kind are fragile. They go from start to finish and if you start trying to pull them out in the middle, they have a tendency to really do some bad things. So that's another way to look at it. If I'm going to change products, is there a way to maybe run them in parallel while I still have a license and then start moving? So if I'm going to do that, that would be my way of considering if I've got JMeter, it doesn't even do the same thing. You may as well move because now you're not getting the same thing that you're looking for. So again, this goes back to what are my requirements of my performance engineering? Do I just want to feel good the APIs work? Do I want to stay with Loadrunner because I've invested a lot of ton of money and I think that I can't get out of it, or do I want to move to a tool that gives me more flexibility and dramatically reduces the footprint cost? I think that's really what you got to think about that roi. We all hate those words. But of course, if you can't give them an roi, companies have a hard time paying for things. So if I can cover the loss, I'm going to take in a reasonable time and give them an roi, then I can do something. But if I can't, I'm going to have to step back and say, like it or not, I'm stuck with a load runner.

How easy is it to acquire Web Loader get started? Because I know with Mercury they made a pivot from talking to performance engineers like they want to talk to the cto and so they bypass you completely and sell huge enterprise. You never got to try it or even engage with it. How was your experience with Rad View and Webload when it came to that?

Easy. Go up and download the eval copy, Go have fun, go play. Probably the advantage I had, because I do talk the language and I do know what I'm looking for, is I could easily go ahead and answer all the questions that they started asking. Now also depends on your company and also what are you trying to buy? Now if I were trying to buy 15, 20,000 virtual users that they might wind up have to talk to someone. But I can tell you they really can't sell into the cto, they can't sell into the CEO. It's still going to take someone like me that speaks about the tool. So with Webload, if you're interested in trying to figure out if it's going to work, go try it. Go do a recording. But don't get disheartened if you do a recording and you can't get the correlation to work. Usually what they'll tell you is send us a support ticket. You might not get as quick of response as I do, but they will get back to you because they did with me when we were doing the eval. They'll get back and they'll say, hey, try this. Or they may even do a session with you, especially if you really show that you understand and that you know what you want. And I think that's really where they stand out too, is if you understand what you're looking for and you talk to them and communicate. Once you do the eval, I think you'll be able to get the answers that you want. And they also won't try to sell you something you don't need. And they'll let you start out where you want. So, like we have a set license right now, but if I need to up it, not a big deal, I make a quick call, I can get that done.

So, Paul, any other features that really made webload stand out that we didn't cover?

No, I think those are really the big things you want to look at. The ease of use, the protocols that it covers. I really haven't had a chance to look at their AI yet because like I say, our security people, I talked to them and they're like, no. So I had to hold off on that. We're looking at the cloud watch. We're that just came out. So we're getting ready to integrate that and that was part of our AWS move. So when I look at it, most of the features we've really covered that I can think of that most people would do, because when you start dealing with protocols, when you start dealing with your data, ease of being able to feed data is another big thing. Can you go ahead and use Excel CSV files? Yes, all those are there. How easy it is to add them. Very simple. So the data driving is probably another key feature that I would say the simplicity of it. And this gets back to really the focus of why I think webload is a good tool for most organizations that are trying to do this is that simplicity and ease of use. You're not have to hire the professional load runner engineer to run the tool. Now, it doesn't mean that you shouldn't hire a good load runner. Professional should say a load engineer professional, not load runner, but a performance engineer professional. They need to manage the process. You want to get people then that can focus on building out what you're trying to do. And how to do it best because they got a tool that they're not spending the time on. I think that's what really makes web load stand out, that I can do some of these high end things that loadrunner did that a lot of the other tools don't, but it's easier and my budget didn't take the big pinch to where I couldn't afford to do it.

So Paul, before we go, any parting words of wisdom? You want to leave the guild as it comes goes to around performance engineering and webload.

Well, with performance engineering is with anything. Make sure that before you start looking at tools, you understand what you're trying to accomplish. Make a list of what technologies you're going to support, what are the processes that you need to work on and then like I said, go out there and start looking at. Don't go looking just at JMeter and all the open source, say I've got an unlimited budget. What is the tool that I would want that'll do everything I want. Start there. Once you get what it will do, say do I need it all or what things don't I need? And then start looking for tools that will do based on that list what you need. But also be ready to look at it and say do I really need that or was it just I'd like it if something's missing in a feature because it could be you don't really use it or you really don't need it. Customers buy things all the time because they think it's neat. Make sure you're getting what you need that will make your testing job more efficient and that you can focus on testing. Don't buy a tool that you have to do a lot of heavy lifting to either understand make work, use day to day or you have to spend a lot of time extending QA performance. Engineers, all of us need to get over creating all these tools and instead focus on one thing. What am I trying to do to help my customers? And our customers are not the company. They're the people on the other side of the screen of the product that we evaluate. And we want to make sure that from a performance engineering perspective, they're getting what they need. Because when they get frustrated, they will stop buying your product. They may not do it directly, but if you start costing a company too much time, they're going to be complaining enough to their bosses that you can lose customers. So if you're spending a lot of time trying to do that, instead of actually doing the testing, you're building out a tool, you're probably not helping that customer.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com forward/p162 and while you're there, make sure to click on the Smart Bear link and learn all about Smart Bears awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com to so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and devotional. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation, software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experience experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding test, showing the secrets to behold.