Open Source Testing Survey with Kevin Dunne, Matthias Rapp, Ivan Vanderbyl

Welcome to the Test Guild Automation Podcast.

Where we all get together to learn.

More about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with the folks from Tricentis Test Project, specflow and Flood Communities all about the results from their largest testing survey of the year, the 2020 State of Open Source Testing Report. So we're joined today by Kevin Dunn, the general manager of Test project, Matthias Rapp, GM of SpecFlow, and Ivan Vanderbilt, who is the head of product@floodio. So get ready to learn all about all the open source testing tools and trends across the entire industry. You don't wanna miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. They're CL test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey guys, welcome to the Guild.

Thanks for having us, Joe. We're excited to be here.

Before we get into it, is there anything I missed in your bio that you want the Guild to know more about yourself? Kevin, let's start with you.

Yep. So this is Kevin Dunn. As you mentioned, Joe, I'm the general manager for Test Project. And Test project is Tricentis. Free platform for mobile, web and API functional automation that's built on top of Selenium and Appium.

Cool. And how about you, Matthias?

Hey.

Yeah, hey Joe, this is Matthias Rapp, GM of SpecFlow. Yeah, InspectFlow really is Tricentis BDD solution for any. Net developer natively to build BDD tests and BDD scenarios. So happy to be here.

Great to have you. And how about you, Ivan?

Hey, good morning. So, Ivan Vanderbil. I'm the head of product at Flood and former co founder of Flood. Before we joined Tricentis. I think the audience is probably pretty familiar with with Flood. I think we've been on your show once or twice before.

Absolutely. And you've been an early sponsor of my Perf Guild, which is a performance testing conference we're doing in June. So really excited to have you guys as the Platinum sponsor there. So thank you so much for your support.

Likewise. Thanks for having us here as well.

Before we actually dive down into the results of the survey, can you just tell us a little bit more about what is the survey, who you surveyed, how many people respond to that type of info. And Kevin, how about you kick it off?

Yeah, absolutely. So we set out here, we had a goal of hoping that we would get to 1000 responses to the survey. We ended up going way above and beyond that. We had 1,822 people respond to the survey. So really quite good participation. We owe that likely to just the general interest in open source as well as obviously the strength of bringing these three communities together here and were helped a bit obviously by everyone being at home and having more time to participate in a survey. So it was a really interesting survey demographics. I mean about half of the folks were QA or testing related positions and then the other half of folks were sort of developers, SDETs and other types of IT and development folks. So mostly testing focus obviously being a state of open source testing survey. We had tremendous participation from the community in Asia, specifically India. So we had over half of respondents coming from that geography, which was really interesting to see and certainly shows that the Asian community loves these open source tools and trends around them.

So actually there was a large number of respondents from the Asia area, I think like over 61%. So based on that amount of feedback you got from the folks in Asia, does that actually match up with the amount of downloads you get for your different products across the different regions?

See from the flood side we've got a lot of interest from, from India and Vietnam which I think is also reflective in the, you know, the skill set that you is very much fostered in that region.

Yeah, and that's very similar on the SpecFlow side as well, Joe. We have a really good amount of our users located in Asia and seems to be that their BDD especially is a pretty hot topic there right now. We have a lot of users also in India, so we do see that trend as well.

Very cool. How about you Kevin? Test project, is that highly downloaded in that area?

Yep, I think we see that as sort of the main country where we're getting the most downloads from. So it's been to have that community on board for this survey. And yeah, we're certainly looking to continue to support that community. We've got specifically a Tricentis user conference in India that we typically host in person, which is like the largest in person software testing conference in India and we're moving that online. So we're excited to see continued engagement with that community this year.

So besides regions, you also had a question about what are the most popular open source tool or what open source tools are you using? Were there any tools that bubbled up that you weren't aware of or you were kind of surprised by that people are using.

Yeah. So first in the introduction to the survey we had some questions about just general demographics and in that we included a question about which coding languages people preferred working with. Now this was a multiple selects, so it didn't mean you had to pick one. But just interesting to note, I think because of the fact that we focused our attention in terms of our community, mostly in Asia, we had a high Java response rate. Like 67% of people claim that they were using Java as one of the languages that they write code and write automation scripts in. Next closest with JavaScript with 40%. After that we had Python with 30% and then C with 22%. That was the top four. From a coding language perspective, I think you'll see that that impacts the tools that people are most familiar with using. For example, in Functional Automation section, Selenium was number one. Again, this was multi select. We had 81% of people using that. But the number two is actually testng, which obviously you wouldn't see that if we had done mostly JavaScript or mostly net or C focused community. You probably wouldn't see Testng on there at all. You might still see Selenium. I think the geographic components in the overall language component played into which tools were most popular.

So Martius, were you surprised? What? Not as many people as you would think would be using C. Still, when I first started bdd, one of the implementations of Cucumber that I really enjoyed was SpecFlow because it works so seamlessly with. Net. Any thoughts around that?

No, Joe, we're not really surprised. We do know the demographics in the BDD space fairly well and we always run under the assumption that about, you know, 1/4 to 1/3 of the community is on the C side and the rest is on the other programming languages, which is pretty much confirmed by the language demographics that Kevin was hinting on. So we were not surprised. We also saw an interesting and basically a similar distribution on some of the questions that are related to BDD that basically asks how many users are on Cucumber, how many users are on specflow or other BDD implementations? And the distribution was fairly similar to the programming language distribution. So not a surprise. We had about 20, 22%, I think it was, were using SpecFlow and 70ish percent were using cucumber. So it's very similar.

So, Ivan, I know that Flood IO has a really cool open source project called Elements. Did anyone mention that in the survey?

I Don't think that was one of the options we specifically asked for, although 12% of respondents indicated that they were using Flood as a tool. But Overall I think JMeter no doubt still has the vast majority of the response of more than 50%.

What was interesting though is the Selenium and WebDriver in general from a JavaScript perspective, comes in at nearly 40%, which is interesting from a performance standpoint that sort of backs the trend of moving towards browser based load testing. Element is obviously the tool that you mentioned that we've built specifically to kind of work in a similar way, and it's built on a similar syntax to webdriver, but runs on Google Chrome's Puppeteer framework under the hood instead of webdriver, so generally gets more performance. But that I think still has a long way to go as far as getting the mind share in the development community, which the back part of the field is definitely still being held up by the likes of K6 and Siege and Locust and Gatling and so forth.

So even though I think the survey was kind of skewed towards Asia in general, do you know if there are any trends in open source tooling or technologies that people need to be aware of that you think they should be learning because of it seems like there's an uptick in usage or popularity.

Yeah, I think one that was very interesting for me at least we asked people for their common challenges getting started with open source tools. The technical training and skills was the number two thing behind time. So obviously training up on these tools is something that people can do with, especially if they have more free time. Now I would think that training up on some of these most popular tools that we mentioned, like selenium, appium, specflow, cucumber.

Elements, JMeter, et cetera, they can't really go wrong there. But one of the really interesting things was the third biggest roadblock that people saw was actually getting test environments set up. This is an area we want to dig into More in the 2021 survey, obviously see how these trends go year over year. But I think some skills as well that people might be good to look into would be working with cloud providers and working with things like Docker, Kubernetes, et cetera to see if they can get some skills around. Learning how to set up test environments more easily and also putting the test data in them as well was the fourth largest challenge, I would say, Joe, not just the obvious things like learning how to use a tool like let's say, Selenium, but also maybe how to set up Selenium grid implementations and also maybe how to also deploy test environments themselves or some basic skills around Docker and Kubernetes, aws, Azure, things like that might be good to look into.

Yeah, I think also with COVID 19 more people working remotely, I think cloud, if it wasn't huge to begin with, I think it's only going to get larger now that more teams are remote, I would think.

Right, yeah, that's something that we certainly believe to be true and we are definitely interested to dig into that a bit more so in next year's iteration of the survey.

Matthias, any thoughts on the trend going towards cloud?

We actually saw a really interesting increase in Docker usage in the last two months. So I think there is actually already some evidence on our end that our users have been using containers and cloud environments more than they used to. So definitely happening.

So Ivan, when folks want to be able to test performance against all these multiple environments in the cloud and on Prem solutions, is anything that came out of the survey that you saw relates specifically to cloud or issues with people trying to really do proper performance against all these different environments?

Yeah, so it's interesting. I mean, from the respondents to the survey, there was quite a lot of roadblocks for load testing. So overall we had about 23% of people saying that load testing is something they're doing every day or, sorry, every release cycle. But the roadblocks to getting started are still very much entrenched in getting test environments set up, test data, technical skills and so forth that are required for doing that. So as far as the environments go, we still get customers who just want to run the tool themselves. They want to scale up, for example Element without running it on flood, or they Want to run JMeter, then upload the results to flood for analysis or comparison.

As a result of that, we've moved more from just being a cloud based load testing platform to also supporting agents that you can run on your own infrastructure and so forth. So hopefully kind of addressing the demand from both sides there.

Very nice. So another thing that surprised me about the survey was the role. 52% said that they were QA. And another thing that surprising that goes along with that is that 83% still say QA testers are responsible for functional automation. I assume it's the same thing with performance, but I don't know. So the past few years it's been all this talk about developers helping out. The reason for that is we use the same languages as the developers, the same IDEs and we'd be all one happy kumbaya sprint team, everyone contributing to testing. But it doesn't seem to be the case. Kevin, were you surprised by this?

Yeah, I mean, I wasn't too surprised to answer your initial question. The one thing that wasn't interesting, and I guess you could boil it down to it being a more specialized testing process, is that load and performance testing was actually more skewed in terms of, or I shouldn't say skewed, but more distributed in terms of who was doing that testing. And that type of testing specifically is only 50%, roughly of it being done by testers. The other 50% was split up between dedicated performance engineering people, developers, operation folks, et cetera. So I think what you see as a trend is that in general testers are carrying the weight of doing testing still, which is kind of what we thought the trend was going to reverse. That's not happening, but I guess the areas where it is happening the most quickly or coming true the most quickly is sort of around more of the performance testing. Maybe that Ivan might have specific thoughts around that, but might be just because it's more specialized type of testing and not every company is going to be able to hire someone dedicated to do it. So then people sort of have to share the load. But certainly, as you mentioned on the functional testing, it doesn't seem like developers are picking up the slack there in terms of writing functional test cases anytime soon.

I guess the whole point of BDD was to get more folks involved. So Matthias, any thoughts? People not implementing bdd, you think in a way that it could help facilitate this to get more folks involved in the team with automation?

Interesting question, Joe, and you're hitting the point there. So that's essentially the intent of bdd. And funny enough, we have run another survey a little bit earlier where we were diving a bit more into the question of how many different stakeholders are part of your BDD process. And one of the findings of that survey was that.

Devs and QA are fairly involved in bdd. QA is mostly for authoring the scenarios and the feature files and then dev mostly for implementing them and coding them. So this is. And you have very little involvement still, unfortunately from any business stakeholder like POS, PMs or any of the likes. So I think there is room for improvement there. And.

And certainly.

If all these roles would be involved, you would definitely see a better distribution acceptance of automation.

Awesome. And we did touch on a lot on performance. So Ivan, any surprise? Once again, I would think the opposite. I would Hope more QA testers would be involved with performance. But then again, it seems it's the opposite trend still with other teams picking up that load. Is it because it's a skill set that needs to be expanded a little bit more?

Yeah, I think you hit the nail on the head. It really is. I mean, while we do see, like Kevin mentioned, about 47% of people from the QA side saying that they're doing performance testing, performance testing, as we know, is a very complex and nuanced kind of type of testing. So even if you're doing the test, you know, the actual running of the tests, if you're from a QA side of things, you're not really going to be responsible for, you know, implementing the findings and figuring out how to scale the servers or make the application more efficient, which is probably what's driving the other side of the equation, which is, you know, the, the other 50% who are, you know, from all of the other disciplines within the organization who are, you know, also conducting performance tests, from site reliability engineering to development teams to operations and so forth. So I think that honestly I would actually expect that in the future we're going to just see performance testing rolled out or become more of a common testing discipline within a lot of different teams within the company, not necessarily just testing in general. And we're already kind of seeing that with the adoption of more developer focused load testing tools on the market. I mean, K6 is doing really, really well. We're seeing now 35% of all tests on Flood are done with Element, which is our developer focused tool. And I think it's just generally a move away from.

The kind of build your test in a GUI kind of thing and working more at a low level with what's actually being tested. That's not to say that there's still a really strong role for QA to play in terms of a broader picture of what the quality of the product is and the quality in terms of the properties around performance testing.

So besides that, was there anything else that stood out to you? Maybe any myths that you saw that people believe in, but based on these surveys results, you're like, oh, wait a minute, maybe that's not so true.

Well, actually there was a really good point about that. Not necessarily aligned just with load testing, but general perception around open source and security in that I think one of the biggest roadblocks that companies, particularly large enterprises, identified as a roadblock to adoption of open source tooling was that, you know, the perceived security or security vulnerabilities that are associated with open source tools as opposed to what they'll get in a closed source vendor supplied product. And I think this runs contradictory to what any security professional would tell you, which is that an open source product is inherently going to be more secure because you've got more eyeballs who can discover vulnerabilities and potentially fix vulnerabilities and more importantly, make those vulnerabilities known. Which personally I'd rather know that there's vulnerabilities in a piece of software or a system than kind of run blind and say, oh yeah, it's 100% secure, there's nothing to worry about, if you.

Know what I mean.

How about you, Kevin? Anything that really stood out to you as a myth that you thought you believed and then you saw the survey results and thought maybe not so true?

Well, we went over a bunch of them, but I would say the one that we haven't touched on, sort of to Ivan's point, you know, we did a kind of question that was, you know, pick what are some of the biggest benefits of using open source tools. And we did see that, you know, cost was the number one in terms of 39% of organizations saying that they thought that one of the biggest benefits they got from open source tools or that they're typically free. But it was interesting that some of the other factors in there, a lot of people were 60% put something other than cost as the biggest benefit. Community integration and customization seemed like the big three, that if you add those up there's way more. About 46% of people in that bucket and saying that those were the biggest benefits rather than cost. So I think that was a good trend for me to see because we've been saying for a while that open source tools are not just valuable because they're free, they're valuable for a lot of other reasons which ultimately make them better tools than commercial or closed source tools in many cases. And was excited to see that more people were saying something other than cost was the biggest benefit that they get from using these tools.

Matthias, I know BDD gets a bad rap because people, I don't think really implement it correctly. Were there any big challenges that came out of the survey that you think people are struggling with with bdd?

Well, one of the things that not so much a challenge I think, I mean people said that authoring good given when thens can sometimes be difficult. And I can see that.

Right.

It's one of the biggest issues is to write good even when then sent make sure, you have great, especially the assertions at the end to have them well written and specific. But I think there's a lot of education still to be done to make sure the market matures in this direction and understands how to really do this. And you know, it's part of our mission to educate as well as the SpecFlow team. But there's another super interesting finding that we saw that we didn't really expect that way and this was that the percentage of of respondents that are doing actually automated unit testing was fairly low at only 31%. And we actually assumed that number to be much higher because we were assuming basically that we have multiple different automation techniques be applied, unit testing being one of them. But you know, it sounds like 2/3 are just skipping over it, are doing something else before they're doing unit testing. What was also interesting to us was to see that for people.

That have responded here, the percentage of people that are doing both functional tests and.

BDD automated tests, I should say is relatively high. So we almost have in the group at least that's using BDD, it's almost even. So we have 77% of people doing automated functional tests and about 70ish percent of people that are doing BDD automated tests. So that's a surprisingly even distribution in that group.

However, and this was the real bummer to me actually. A lot of people said that about 65% of the automation that is being done, no matter what it is, does not run as part of their DevOps pipeline, which I was super surprised about because I would assume the whole reason why I'm doing automation and I'm spending so much time to get my testing this mature that I can automate it, is to have a super fast build pipeline. So I was very surprised about that.

Yeah, that is pretty surprised, Kevin. And any reason why that would be based on your experience with functional testing?

No, that was a head scratcher for us as well. So I think an area that we want to dive into more so next year is sort of CICD trends in general because we touched on a little bit of it this year. But to materials ts point, we can't really connect the dots and we do know that there's generally a trend towards people doing their functional testing more frequently. We saw that about 80% of people want to do it with every build or at least with every sprint. But we really want to dive into a little bit further next year. How that's happening, who's doing it, the process of building those tests, coverage of unit versus BDD versus functional tests. So I think that's going to be an area of focus for next year.

What was a head scratcher for me, but in a good way, which was a myth, I thought that people didn't do performance testing. But Ivan, I'm looking at the results and when you add up people that do very frequently to frequently do performance or load testing, it's like 56%. So majority of people are doing performance testing, which I don't, I don't necessarily. I didn't believe. So is that something you see at Flood? Is that.

Yeah, that's, that's, that's also another surprise that stuck out to me as well. I was wondering whether or not people were answering that aspirationally or realistically.

But it's interesting that, yeah, so we had 28% linked to a continuous integration service, which is definitely a lot higher than what we've seen within our own product. But you know, it could also be.

Connected with other products that are in our space that have, you know, a more entrenched kind of DevOps ecosystem. But at the same time.

We still find that throughout the year there's definitely a very strong seasonality to performance testing either, you know, corresponds to the upcoming big events or corresponds to, you know, general business trends throughout the year. You know, certain, certain times of the year are busier than others for different regions of the world, summer versus winter and so forth. So it's quite possible that during those particular times we see a lot of engagement, a lot of setup with CI and then the rest of the year maybe there's not much development happening. The CI setup still happening or the linking within jira so forth to back to your performance testing is still happening, but there's just not much activity going on within the R and D kind of unit which is driving that. So it's still, you know, could be true in both cases, you know.

Okay, before we go, based on the survey results, is there one piece of actual advice you can give to someone to help them with the automation, their performance or the BDD testing efforts based on these survey results? And what's the best way to find or contact you all or to learn more about the survey? Kevin?

Yeah, absolutely. You know, my advice would be take this time to skill up. We talked a bit about the different skills of you that were in demand and that we thought could bring value to organizations. But if you got some extra time on your hands at home, I think we have certainly discovered that a skill gap is one of the biggest roadblocks to people using open source tools, so certainly think that's something to invest in. And for those people who are interested in learning more about the survey results, we are going to be hosting them on our website with a link that we'll provide to you Joe. It's tricentis.com commencement state of open source 2020 and also we'll be hosting a webinar which we can provide the link to you as well on May 27, which will also be recorded for later viewing on demand. And we'll be going together through the results, myself, Ivan and Matthias and talking about some of the more detailed findings.

Very cool Matthias. Anything pounding words of wisdom or put.

The survey yeah, absolutely Joe. So you know what I would like people to wrap their head around when they think about BDD is really that this is not a single individual activity, it's a team activity. So basically on what we discussed earlier, Joe, it's really important that people understand it's a collaboration effort that tries to involve everybody and spits out good automation and automated documentation. So if you do it right. So I would like people to keep that in mind when they apply it, make sure that every team member is involved and that usually yields the best results.

Very cool. How about you Ivan?

Yeah, for sure. So I think right now, as Kevin mentioned, everyone's kind of stuck indoors with a little bit more spare time on their hands. So there's never been a better time to explore the field of open source testing tools. Especially if you're looking to save your organization a few dollars. There's no better time right now to adopt an open source testing tool. And yeah, if you have any more questions about Flood or about Flood Element, you can find us on GitHub or you can reach out to me directly. Ivanlud IO thank you Kevin, Matthias and.

Ivan for your performance, your functional automation, your BDD testing awesomeness. For links to everything of value we covered in this episode, head on over to tuskill.com a305 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Source Lab's awesome products and services. And as Ivan mentioned, a lot more folks are getting involved in performance testing. I believe it's a big new trend. If you want to learn more about how to succeed with performance testing and performance engineering in 2020 and beyond, you definitely want to register for the 2020 PERF Guild, an online conference dedicated 100% to helping you succeed with creating performance testing. Awesome. You don't want to miss this year's event, head on over to perfguild.com and register today. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to Test Guild for full show notes, amazing blog articles and online testing conferences.

Don't forget to subscribe to the Guild.

To continue your testing journey.