244: Fast Forward Your Entire Development Cycle with Israel Rogoza and Avishai Moshka

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. In today's episode we're going to be test talking with Israel Rogoza and Abhisha Moshka from incredibuild all about how you can fast forward your entire development cycle, discover how to reduce your builds, tests and other tasks leveraging grid computing. So if you look for ways to accelerate your software development efforts without reducing the quality delivered to your customers, you want to listen to this episode, check it out. Also, before we get into it, make sure to head on over to perfguild.com and register for an online conference dedicated 100% to to performance testing. It's one of the most overlooked areas in software development and if you really want to gain an edge with customer satisfaction, you should check out Perf Guild and discover all the ways you can help increase your application to be more performant to make your customers really not worry about slow response times. Check it out@perfguild.com Testox is sponsored by the fantastic folks at Source Link Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the Sign up now link under the Homepage Sponsor section.

Hey Israel Navashai, welcome to Test Talks.

Hi Joe, how are you?

Great, great. Great to have you on the show. Israel was one of our first speakers at Perf Guild in 2017 and so he has a lot of experience around performance testing. And Avishai also is from a performance background so these folks really have a lot of experience around performance testing. But also I think you've been speaking with a lot of teams about how you can help increase the build times and how to shift left and really help the development cycles with DevOps automation testing and release automation. So that's what I thought we'd talk about today. But before we get into it, can you just tell us a little bit more about yourselves?

I'm here in incredibuild almost half a year as the Product manager of the new product of Form IncrediBuild, which will take IncrediBuild abilities to the cloud. Before that I was the inbound product manager of StormRunner load, which is today Micro Focus Stormwater Load, one of the leading SaaS solution for performance testing. Today as the product manager of Incredible Cloud, this is the first time that I am getting a product from scratch to the cloud as a new product and I think incredibly have a lot of solutions and big advantage to accelerate many of the time consuming tasks that we are all familiar in the qa and the DevOps world.

I'm Ishai Moska, I've been in incredibly for six months also and I am the product manager for the infra and testing solutions of IncrediBuild. Before incredibly I was 10 years with Mercury, HP, HP Enterprise and Micro Focus Load Runner. Started as QA engineer, moved my way to Dev teams team leader and then ended up as the inbound product manager for Northerner Protocols team. Right now working with credit with customers on how to accelerate their unit test process as part of the accelerated development process and how to encourage shift left for testing in the developers IDE or in a natural environment.

Very cool. So I just think it's curious that both of you are from really hardcore performance testing backgrounds. I guess my question is what made you choose a credit builder or do you think performance testing isn't as important? What was the shift? What drew both of you in almost a different direction from performance testing which seems to be what your core focus was previously.

So for me the idea was that after 10 years in the performance testing area I wanted to refresh and move and expand my knowledge and expertise to different areas in the development cycles. Course still in developing products but in a different aspect of the testing world other than performance testing. I just wanted to search my next episode, let's call it Startup life or the small company life and experiencing something which is more broad than the test involved.

Actually to tell you the truth, after Rabishai got his idea to move to Incredible, the first call was to me and once he introduced to me incrediBuild, I was really fascinated and very excited to hear about the product abilities and the things that he can do and the great potential. And since I'm a huge fan of cloud and SaaS solution and that was part of the roadmap, the plans to incredibil to extend not just what we have today which is a great product, the on premise product though to extend and take our ability to the next level to the SaaS and the cloud solution, the advantage of building something from scratch, from new something that I think for quite some time this is something I'm looking for and I would like to do that and to add it to my expertise, to my experience to learn new things.

Awesome. And what I Love about both of your answers was you wanted a new challenge and I guess a new challenge is figuring out what customers want. And based on previous conversations we had, it sounds like you two did a lot of research before you started developing a few new products that credit build has come up with. So just thought maybe we can get a little insight into what kind of research you did and any insight that you found from your research that actually turned into our product. I'm just curious to know what that process was like or what how maybe findings you found what people are struggling with when they're developing software to get it out the door quicker and faster with higher quality.

From my point of view, the cloud solution is not something that you can take for granted. You have to speak with your customers and understand that if there is a common pain, if there is something that we can solve by using cloud resources. We had a lot of interviews to understand first what is the requirements from this product. Maybe you know what, maybe before that if this product is even necessary, we understood that one of the answers can be no. The on premises is good for us. We don't need cloud. My guess was even before we started that this will not be the answer that we will get. Customers understand now that the cloud and the SaaS is not something that is only for endpoint enterprises or most of the security issues are now solved in that way or another. We interviewed, I had like almost 20, 20 customers each customer. We interview for one hour, I interview for one hour to find this challenge. And we got to the common case that customers now compromise since they are. We heard that I think from most of the customers and we interviewed basically the R and D. So. So we have this perspective and the common thing was that they have to deliver fast and they have to deliver fast because let's take the gaming world for example. We have a lot of customers. The leading companies in the gaming industry is our customers. And they told us guys, we have a lot of competency and there are a lot of competitors in the market and there is a new game almost every week and we have to deliver much faster. And the way we sometimes compromise is that we doing less testing since the testing cycles is long or our DevOps cycle is too long because we don't have enough resources. We release sometimes half baked products that eventually influence us in many ways in customers abandoned us and customer not buying our product in customers complaining about us in the social media and we don't want those kind of things during for example the Christmas time which is the most selling point of the year and we are looking for some solution that will help us to keep our product quality high, not only with the product today as we have other companies in the finance for example, that adding more and more code. We have customers in the AI world that are looking for analyzing more and more data, doing more and more simulations and need more resources. And this got us to conclusion that the cloud product is necessary because the market need that kind of solution. They need to do things much faster. And this is the basic knowledge and the basic idea that we got and from that we took the MVP for the product and we understand what is the most painful area and we hopefully I really hope that we answer this so our customer can enjoy this and can have a much faster cycle. We saw customers struggling with 10 hours builds or customers that have a very impressive testing suite and all of that and then Moshka will address in a second. But they are basically skipping tests, you know, just have a lot of testing comment just to make sure that they are delivered much faster. Moshka, I guess you saw that also.

Well yes, even before. Incredible. Working with customers or companies on agile transaction, how to move to be more agile or moving from waterfall to Agile or DevOps. We see customers that tells me look but Agile methodology requires me to add more and more and more tests and more test coverage, more unit tests, more functional tests and more any other tests, but leaves me with zero to no time to test. So I either need to compromise by selecting which test to run or I need to compromise my release time by choosing to run more and more test cycles or more tests off my test suite. We took that kind of problem to customers, to existing and new customers and potential customers. And we asked them why you don't run your whole test on every commit or any change or every time. And the answer was the same. Because we don't have time. Our Test time takes 30 minutes, 2 hours, 12 hours, God know how much time. And if you run the entire test suite on every commit or on every build on every change on our system, then we would release basically nothing. Another answer I got from customers said that Agile methodology declares or requires us to run only test only on areas that was changed recently. We assume or want to believe everything else is still working. And now we need to change the test, only the change product. And by that we are running only 10%, 5%, 1% of our test. And once in a month, two months, one in the release we run the file test suite with that information. We ran some tests to understand how can you accelerate how can you use existing resources or additional resources to run more tests on the same time, at the same time given without adding extra layer of complexity to your testing process. And another conclusion we have is that testing environments cost money. It takes a lot of expertise to raise another test environment, another database, another prerequisites, another connection, another machine with enough software components and so on. And every machine, every test environment that you have cost a lot to maintain. It costs, it's a security cost, the viewers, firewalls or whatever you need. And also you need to make sure that all your test environments are literally the same. Because if you find a defect on one environment, we don't want to run the entire test again or on a different environment just to verify. We want to be sure that a failure is a failure and not some kind of an odd behavior of the environment.

All of these inputs during several months of research and several hundreds of customers took all of those into an R and D meeting inside our company to understand how can we answer all of those challenges and focus is on a simple solution.

Great. So I guess one thing. When you come up with the solution, how do you know? I guess if someone has a lot of tests and they say they don't have time to run them, a lot of times, even if they could run them, the tests weren't designed to run all at once. So is that something you had to take into consideration when you're coming up with a solution?

I wish. I have all the answers to every problem in the world, especially in the testing world, then I was rich. But taking under consideration the testing world, which is a very complex and complicated world, we knew that we would not be able to solve everything in a single blow. The same way that nine women will not deliver a baby in one month. Some things or some process cannot be run in parallel or distributed, for example. So we narrow down the problem or the challenge into specific problems that can be parallelized. I must say there are some several assumptions on the testing environment. For example, you are running isolated code or isolated tests. You minimize the dependencies between your tests. Focusing on unit tests, we understand or assume that your unit tests are isolated with minimal dependencies. And the test each test a very, very specific thing, as unit test should. Of course, there are some abuses developers who say, hey, I'm already running the test. Why shouldn't I open the database connection and do this, this and that? But these are all bad designs or cases. We usually say, hey, you need to fix your code to be able to run in parallel. Otherwise we won't do it for you. That's one aspect. The other aspect is understanding that even if there are dependencies in test, if you have one case we saw is 100,000 tests on a single project, not all of them will be dependent on each other. And you can divide those tests into several groups. One group can be run in one machine and all the other tests can run in parallel. So there is some work we are doing or the customer need to do in order to make sure that everything can run in parallel. But it's a minor the cost is minimal compared to the benefit that you get. For example, these 10,000 tests instead of running for almost 24 hours, we were able to execute them in 12 minutes. So compared to the amount of work that the developer needed to do to change one or two things on his tests, saving almost a day of work to running the entire test with in 12 minutes, I think it's worth in terms of ROI.

So I'm just curious to know since I have a performance background, as you were speaking to customers, did performance testing ever come up as being a bottleneck or because it happens at the last minute rather than earlier in the software development lifecycle, is that even something that this type of solution would help address?

Performance testing, unfortunately, and coming from years in that area is not very common in companies who don't have the performance platform or haven't been burned for performance issues. Unfortunately, there was no special requirements about performance testing as incredible. Specifically, AIM is a component aimed for developers and if we take shift left in the perspective of developers, most of them as far as they can see they need to do unit test and maybe some integration test. Unfortunately not many organizations or dev organizations are familiar with the needs of doing performance testing as early as possible in the dev cycles. The one that do performance tests usually don't design the performance test to be distributed test. They use the infrastructure of the product, maybe LoadRunner, JMeter or whatever, or maybe even some guys use Selenium for that. They rely on that infrastructure to distribute their tests.

I also think Joe, that I'm really sad to say this, but performance testing is something that is today for customers is a bit less important than the unit testing and the G test and the functional testing. I heard a few customers it's not a lot, but there is some trend to skip performance testing since you can always use the auto scale. And a few weeks ago I heard some very the CTO of a very famous music service that we are all familiar mentioned that he is not doing any performance testing Once he's monitoring his system, understand that there is the peak time they just spin up more dockers, more servers for that. And once they see that the peak time is over, they're shrinking again. And the same methodology I heard from the CTO of one of the leading banks in the US and he mentioned the same thing. He said guys from 9 to 5 we have a lot of servers. The autoscale is working really hard and at 5 o' clock we start shrinking until 9 or 10, 10pm and overnight we are in a very very light footprint on the, on our servers. And the same thing goes at 8 o' clock we starting to expand again. And this, this eventually the reason why we don't do any performance testing. And instead of that we take all the resources to make sure that our mobile application is working, that the functional testing, the UI is, is working. The UX experience is at the highest level, that the code is running, that we can do large and heavy compilation. This is also what we heard from incredible customers. This is the things that is most interesting for us. We want to make sure that the backend is working, that the UI is working. Basically this is what important to customer. I think Avishad also heard that from a few customers.

Abhishai yes, there are, there is a variety. It basically goes in a very 1 and 0 customers who are based on microservices infrastructure or know how to expand. At the last they really use monitor and on test. That's their methodology in terms of performance testing and so on. And they heavily rely on unit test, integration test and any other in dev tests. While customers who have non expandable back end. Most of you, the guys with very very heavy background, if your system is on a mainframe or in house non scalable system, they do a lot of performance tests because they don't have the hardware or the capability to scale up and down.

What it sounds like is based on speaking to a lot of customers. You've noticed that there's been almost a shift in the industry where with agile the main concerns now people have are trying to keep up with velocity and also deliver quality, run all their tests and be able to scale up all the environments that are needed for all these tests in a way and still ship a feature as fast as possible. And now that I think of it, I don't even think we even explain what is a credibuild and what exactly you guys do. So you could explain that a little bit more.

So incredibuild is basically an infrastructure that allows you to convert your machine into a virtual super machine, allowing you to harness all the available resources on your network in order to aid one of your computing heavy resource computing tasks, and by that reducing the time of the task to compute. And I gave you a very simple example. If you take a compilation, for example, the compilation process is dependent on how many cores you have on your physical machine. So on your developer machine you have basically four cores. This is the maximum amount of parallelization that your compilation can work with. If you will use incredibly incredibly will literally simulate to your compilation process that you have 101,000 or 10,000 CPUs available. And therefore the distribution level or the parallelization level of the process can be much higher and therefore computing time will be much lower or shorter. We are doing that by a very unique technology we have called process virtualization, which allows us to run any process on any machine on your environment without any prerequisites and simulating to that process that it's actually running on the main or your own machine. This is why in the compilation or testing solutions, you don't need to prepare multiple compilation environments or multiple build servers or multiple test environments. All you need to have is that your tests or compilation will run on your machine and after that install incredibly agent on several computers on your network. Run the compilation or test on your machine and the tests, the process will be distributed across your network and at the end all the results will be aggregated the same way that you might have been running it on a supercomputer with 1000 CPUs. That's in a nutshell, what incredible is. And to answer your main question on how incredible and how the research we did helped us deliver or develop the solution for unit test and cloud. So in terms of unit test, what we did is basically we analyzed the unit test frameworks, the process that run unit test frameworks, and understood how we can divide unit test DLL and divide the execution instead of one DNN to run per call, take that DLL and divide the test inside of it into multiple cores across your networks. So the same use case I mentioned before, 10,000 tests running on a single machine for almost 20 something hours, running each test inside each test method on a different call on the entire organization took 12 minutes because there is no processing time or waiting or there is no queuing for test to complete execution.

I will resume from where Moshka stopped is that you can say that we allow a software that is parallel but is bound to a single machine to scale up and use unlimited cloud resource in Order to run much faster. And the way we do that in the cloud emphasize this advantage is the advantage that Incredible is a generic software solution. So we can basically accelerate, I think from each industry. If this is a pure software development, it could be from the finance, running simulations. If it's for the gaming, it's for shadowing and rendering and cooking and all of that. And then as Moshka mentioned this, the way we do that is by aggregate available CPU across the network and using it to create each machine as a virtual supercomputer with many CPUs. And in addition to the interviews, we got many customers with the legacy software like banks and insurance company that even without Incredible, even if they like to use or test their application by using HPC or using any mo and using even the basic distributed computing, they are bound to a one single machine. And this is something that IncrediBuild really can solve this problem. And if we take it to the cloud, as Moshka mentioned, we have kind of a helpers helpers machines helpers agent that give a hand in the job, in the task that is running and by leveraging, by taking it to the next phase, by using Incredible on the cloud, we automatically provision up to thousands of calls to accelerate this workload and deprovision take the advantage of SAS and CPU once the workload is completed. So this is something that is highly optimized performance and we saw benchmarks and ROI calculation extremely fast and reducing the cost and the time to market. And again we are returning to the same point where we started in the interviews. There is a problem that we can solve by that.

Very nice. So it also sounds like you said you made it so it's generic enough to handle pretty much any environment. So does a team need to do anything in order to start using credit build or is it just a matter of you coming in and setting it up and away you go.

It's as simple as installing an agent on each machine.

No dependency, cross platform we have Windows and Linux and I think.

One of the things that make it so quickly is that there is no dependency. You don't have to put your project in each machine. So imagine that even if we take the on premise example that everyone has an agent on their laptop, we have customers even install the agent on the reception laptop, the secretary laptop and everyone is giving a hang to help accelerate since you don't have to have any prerequisite. And Moshka, correct me if I'm wrong, but even if you are running compilation and you need the DLL file, for example, we know how to bring this specific dll. So although we are running on other computer, on other machine which don't have on her local operation system this file we can have some kind of a tweak and bring this missing file so we catch this request before it arrives to the operations system. This is something that is very unique and this is one of our biggest.

Advantage in terms of ease of use. We are actually the only commercial tools that can be installed within as part of Visual Studio. You can actually form your Visual Studio if you have for example a big C project or solution. You can use Incredible from within Visual Studio. It's literally zero knowledge. All you need to do is to install the agent and the license, verify that you have other agents in your network. And for Visual Studio just use incredibly menu, choose Build with Incredible and that's it. Your build gets distributed and you literally don't need to configure anything or change anything in your code or your build system or any other knowledge of what we are doing.

Awesome. So when I watch infomercials a lot of times they have people that lose weight and they say that these results are not common. So I think one of the examples you gave was you were able to bring some unit tests down from 24 hours to just 12 minutes. Is that a common that people can expect when they use a tool like a credit build or does that depend on each team, each situation?

It's basically there is a lot of dependency of what the process is doing. But in terms of unit test, we see a very, very linear connection between how much parallel tests you are running to how much time you save. If you are doing UB test the right way, then each unit test is isolated and there is no dependency between tests. So there is no reason why not to run all of your tests no matter how much there will be in parallel. And that's, that's how you save a lot of time with incredibly just don't need to configure anything. We just give it a test. I will run them in parallel. I can give you one more example. You can take a big commercial company who had, I think it's 72 hours of testing and they kept buying a bigger and bigger build machine. No 32 calls, 64 calls heavy lifting machine just to run the test one a week, once a week. So using incredibly you can do. You don't need to buy those heavy lifting machine. You can use every developer machine or every machine in your organization just as distribute test. Not only test, we have companies doing computation models, AI training, you name it, whatever shader compilation for the graphic designers, whatever multiples calculation you have, we can distribute it.

I also think that, you know, if we are talking again about quality, then we have customers that we manage to bring them to the, the much higher level since they, they could use IncrediBuild and accelerate it and use it for many of their tasks. So then there is a cycle, you know, they, they can run more compilations, they can run more unit tests, they can run more integration tests, they even can run code analysis, coverity and so on that we help them improve their performance. And it's not just in the software development area. There is an interesting case with a company that is dealing with diamond polishing, you know, something that is not that typical for software and they are also bumping in the same issues like the software. They are doing some simulation to understand how they can polish the diamond in a much more efficient way. And the simulation took them almost seven hours. So it's not testing and it's not something that is related to the pure the development, but is also a task that take them a lot of time. So the pain area again time to market and instead of doing much more, they are limited because of this time and we managed to reduce the time from seven hours to seven minutes. And again industry manufacturing diamond. This is something that was, we were really fascinating to hear the case study during the interview because this is something that we are not bumping with manufacturing a lot. And once we do that, we are very happy to see that what we have in the value proposition, the thing that we called incredibly generic customers, take it to the next level by really making it a generic tool, makes you.

Want to say that's incredible. So all these results sounds pretty good. Okay, before we go, is there one piece of actual advice you can give folks to help them increase or help with their quality efforts as we move towards this rapid development and release type software efforts that we all are involved in.

So that's an excellent question, Joe. The idea, my point of view of being agile or moving to faster releases does not mean you can compromise on the quality or the user experience of your users. If by going more agile or more DevOps kind of style means that you will run more, less test or invest less in your quality just to justify that you will release faster or release more versions in amount of time, then don't do that, don't compromise on your quality because fixing a defect in dev time will cost you much less than in production time regardless of the social media and users that will leave your product in terms of mobile and so on. So regardless of if you're doing really using credit build or not, if you do run more tests or not, not compromise on your quality for quantity. That's my message to the developers world out there. Any release manager can hear us. Please give your QA tests and even performance tests a big amount of weight in your release planning.

I think though that from my point of view we mentioned a lot of challenges. I like to say that today time isn't just money, you know, it's the key to growth and there is a never ending race to release to the market faster. There is always more and more tasks handling time consuming development and we all know the results in rushing from releasing half baked products even can also lead to financial implications. And this is exactly what Moshka mentioned. Don't compromise. Although you have to release faster and the board and the CTO and the CEO, they are all pushing to release much faster. Find the ways to make your process much more efficient. This is using incredibuild. If this is adopting and learning new methodologies, this is even moving to a different technology. If it's change in the way you work, don't compromise. The market is full with applications, with software, with SaaS, solutions that can be replaced very fast and very quickly. We have a lot of integrations companies helping you to transition from one application from one service to another. So you have to do you have to make your own advantage and the quality of the product is I think the number one advantage. Customers willing to compromise with the features and sometimes will be more patient if your product quality is high. So this is my take and personally this is how I treat my things, my applications. I like my applications in a very high of quality and sometimes I see applications with the fabulous UI with a lot of features but poor quality, crashing buttons that don't work, load that keep crashing and the first thing that I do is how do I get a refund for that. So this is my point of view.

Thank you Israel Navashai for your grid computing awesomeness. For links to everything in value we covered in this episode, head on over to testtalks.com 244 and while you're there, make sure to click on that. Sign up for a free trial link under the exclusive Sponsor section to learn all about Source Labs, awesome products and services. And as I mentioned, Perf Guild is now open for registration. You don't want to miss it. Register today. Head on over to perfguild.com so that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed in creating automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes available. Amazing blog articles and other automation awesomeness.