Is Cross Browser Testing Still Needed with Mudit Singh

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Mudit Singh all about Cross Browser Automation, Selenium and more. Mudit is the Director of Product and Growth at LambdaTest and has years of expertise around product development, testing Automation, Selenium and Selenium Grid. So listen up.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com test source labs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Mudit, welcome to the Guild.

Hi Joe, thanks for having me here. Thanks for taking the time to connect with the call.

Awesome. So before we get into it, could you just tell us a little bit more about yourself?

Right, so My name is Mudit, Mudit Singh. I've been since I graduated around 10 years ago, I have been in the space of software development, specifically web software development. So I started off as a coder myself and after two to three years I went into the product space. I started helping the teams in the management of the product and defining the product guidelines, product development guidelines. And I have been Basically for past three years now I'm working with a company called LambdaTest and here we are building up a cross browser compatibility testing platform focusing on helping users perform both manual and automation tests. We are building up an infrastructure that can help users to.

Eliminate the need for building up an in house device lab.

Very cool. So I guess the first question I have is around cross browser testing, is it still relevant? What I mean by that is I always hear that the JavaScript engines have become more and more similar. So as a company you probably see a lot of clients, a lot of scenarios. Are there really differences still between the browsers that people need to be testing for?

Yes, that's one of the most important questions that comes to us and the simple answer is yes, there is still a relevance of testing on browsers. To understand it more, we need to have, you can say, look into the history of the browser development itself. So there's a term that people use nowadays and it's been still going on, it's called browser wars. So since the first browser was launched nearly around 20 to 30 years ago, called Netscape. So from that time itself, the first competitor started off from Microsoft themselves as Internet Explorer. And that triggered the first, you can say, browser was safe. Since then, there are multiple players in the market. Each have their own different technology that they are using in the backend. Each have their own different motivation of developing the same piece of technology called browsers that help you access the Internet. So right now, if you look at the different browsers that there are, for example, Chrome, Firefox, Safari, Opera, Internet Explorer, Edge, each of them uses a very totally different technology in the backend. So for example, Chrome, Chrome uses browser engine called blink and a JavaScript engine called V8. On the other hand, Firefox uses a different engine called Quantum, it's the latest one. Earlier it used to use gecko and a JavaScript engine called Spidermonkey. And because each browser has a different, we can say engine working in the back end, the end result of a same piece of HTML, CSS and JavaScript code comes out totally differently in each different browsers.

Absolutely. So how does one test that then? Is it more just visual validation or.

Can you give us an example of a solution to get around that particular issue with running being able to validate against all these different browsers? And I'm sure also the desktop versions and the devices also kind of can cause issues as well on top of it, Right?

So it's what we call cross browser testing. It's the core of it. So the only solution to this problem is that you would have to test your same website, your same piece of code across all the different browsers and all the different operating systems, in fact, and all the different mobile devices as well. There is no workaround for this. You have to test on all the browsers operating systems. It all depends upon how important different browsers are for you. And based on that, you can define a strategy for your cross browser testing. For example, each browser has different demographics that they are using. So you can differentiate based on the target audience that you are going to target. Your website or web app that you are going to build, it would have differences on the audience on the basis of the device they are using. So for example, if it's a banking application, banking software, which is used by mainly you can say bank employees, it would be targeted for desktop web browsers. However, if you are building, for example, a news website or a social media channel, you know, most of the users would be using it on mobile. So you can prioritize your testing based on your audience. So there are a number of tricks how you can do it first is obviously the audience, then you can also see. So this is all predictive analysis. So you can predict how the audience will, which type of audience will come and based on that create a strategy. Then other would be a reactive strategy. So for example, you would have a Google Analytics code in your website on your web app. So in Google Analytics tells you like these are the browsers and these are the operating systems. Even browser versions, devices your users are using to access your website. You can find it in Google Analytics and there's a specific technology section in the user section there. And based on that you can define a strategy like these are the most important browsers for me and then I have to perform a test on these browsers. Once you have a list of browsers that are most important for you, we can build, you can see a browser metric. These are the browser versions, these are the devices of those browser versions. These are the operating system. You build a matrix of all the operating systems that you will need to run your test on.

Once you have a list of the browsers, the next step is to find out what would be the best strategy to test on all these browsers. So.

Earlier what people used to do is they used to create a device lab in house. So if you are a big enterprise, you can, you have deep pockets, you can purchases, purchase multiple different browser, multiple different devices and multiple different, you can say desktop applications as well. Desktop devices as well. So for example, if you want to test on Safari, you would need a Mac laptop. And if you want to test on Edge, you would need a Windows laptop. So you have to invest on different, different.

Hardware as well. So this is an expensive method. Agreed. But if you have a very limited number of browsers that you want to test on, this is the fastest and the, you can say idle solution for you. However, if for example, you have to test on a huge list of browsers, a huge list of different browsers and browser versions, the best solution right now would be to go for a cloud based solution.

I definitely agree. I think a cloud based solution saves so much time. You don't want to spend your time as a tester worrying about a lab when you have solutions out there like yourself and Sauce labs that provide these for you. But I guess a lot of pushback I get is, well, I'll just set up a Selenium grid or I'll just have Docker instances come up and fire up I.Q. something like selenium, which is a cool solution. So what would be some pros and cons? To someone going with more of a Selenium grid type approach.

Right. So again Selenium grid is a very great way to set up an automation system and test on different, different browsers. So let's start with the cloud based approach as well. Right. So there are multiple ways that you can create, you can say cloud based infrastructure for yourself. You can go to a service provider like Lambda Test and Source Lab. So they have built in machines that are ready to fire up, ready to scale up. You can run your test on there directly, but that's, you can say easier approach. Another approach would be to create and set up this kind of infrastructure for yourself. So you go to a cloud based infrastructure provider, for example like AWS or Rackspace or DigitalOcean and you can even go to a bare metals host service providers and then use containers, Dockers and Selenium itself to create a grid of your own. The challenges here would be that you would be limited to the number of browsers that you can really test on. There are the limitations on that as well. So for example, Mac machines, if you want to test on Safari, you need online Mac solution provider. And they are totally different than the people who are providing Windows or Linux machines. So Mac only works with authorized Mac resellers. And they are also very costly to. You can say lease on different, different resellers. For example, there's Mac Stadium, you can contact them to lease Mac machines online. The other challenge is to maintain the browser versions. Chrome has a very fast release cycle. Every two to three months they come out with a new version. Firefox has a similar fast release cycle. Every two to three months they come out with a new version. So every time a new version is called, you will have to change your Docker image to like update this change in the browser. So you would have to maintain your browser infrastructure every time. And the same problem comes with whenever a new operating system is launched. So for example Mac is going to launch. Mac just launched their new operating system Catalina. So every time a new operating system is launched, you have to update your all your infrastructure to match this new update. And of course.

Another challenge comes is in the execution of the test itself. So a cloud based service usually have a little bit of more latency because there is, you can say a cloud network involved. So every command thread takes a little bit longer to go from your local machine to the cloud infrastructure and then execute back. So test thread wise execution takes a little bit longer. However.

The advantage here is that you can create multiple different machines at the same time. You can test on multiple different machines at the same time. So that is called parallel testing. And that will drastically reduce down your test execution time. So for example, if you have hundred tests, each test take 1 minutes to execute. So for now it would take for example 100 minutes to execute your whole suit in a sequential. However, if you have 10 different machines and same you run all these tests on parallelly on these machines, your whole suit will complete within 10 minutes. So you cut down your test execution by a factor of 10 by increasing the number of machines you want to test on. But this again increases a new added. You can say, work for the DevOps teams that they have to maintain a test infrastructure of 10 different type of machines. They have to scale up those machines, scale down. They have to optimize the performance. They have to clean these machines every time the test is completed. So there's an added overhead related to the DevOps team, but it is offset by the fact that now you have a better browser coverage and better test execution times.

All great points. I know the company I used to work for, they were in healthcare and they had to maintain their browser versions for IE because they had customers that still were on IE 10. So.

If that's the case, do you find that often with companies, bigger businesses being having to deal with certain browser versions that older not even maintained by the browser vendor anymore? And if so, those still versions that you support yourself as a company?

Right, Exactly. So that is one of the main reasons as well. So for example, Internet Explorer 11, it is still like people ask me, like these are very archaic browsers, very old browsers, nobody is using them. But if you check the available stats online, around 1% people are still using Internet Explorer. So 1% is not a very big term. If your traffic is, for example, like 100 people are coming to your website or like 100,000 people are coming to your website. However, if a million people are coming to your website, that one person's amount to like 100,000. And that may amount to, we can say $100,000 of revenue. So that Internet Explorer suddenly becomes very important for them. So testing on an Internet Explorer browser, which is not even provided in the latest Windows infrastructure, it becomes a challenge. You will not find Windows Internet Explorer 10 anywhere in any latest Windows machine. So you have to go back and like you have to search the Internet to find that the older browser versions and Internet Explorer is a little bit easier because they had very few versions till 11. But what about Firefox? So when the Firefox started from Firefox 4 till now it is Firefox 76, Firefox 70s version that is still being used right now. So there's a 60 to 70 different versions that you have to collect and kind of maintain to make sure that you are covering all the type of users that may come to your website.

Absolutely. So I guess that's one extreme example is ie. But I guess another extreme I've been seeing is more and more automation tools that are coming out just support Chrome and not even support the other browsers. So have you been seeing this in the market and if so, any pros and cons for Chrome really dominating this market as of now?

Right. So most of the so Chrome has you can say market share of 70%. So quite a lot of tools bank upon this fact that like Chrome is the most important browser for you. And in some ways this is one of the main reasons why a little Selenium is still not, you can say have a hundred percent coverage. Like it does not dominate a full 100%. It is most dominant used automation framework. But there are others as well that are kind of giving Selenium a challenge. So, e.g. javaScript based testing tools like Cypress Zest Mocha, these are all kind of works with directly with Chrome. So they however do not have Cypress for example does not have browser compatibility testing capabilities. And this is one of the main reasons why Selenium is still one of the most preferred automation testing framework in enterprises. And so there are pros for Cypress as well. I will not say that Cypress is a blood platform. Cypress has its own users. It is built from the ground up to help front end developers in their testing. Right. It is very, very fast when we compare to selenium. It uses JavaScript which is like the language that front end developers understand they won't have to go to another different language and it works. You can see it has a very fast feedback and turn on time. Debugging using Cypress is little bit easier as compared to Selenium. But if cross browser compatibility or browser compatibility testing is important for you, you would have to move to a platform like Selenium which covers a broad scope of number of browsers. Like nearly all browsers are compatible with Selenium.

So again it's almost like a catch 22 though. I understand the reason why Cypress works for Chrome and I think it's a good solution for certain folks. But the other issue is if you're using Selenium and you're running it against Chrome and then you run it against Firefox the results might be different, so you have to do strange hacks through it. So have you seen, I understand that it'd be great to spin up all your tests against all the browsers and test all the things, but at the same time, I've seen certain functionality like hover over work differently in Chrome than it does in Firefox. So if someone's using Selenium, are there any workarounds for that or any best practices you've seen around that area?

So usually there's obviously a few, you can say hacks involved in making sure that your website works in all different browsers. And similarly, that goes with the testing itself. There are few functionalities in Selenium which works perfectly with Chrome. However, when you run the same code on Firefox, you would have to tweak the code there. This is something that you can say the community is working upon bit by bit and this problem will solve in itself when the differences between browsers will kind of close around. However, there's still, you can say you would have to tweak your code every time you want to run your test run. So there are other libraries that can help you in. You can say bridge this gap. For example, there's a library called Modernize, you can use them. Then there's a library called Babel js. You can use them to define how your website works on different browsers. But again, it's owners of the tester to make sure how their code is working with different browsers.

As a company, like I said, I love speaking to companies because you have a lot of customers and you probably have statistics. So for best practices, do you recommend people run their entire test suite against all the browsers or do you usually recommend, okay, you know, run a majority of tests maybe against Chrome and then have maybe, maybe 10 high level end to end tests that you've run against all the other browsers? Is that something that you even get into?

Right. So as an automation testing platform, we are usually integrated in the complete CI CD development pipeline of the tool itself. Right now there's a term that people use called continuous testing. Every time, every day. In fact, whenever all the people commit their code, an automated test using Jenkins and other CI CD tools automatically fires and it executes a complete test suit of that present code. People usually prioritize basis of that as well. So Chrome and Firefox, Safari, these three are the most important browsers. You have to run a test on them because they have the largest browser share.

Every time you put a commit, you can.

Run out a new, you can say infrastructure grid to run your test on. However.

You can delay the testing on other browsers, for example Edge and Internet Explorer. You can. Again, as I said, these are all based on the priorities. When we build up our browser matrix earlier, we define which browsers are most important for you and which devices are most important for us. Based on that, we prioritize our testing cycle as well. Again, most of these whenever we build a matrix, Chrome, Safari, Firefox are the most important ones. However, Internet Explorer and Edge are still used by a lot of people and there is a testing involved over that browser as well. But the comparative time is little bit less.

Besides cross browser functionality, which is a given benefit for using an online provider, are there any other benefits you see that people don't think about or know about or any features by using an online cloud provider, you get that you may not even think about.

So of course there's, you can say a significant cost involved as well when you set up your own grid. So for example, if you want to set up your own grid. So for example, let's take a simple example of.

Let'S take a simple example of aws, right? So aws, if I want to run successful tests using AWS and I wanted them to run in a like you can say decent pace, I would have to go with a machine that has around 16, 8 to 16 GB of RAM and kind of have you can say SSD hard drive. So these are usually what we call C5D 2X large machines that cost around 0.75 75 cents per hour, right? So but if we calculate on this basis, like I want to run this machine, I want to run 100 of these machines and I want to run all this test for around let's say 12 hours a day and I want to run them every like for 30 to 40 days. For my, during my six months of development process, the costing of this machine that is being maintained with AWS goes to like six to seven figures, right? Six figures in fact. So based on that cloud based provider has, who has optimized their infrastructure, comes out a little bit cheaper. They are you can say, because it's a kind of a shared infrastructure, the test that you are running after that those tests have been performed, those machines are cleaned and then they are being given to other users. So there is a, you can say a community of users who are using that infrastructure. So on a shared basis the, you can say test the margins of maintaining that infrastructure goes down. So in a longer term these Test service providers are cheaper. Another aspect is. So as I said, like if you want to run your test on 100 different machines, you would have to maintain those 100 different machines. So that requires a significant effort in both development and maintenance. Running 100 different machines will require you to build up, require you to use different, different type of tools like VMware, container docker. You would have to create templates for all these hundred machines. You would have to maintain those templates, you would have to upgrade those templates, and then you would have to create scripts that will automatically scale up those machines. And once that testing is done, they would automatically clean those machines so that during the next testing cycle those machines do not interfere with the already like with the new running tests. So you would have to clear all the cookies, all the release, all the history. So you have to create scripts for the maintenance of those 100 VMs as well. So this is the stuff that is done by our platform. We are eliminating the need for infrastructure development, infrastructure maintenance and infrastructure scaling up in house. We are eliminating the need for having a dedicated DevOps team just for your testing. That way you can focus more on building code and shipping products faster. You can ship products with better quality and ultimately write more tests.

You're the first person I've spoken to since Selenium 4 has been released and I'm just curious to know how a provider even handles that. Are there any new features in selenium 4 that you think are really going to impact maybe cross browser testing or testing in general?

Right. So if you are using Selenium right now, running the new release of Selenium 4 won't impact you that much because the new Selenium 4 would be backward compatible with all the Selenium code that you have already written. However, if you want to start off by building your new infrastructure, like new Selenium codes, new Selenium suits. From now you would have to take into the account the new protocols, the new grammar that is there involved in writing the code. So earlier the Selenium was using. So Selenium is basically made up of four different aspects. One is there the Selenium client libraries, then there are browser drivers and then there's browser itself. However, the fourth one is what we usually used to call JSON wire protocol. That is the, you can say collection of APIs that sends the instructions to browser drivers and those drivers run the browsers and the like all the instructions on the browsers. This protocol, this API, the instructions of APIs, instruction sets of APIs has changed a little bit. So.

Since the last four to five years people have been working on standardizing these API sets and we are now using W3C protocols, the set of instructions to send back to the browser drivers. A little bit different. There are new added capabilities as well. For example, there's a new thing called action APIs. Earlier it was a challenge to run multiple actions at the same time that was used in for example mobile devices. Mobile devices has two touch gestures to zoom in, zoom out. These gestures were not earlier possible. However, the new action APIs will help you perform.

Multiple different test actions at the same time. This is what is changing in the new Selenium 4. Another change would be the deprecations. Earlier. All the deprecated APIs that were there on Selenium, they used to give a warning. Now they will start giving you an error. You would have to make sure that your Selenium code is updated to at least 3.4 or 3.8 or 4.0 standards. If you're using very old versions, you may have to change a few of the APIs. But apart from that everything works perfectly. It is very backward. Selenium 4 would be completely compatible with all Selenium 3. X versions and you can use them as it is.

Before we go, is there one piece of actual advice you can give someone to help them with their automation or cross browser testing efforts and let us know the best way to find, contact you or learn more about Lambda Tests?

Right? So the best information that I can, the best piece of advice that I give to.

Every, even enterprise clients as well is to build up their browser coverage bit by bit. Start off with Chrome, start off with Firefox, Safari and then see how many users are coming to your platform and based on that increase your browser coverage bit by bit.

There is a movement you can say, a way to automate each and every aspect of the testing. But you cannot overlook the fact that human testers are still the best eyes there is. So you cannot overlook the manual or exploratory testing part as well. So you would have to make sure that you as a developer are giving at least three to four hours a week on making sure that your application is working as indebted.

If you want to know more about LambdaTest, you can check out our website LambdaTest.com we have in app in website chat support as well. We are very active there. Anytime you face any issue in using Lambda Test or want to know more about cross browser testing, want to know more about automation testing or Selenium just ping us there and there will be somebody out there to help you out. Even I am they are very active on our in app chat support.

Thank you Mude for your cross browser automation awesomeness. For the accepted value we covered in this episode, head on over to testguild.com a like Automation 275 and while you're there make sure to click on the Try it for free today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating Full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.