Pylenium.io with Carlos Kidman

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Carlos Kidman all about Test Automation and Pollenium IO. Carlos specializes in creating test automation frameworks for UI integration and service test. He focuses on scaling tests and empowering developers and testers with CICD tools like Jenkins, Docker and Kubernetes and works closely with the Infrastructure and DevOps organizations. Although he is currently a QA Manager at Workfront, he is also very active in the community. He is the author of the Test Automation University courses from Scripting a Framework with Selenium and C Sharp and Scaling Tests with Docker. Carlos is also the founder of QA at the Point, a board member of DevOps days, and open source developer for Pilonium, which is a python based automation framework that we're going to talk all about today. So you want to definitely check this out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Carlos, welcome to the Guild.

Thank you, Joe. And it's awesome to be here. Thanks so much for inviting me.

It's great to have you. I've been following you for a while, I guess. Before we get into it, Carlos, is there anything I missed in your intro that you like the Guild to know?

I would say like a couple things. Number one, I love soccer. People have to know this about me. Soccer or football? Let's say football because I'm saying soccer because I'm in the United States, but I love me some football. Sorry for the Manchester fans because I'm a fan of Barcelona, but hey, let's go. And then the other thing too is I'm a big fan of like trading card games like Magic the Gathering and anime too. So things that are not like testing or coding related, you know.

Very cool because I think I may have missed. You're also a host of a podcast. Could you just tell us a little bit more about that?

For sure, yeah. Actually it's for the Ministry of Testing, so the Ministry of Testing podcast. I'm one of the hosts for that. So tune into this one first, of course, and then switch over to Mother.

Already we have great actionable advice. So why another framework? Carlos, I know you've worked with a lot of different tools before, and currently, so why. Why choose to create another framework?

Honestly, excellent question. Because I asked myself that before I started it and I'll say, like, part of what I do in the community is here in Utah. My organization, QA at the Point. We are very heavily invested in the different organizations here, like women who test coding color, but also trying to get it into high schools. So we partner with the board of Education, as well as Code to Success, to try and bring more like programming automation and testing courses to our youth. And so as I've been doing that, I ran into the situation where Selenium, as much as I love it, is actually kind of clunky compared to Cypress. Right. Cypress has a great onboarding experience. It just feels great. You get started so quickly. And I couldn't do the same thing with Selenium. And so what I wanted to do was I was already teaching these kids how to use Python, so I figured, what if I just made the Selenium experience as awesome as Cypress, but still had the flexibility and scalability of Selenium, and that's why I made.

Very cool. So Pollenium, it sounds like, is it a wrapper on top of Selenium that makes it easier to work with?

Correct. Yep. So thinking of Cypress, the whole point of it was to make it so I could just install a single package and start writing a test. I didn't have to worry about any of the different types of weights. I didn't have to worry about my driver binaries. I didn't have to worry about any of that. We could get to the real meats and potatoes of programming and test automation and just get them into the. The content much more quickly.

So when you say youth, what's the age range we're talking about? You were teaching?

Yeah, 12 to 18.

So did you try them use Cypress first? And do they have better luck with that?

They did okay with it. The problem was JavaScript is not as easy to learn as Python. So, yes, up to a point. And then it just became another hindrance where the questions they were asking, like, why won't it just run this line? It got to a point where it's like, okay, hold on. How about let's just stick with the language that we all know and love. Let's do Python where it's already synchronous out of the gate. And again, we could focus more on the problem solving, the analysis, the design, Rather than I need to get this one line to run first, you know, Perfect.

And do you also do something with an environment like a virtual environment? I thought I was reading the docs and it says something about just you have a virtual environment. And is that a way to get up and running quickly without having to do all this installation?

Yeah, in Python, the virtual environment is essentially, it keeps all of your packages and dependencies inside of your project, so it's all self contained. Without a virtual environment, all of your installations would happen globally. And so multiple projects might be sharing dependencies. And that already causes a bunch of headache. And so the only requirement really to get started with it is have a virtual environment, which is really easy to do, and then install Pylenium IO.

Very nice. So can you just go a little more detail of how it saves time, a little more about the weights. It already has built in logic to know that when I click on an element, maybe it waits for the elements to be visible before I click on it or things like that.

Yeah. And it's not like a bunch of voodoo magic or anything. Right. Like we said at the beginning, it's just a wrapper of Selenium. And so we're just wrapping the explicit weight, which is just webdriver weight. We're essentially waiting for the element to exist in the DOM and then proceed from that point. And so you can customize it like, I know I'm kind of simplifying its usage here, but you can customize the weight per command, per finding an element, per click. You even have like dot should, so you can do expectations or assertions on different behaviors, and you can control those wait times as well. So you actually have a lot of control over it if you'd like. Otherwise just using it by default, it. It works pretty well right out of the gate. So there's lots of flexibility and control.

Talking about control, I know a lot of people also struggle with getting the test run in parallel. A lot of times they think about it after the fact and their frameworks are usually a mess by then. So I think this. Does this have something built in to help you run in parallel better or faster?

Yeah, the way that pytest, which is the testing framework I'm using, the way that pytest works is you have these fixtures which are essentially these special functions that let you do a before and after. And so by default, when you use the just PY or the PI fixture, it's already built for you to have all of your tests be isolated within each other. So out of the gate. If you. Once you install Pollenium, there's not really a way for you to make it so your tests aren't modular. Like you kind of have to code your tests in a modular way and Pollenium will give each one of your tests its own driver. So right out of the box it's ready to be parallelizable.

Nice. Now, I'm not saying we're going to choose or say what tool is better than the other. Just curious to know how it compares against. Don't know. Have you used Selenium base before? Because it sounds similar to that. I don't know if it's the same, different. If you tried it, didn't try and if so, what the differences are. If someone's trying to decide which one to use. If they're really into Python.

Yeah, yeah, for sure. I actually have used Selenium base. It was. I used it first before I built Pollenium because I was looking for something already there that I could use. What I'm not a fan of and man, the people that worked on Selenium base it awesome for what they were trying to do. So I don't want to like bash or anything. What I ran into though was I wanted not just like this whole experience of like being able to just write the test in the way they wanted. I really just wanted a simplified, easy to use, intuitive experience with Python and I don't get that with Selenium base. So selenium base I have to do like. It still feels like Selenium. It doesn't feel like there's much of an improvement. On top of that, I still feel like I have to wrap my own things. I still feel like I have to extend it anyway. And that just added to more work that had to be done where with Pilenium. And I'm biased obviously, but with Pilenium I haven't really had the need to do that. I could, if I wanted to, I could essentially screen to test, quote, unquote. And it works as if I have like a whole framework behind it that's running it. Yeah, I was going for more for that, like tester and developer experience, more so than give me all these reports first and give me all these other cool things as part of it. I went for the experience first and now I'm going to start adding things like reports and visualizations to it later on.

So with Pylenium, because this is audio, can you give us like a visual idea how you would create a test so people get a flavor for maybe the syntax, how it May be different than Selenium straight up.

Yeah. So the way that I would start it is you just make your test file. So after installing Pollenium, make a test file. We'll call it like TestGoogle Py, right? We'll make our test and then we'll make a test function. We'll say def testgoogle search and then in those parentheses just put PY for pylenium. And then from there we would just say PI. Visit google.com, pI get name equals Q typepuppies, let's say. And that will type in the word puppies in the search field and execute it for you. And then on the next page we would say like PI title contain puppies. And literally that would navigate you to Google. Enter puppies, do the search, and then check the next page has the title of puppies. And just like that, like it's that quick.

Awesome. And like you said, I think Python's a great language for folks to learn because it doesn't have all that boilerplate code you need for, say, Java or JavaScript. So is anything you learned from your students from 12 to 18 that not only from an automation perspective, maybe some other concepts that they might be struggling in that you're trying to bake into Pollenium? Don't know if that makes any sense, but like you said, sometimes we get hung up on the syntax and running the test, but I don't know if there are any core concepts automation wise. You think that people struggle with as.

Well, with how simple Pollenium is. A lot of the things that we were stuck on beforehand, like for example, putting functions on the functions or chaining or things like that. With Selenium, it was a little hard to understand. Like when we say driver, find the elements with all this stuff by css, the string inside of it, then outside of those two parentheses, then you can do a dot click, stuff like that. It just was harder to understand. Which parentheses do I put it in? Is it after the first one or the second one? What is a locator versus a selector? Like that kind of stuff was what we were really hung up on and what we were teaching. Where with Pylenium you would say PI get and then click. And that just made sense to the students. Another piece that I think even with Pylenium is still something that's hard to grasp would be object oriented programming. Once they get a feel for scripting the tests with Pollenium, then we start talking about refactoring and doing things like the page object model or Just using different modules and functions. And that's the next part that gets a little complicated. Because once you introduce class this page and then def in it and you make a constructor, all of a sudden that's a whole different. Like, we've stepped into a whole different zone of programming. And everyone's like, I used to be smart, but now I don't know what I'm doing. And so it's been that kind of stuff. But what's been nice is these students that, like, let's say even just 12 years old, when they're looking at it for the first time, the questions they ask are really, really deep. And it's something sometimes even super simple, like, why couldn't we just have this. Be this one line inst. Why can't we just say dot, click? And when I was using Selenium beforehand, you look at that and you're like, well. And you don't have a good answer. You're just like, huh? Yeah. I mean, well, because you have to do these three things first, and then you can click on it, and they're just like, I don't get it. That doesn't make sense. That seems so hard. And so even when I was building Polynium at first and I was sharing it, like, with other people to try out, even some of the things they were saying, like, I don't think that should have this. What if it was just dot should with parentheses? And if they wanted to add anything, they could. And the whole time just writing like crazy, like, okay, this, this and that, you know, and these are just kids tell different ideas as to what would make it easier for them and what makes sense. But now that I'm using it for QA at the point and our Autobots weekly training, those kids, like, what they taught me and what they showed me and the ideas they gave me, have actually made it easy for anyone in our community to pick up Highlandium as well. So it's. I love that those ideas and them trying to simplify, to make it easier for them, really is making it easier for everyone else. So it's pretty cool.

Nice. So a lot of things I also know people struggle with is locators. Which locator to use, which one's the best one? How does Pylenium handle that?

So actually, Crowpath handles that. No, I'm just kidding. No, I'm just kidding. So, I mean, you have like the PI get or PI Find. We'll use a CSS selector. You have a, like, PI XPath if you want to find elements with XPath, but otherwise like there is a PI contains function which will find you an element that contains the text that you pass in, but it doesn't really solve the problem of making that part easier for you. So you would still need to know how to find elements with CSS or find the elements with XPath. So that's definitely still something that we have to teach the students is how to do that.

So I must sound like a really grumpy old man, but when I started I had a tool called Windrunner and Windrunner did all this. It did all this and now 20 years later people are coming up open source solutions and they still lack what I had in my vendor tool 20 years ago. So I love how you're working towards that vision though. Any other things on the roadmap that you mentioned, like reporting I guess you said is one or any reporting in Pylenium as of now?

As of right now, yes. But it's all local so we do generate like a test results directory for you and we have logs for everything you've done. We take screenshots on failure for you. Again, all out of the box. But we don't have anything like creating HTML report or something like that. You can add that pretty easily with Pytest because there's a bunch of plugins to do it, but it's not something that just comes baked already with Pilonium. So definitely like visualization and just data. Better data reporting is something that we like and then also just integrations with other types of tools from the community. So like right now we don't have any accessibility or visual testing. It'd be nice to have that out of the gate. It'd also be nice to be able to integrate into tools like Browser Stack or Sauce Labs much more easily. So that kind of stuff is what we have in the roadmap right now to just make integrating with other things easier for you so you can plug and play and use whatever tech stack you have.

Nice. Now this also might be controversial. I like C. I like it better than Java and I know you have a course on scripting Selenium with C, so I just curious to know why Python then over C? Is it just a matter of I love Python, but is there any pros or cons why you would start students off with Python rather than C?

Let's see. And I love C too. I am most definitely a Net core guy. Love it. And I do agree, I do like it more than Java as well. Yes. Right now on stream every single weekday When I'm streaming on Twitch, if we ever have time, we'll pull up a project called Silenium, which is the exact same thing as pilenium, but for C. We are working on it. But the reason why I started with Python was a few reasons. Number one, my current job at Workfront, we are working in Python. So it's just the thing that's at the top of my head all day long. Number two, it's what we were teaching in Autobots, which is our weekly training. And then also for code to success again, it's just a seamless transition of, hey, let's stop using Python with Selenium and let's just use Pollenium. But that's really it. Otherwise I really like C Sharp as well. So that's why I'm like, I got to do this for C. And we're working on Silenium, but it's. Yeah, Python's awesome. So is C. They're definitely my two favorite languages.

Nice. So, like I said, you do have a croissant from scripting a framework with Selenium and C. Any core principles out of that that would apply to any languages. How would a frame that you think everyone should know about when creating a framework?

Yeah, even though that course was in C, it definitely was something like all the concepts in there would apply to any language. One of the things that I think people have a hard time with for some reason is simplicity. And granted, to make something simple is really hard, I get that. But I think one kind of anti pattern is I see a lot of people that want to write the page objects first and then write the text. Right. And I like flipping that and being a lot more TDD where I write the tests first and only do the things that I really need, only find the elements that I really need. And then when I refactor, I'm not adding any more bloats, I'm just. I'm only adding what I used in the test. That right there alone simplifies it quite a bit. Another piece is naming your functions. Well, I see too many times where someone will have like homepage dot click on this element. Why would they call a function click on this element? That's not how we talk to our users. We say, hey, this user on this page, do this action. Add to cart. How many of this item do you want? Add to cart? Quantity equals 3. Add to cart, this thing or cartpage go to or anything like that. Just looks a lot more readable already out of the gate. Rather than cart page dot click Drop down dot, click this option, cart page dot, add this cart page dot. Like it's, it's a lot of lines of code and I see these tests that are massive when they should really just look like a user journey that anyone on the team could read. And so because of that like weakness in naming and weakness in refactoring, that's where the prevalence of other frameworks like BDD for example, are getting a lot of popularity because that to them is now this is English readable, this is easy to use when in my opinion you could have already made it easy to use without needing that. If you just worded or named your variables and functions well at the get go it would look like BDD in the test without having the whole BDD abstraction layer on top of.

I love what you just said about tdd. You know, you either have people that over architect a framework or do no architecture of the framework whatsoever. And sometimes teams will spend like six months just architecting it and having these, like you said, big bloated page objects that have everything conceivable you can do and you don't touch it and it just makes it a big old mess. So I love that TDD approach.

Yeah, I like to be as lazy as possible. So I'm only going to write exactly what I need. I'm not going to write any more than that, you know.

So I wonder if that only happens in large teams or enterprises. When you have a bunch of different teams working on a framework at the same time. Do you have any tips for working in teams or is it usually like a one or two automation type team that what's your normal flow when you're creating a framework?

Then yeah, my normal flow would be number one, the first thing I have the teams focus on. Like I'll use Workfront as an example because I just barely did this. This is a large enterprise. You can think of them as like the Trello or Jira for like huge enterprises. So our customers are like Microsoft and Google and we are the Trello but for these massive corporations. Right. And so our quality organization is like a hundred people. Right? So a big team. This isn't a startup or anything. So the very first thing that I have them do is I have them tell me what the product coverage is. What is your product coverage and how confident are you in this product coverage? So nothing even test automated related yet. Like we're not even thinking about automation. What I want to understand is how do they see the current coverage? How do they see where the tests are at do we have unit tests? If we do, do they make us feel good? Do we have API tests? What services do we have? What areas does my team own? Like, all those kind of questions. I want to have that in a visual way, because that visualization is what's ultimately going to give me my roadmap. If I have five areas in my app that my team owns, my team is maybe like 10 people, right? I have my front end and my back end, my product manager, all that good stuff. If my team owns five different areas, how do I know what behaviors, what actions, what functionalities these areas are supposed to have? Do I know why we created this area in the first place? What its goal is supposed to have? Is it actually accomplishing those things? And do I have the test to meet those things? So we're not even thinking about automation yet. I need them to understand that piece. And usually what ends up happening is it's always a comedic effect where we'll ask the team, all right, quality engineers, I want you to go to the rest of the team and ask them, on a scale of 1 to 10, how confident are you in our team testing? Right. Just real quick, real simple question. And the best part is, this is usually what happens. You have your developers that'll say something like, oh, gosh, like a two or three. I wrote some unit tests once, maybe a four. Like, it's never above a five. It's always really low. And then you ask the quality engineer the same question. So how do you feel about the testing strategy we have? Oh, it's like a 9 or 10. I wrote the thing and they're super high. But why that discrepancy then? And so it's kind of trying to get the team aligned into what the team's dream product coverage is, the team's dream test strategies, aligning ourselves. Because we don't want to just write automated tests for testing sake. Right? We want to have a goal in mind where we want to get to, but we want the team to also be aligned with that. So, anyway, that was a long winded response, but I start with that first, because once we understand that, then we can really say, which areas do we feel good in. If this area is a 10 out of 10, I'm not going to focus my automation there right now. I'm going to focus on this other place that has more gaps or less confidence and either optimize that or add tests there and then continue with what we talked about before with the tdd. Let's write the first, then let's optimize it and refactor and that kind of stuff. So start with the product coverage and confidence first. Align the team with the vision we have for test strategy, including the testers for sure. And then executing on that.

Yeah, another great strategy. It's funny, low tech things, talking to people before you actually start coding can be a big help for sure.

And one last piece of that too. I think it's really interesting. I've gotten this comment a lot of times now where when people watch me automate or they watch me like write an automated test, right? The comment they say is, wow, you do more testing than coding. Which is something I think is really interesting as well because it's right. Like the coding part of it for me at least is not the hardest part. The hardest part is understanding why I'm writing this test. Do I even need this test? Is it at the right place? Am I actually mitigating the risk? Like there's all these other questions I have before I even just write the code. So once I have a great idea, okay, this is what I want to test and how I want to test it and why. The coding part of it's not that bad, but the design and the analysis, all of that is the, to me, the more fun, the more creative, you know, part of testing.

You're hitting all my triggers today. Another pet peeve of mine is when you have a team and developers doing automation but not necessarily testing, and you get just a lot of automation scripts, but they're not necessarily adding any value. So I think, like you said, that that's the perfect example.

Yeah, we actually had a team where they were showing us when we did the product coverage for their team, they were showing us they had like 90% coverage, right? We're like, wow, that's really awesome. But immediately we saw that and we had this, I forgot the name of it, but you basically have like this cross metric where we see 90% product coverage. That looks great. But then here's all these bugs though in that area. Wait a second, that doesn't make sense. If you have 90% coverage here, this should be the area with the least amount of bugs. How is it the most. And so when we asked the developers, how confident are you in it? They said, oh, really high. The testers, well, really low. And so when we showed them, here's this area and here's the behaviors that this is supposed to do. It's supposed to allow a user to add a team member. It's supposed to allow the admins to Manage the responsibilities and roles of these team members, these different actions. When we showed them that the developers went, oh, crap, well, our tests don't do that. And we're like, wait a second. Well, then what are your tests doing? Right? If you're not testing the functionality of your code, what are you actually testing then? And they showed us they were taking snapshot tests. So they were actually just rendering a component and then taking a snapshot. Does it have the right HTML and css? So they weren't checking any functionality, really, just the styling and did it render. And so once they saw that we were actually looking at behaviors and actions that a user could take, they immediately said, oh, well, now we're back to a one in confidence, because, yeah, none of our tests actually do that. And it was a big light bulb moment for that team.

It's great that they were at least doing what they were doing, but I think we're both of them. You got a good coverage there at that point. So it sounds like you do a lot of analysis, Carlos. And is that part of your CI CD type background? You're not heads down. You're more looking at the big picture, looking at code coverage, where maybe some other testers, automators, may not even look at that type of information to help influence how they're going to automate.

Yeah, I would say, like, as in my career as I've been, I started from when I was just grabbing a task. I was a junior, and I was just automating whatever got on my plate, right? And my goal at that point was really just how many tests can I automate? And so the number I'd used to brag is, oh, man, today I wrote 10 tests. The next day, oh, I have 2,000 tests. And to me, that was awesome, right? I have so many tests, and that makes me a great developer. That makes me a great test. But as I started going more into cicd, what I saw was, crap. I hate that I have 2,000 tests. This is taking a long time. It was making things worse, the more tests that I had. And so kind of naturally, what ended up happening was I had to prioritize which tests I really cared about at which layer. Because if I really want the coverage that I'm getting from this UI test, but I have too many UI tests, I'm gonna have to start deleting some of the tests. And so instead of deleting them, I said, well, I want this coverage. Let me lower it down then, which started teaching me how to do other types of languages. It taught me how to do API testing and unit testing taught me how to do mocks because I wanted the same amount of coverage, but I didn't want everything at the UI layer. And so I had to start lowering them down. And so as I started monitoring the pipelines and the execution times, what I wanted was to get again the same amount of coverage, just faster. And so I think that's really what started forcing me to think about more like priority and risk. I can still mitigate that risk. Am I doing it at the right place? It doesn't even have to be automated. Like, can we mitigate it a different way? And trying to get creative with that as well. Because sometimes automation is not the answer and you can still have the product coverage without the automation, you know, so there's a few things that you can do there. Another piece was monitoring. In my mind, monitoring is a form of testing as well. You're really just checking states of things and looking for different things as well. And sometimes it's a check, you know exactly what error to look for. Sometimes it's more experimenting and more hypothesis. I think this new feature is going to cause these kinds of failures. And so you kind of put a hypothesis out there and eventually you need to get rid of that hypothesis. You need to answer it. So all that kind of stuff is really what I feel like made me a better tester and developer overall is when I got into the operation side of things, I had to be very, very picky with the tests at which layer to get the best feedback the fastest way possible.

I could talk to you for a long time here. I like to keep these episodes the 30 minutes because that's what people tend to like. I have one last question I ask everyone, but before I ask it, is there anything you wanted me to ask you or you think I definitely should have asked before I asked my last question?

I think you did a real great job, man. Really what I would want to leave is just more like if anyone that's listening has any ideas as to help out, not just the youth, but also like underrepresented communities. We're trying to do a lot of help around our community here in Utah, but if we can help anywhere else as well, that'd be awesome. One thing that we're doing at Workfront is we're donating laptops to people that don't have them. So really, anything like that, please reach out because we are trying to make a difference and help out where we can. So please just hit me up with.

That before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing and efforts? And what's the best way to find or contact you?

So the biggest thing I could tell someone is use social media to be honest, like get on Twitter. You may be like, nah, I'm a boomer, I don't want to do that. But you should get on Twitter for real and follow some people. Another thing that I would say too is as you're learning Automation, find the YouTube channels that you like, find the authors that you like, find the leaders that you like, and then just start subscribing to everything that they have. You really want to expand your mindset and not just get sucked into this is how my company does it. So I have to do it this way everywhere because I see that bad practice happen a lot of the time and it's probably what limits people the most. They hide themselves from the rest of what the world is doing. That's a terrible idea. So fortunately everyone listening, you probably already doing this. But take advantage of the Internet, take advantage of social media and follow the people that you think are going to help make the biggest impact on your career.

College. You're one of those folks. People should follow us. What's the best way they can contact your phone?

You, I would say Twitter again, you just find me on Twitter or LinkedIn. Just arloskidman. Just send me a friend request, write a post and hit me up. Like I would love to talk. I love talking to people. So ask me any questions or challenges. If I can't answer it right, then I will answer it on my Twitch stream. I do it every weekday at noon, Mountain standard time. We literally take any questions or challenges from the community and we try to get it done within lunch. So please send me your stuff, I'll help out. If you want me to check out a project you have, you want me to do a code review on something you're doing anything thing, I'm here to help out.

Thank you Carlos for your automation awesomeness. For links to everything of value we covered in this episode, head on over to test guild.com a303 and while you're there, make sure to click on the try it for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation podcast. I'm Joe and my mission is to help you succeed with creating end to end function full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.