Avoiding Common Testing Pitfalls with Ramona Schwering

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Ramona all about avoiding common testing pitfalls. Really excited about this talk. I saw her give a similar kind of session at Future of Testing this year around these type of topics. Really cool slide deck as well. So I'll have a link to that after the show is over. But if you don't know her, Ramona. Her apprenticeship started as an application developer and Ramona contributed to product development at Shopware AG for more than five years now, first in quality assurance and now as a developer core. She has both views of the product, which is rare. So she's both a developer and tester. So we get to see how a developer really thinks about testing and so really excited about that. Ramona uses this primarily as a strength obviously to trust in test automation and knows how to, you know, create great test to help support the testers as well. And the automation in the end to end area of shopware originates from her pen and she continues to push it firmly. Really excited to have her on the show today. You don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com forward/test and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Ramona. Welcome to the Guild.

Hello. I'm so glad to be here.

Awesome. Awesome to have you. Ramona. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

Well, I guess you summed it up quite well. I see myself as a tester and a developer alike, so I hope I can, yeah, share experience from both world and to get both roles together a bit more, especially when it comes to testing.

Awesome. Now why do you feel that way? Do you feel like most developers you speak to don't have it both together? Or is it just because you have a unique take because you've done both sides of the aisle?

Well, it originates a bit from my beginning where developers and testers, at least it was my impression that they just worked a bit against each other, not sharing the same mindset or working as a team and yeah, kind of understanding each other and feeling more love than hate. So it was a huge motivation for me to get them together and get them to like each other. At least in my team I think I was quite successful and I hope to share it with you all if you need it. And yeah, because we both, we developers and the tested, we do both want the same things, having a wonderful experience for the customer and for the users. So I think we should keep that in mind sometimes.

Well, now that you mention it then how did you do that? How did you or how do you help tell people any tips on how they can help their teams maybe to embrace testing beyond just testers?

Well, we consider testing like a coaching role. So if the developer was finishing his implementation or her implementation, they got together with the tester, talked about the new feature, the task or whatever they did and thought about the thing they might have forgotten to test. So the developer learned to test better and the tester learned to understand the developer's point of view. So they started to share the same perspective. And I quite love to see that.

And so I'm just wondering if this presentation came out of that experience because I think maybe I'm wrong. Developers maybe aren't very familiar with testing so maybe they have certain practices that make testing painful. So therefore they associate bad practices with testing and therefore testing is bad rather than maybe changing an approach to make test more of like a safety net for them before they release code.

Well, I think both things with this impression and especially my way to become a developer and getting to know especially unit testing and the mindset behind of it. If you wait test tests as a tester or in quality assurance, you have more a user focused perspective. If you here I love to use the work workflow based for end to end testing because you try to write them for the user and the user's behavior. If you are writing unit tests you might have another mindset and it was quite interesting for me to see the difference but also the yeah the things which are basically equally for both very cool.

And so ever since your your presentation I can't get your impression of Admiral Ackbar out of my brain. Yeah. And the reason why is I guess you have a concept in testing called it's a trap. And that's I guess you we build off. So what did you mean by it's a trap? When you gave that that particular that.

Session I saw myself sometimes going the simple way to use more convenient or easy ways to write my test thinking oh that will be fine. It's short, it's not exhausting. And I thought I would be good doing that. But sometimes those easy ways became like kryptonite for me because they had consequences, let's put it that way. May it be low performance in my tests. Flakiness, which is my. I hope it's the good expression for it, final boss to conquer. And yeah, sometimes I was a bit. Yeah, the past Ramona was sometimes a fool to believe Easyboy would be the best one.

And I'm sure it's not just you. That's probably. Everyone thinks it's going to be easy for testing and all of a sudden over a few sprints you're like, oh my gosh, what have I done? I think there are three main points or three pain points you've mentioned that maybe cause that situation as well. Can you talk about the three pain points? I don't know if you remember what they are, but there are a few pain points that you associated with this kind of trap.

Oh yeah. The first one being slow tests or at least the slow execution time of those tests. Sometimes teammates came to me and were really annoyed by the waiting times they got when they want their pull request to be merged or something. And I can totally understand that. Especially when you're working in the ecommerce sector as I do, and your software you're working with on the daily basis grows bigger and more complex. You have a certain amount of tests and a certain test coverage and this can be. Yeah, many tests need many time to be executed and so you really need to keep an eye on it being the least painful. This was quite bad. Especially when you are writing tests in a not that effective way or not that efficient. Another one which I stumbled upon lots of times were tests that are difficult to maintain. Imagine this case and you're looking at your old test you wrote maybe one year ago or something like that. And you have no clue what you mean by that, what you wanted to achieve with that test or why do you wrote it in the first place. And yeah, I don't think that a test you don't understand gives you any value. So that's kind of problematic. And yet the third one, which I call the final boss is flakiness. These are tests who are sometimes patching, sometimes failing without you changing anything in between. Yeah. Giving you no consistent value or result at all. Giving you no clue if your test is actually good or not. And this is a point I spent hours debugging. And yeah, it was a bit of a nightmare for me.

Absolutely. And I think it trips everyone up for sure. What I think is interesting is I started years ago, 20 years ago using a tool called WinRunner and these same type of pain points were there runner but people always think the next it's always the tools problem. Like back then was Windrunner, then qtp, now it's Selenium is causing these and then other tools. So do these apply to any type of testing? Like I know you use Jest and Cypress. Are these type of pain points do you see existing in all types of automation or all type of testing?

Well if you are asking the question on if I encounter then I have to admit yes, even with just in Cyprus I tend to stumble upon folkiness as well but not that often as I experience at the Selenium or Nightwatch or any web driver based framework they might come up with any testing framework especially if you think about being product cited courses like Race Condition or another bug which is happening sporadically and in this case you really need to take them seriously because think about every one of five builds maybe failing, how many users will encounter the bug so it they don't need to be the test itself.

Absolutely. When you're talking about your approach to testing, does it apply to both unit testing and functional automation testing? When people hear developer they just tune out and probably think they only do unit testing and may not realize you do all end to end testing.

I think it's alike. You can meet some traps in unit testing or integration testing and of course in end to end testing. I think it's more common in end to end testing, but that may be biased by my experience. I assume it's because you need the whole tech stack to be available in an end to end test. But I counted some traps specifically tailored to unit testing so it might happen.

Cool. So I'd probably like to dive into the three pain points and then how you kind of solve them or how you found kind of workarounds to make it better with the pain points. The first one was the golden rule for testing. What is the golden rule for testing?

Well that was an approach I met when writing the JavaScript testing best practices by Uni Goldberg and I highly recommend you to read this. I think it's a GitHub repo or something like that. It's a wonderful approach to any JavaScript based testing no matter the type. And I think the golden rule is that you should be able to get the intent of the test instantly when you look at it at first sight, at first impression. It should be so easy to read that you actually don't need to think about it. Your test shouldn't be eating up some your headspace. Any exhausting thoughts, anything. When you work, you have enough things to think about when working with production to code. So test shouldn't be a problem. In this case, it's like a routine in the best case.

It's a good point. Some people may think it's trivial. I forgot who told me this. They said most of the time is spent reading code than actually writing it. So if you can't read it, it's going to be a nightmare to debug. Even if you create the code yourself, six months later you're at like what was I talking about here? Right, so.

Oh yeah, that's a great point.

So I think another thing you brought up which is interesting is you said something, I think it was automation code is is not production code. I think people may get this mixed up. A lot of times people say automation is is software development, but then when you say it's not production code, they might think, well, you're minimizing it. So what would you mean by automation code is not production code?

Yeah, that's a wonderful point because sometimes in my talks I like to seed up some things to discuss about. So I really be on on site conferences later on some point run. Hopefully when COVID 19 isn't that much of an issue. Maybe. Hopefully. You know what I mean. So this kind of sentence are perfect to discuss with the participants. I had a wonderful discussion about that later after the conference in Twitter as well because, well, I was a bit straight when I told it's not production code. Let me rephrase that. Of course you should treat testing code like production code with the same value. We shouldn't say, oh, it's just testing code, we don't need to have a look at it. Actually this is a trap we went into when it comes to code reviews in our company and sometimes I read some tests at our repository and I think, oh my God. Who want to prove that? Because they didn't think of test code being so important than production code. So don't get rid of one in this point it's equally important at least. But there are some patterns, some conventions which are important for production code, but not that suitable to test code. My favorite example is the principle don't repeat yourself, the dry principle, which basically tells you you should avoid the application at all costs. But if you do that in tests and you make it more difficult to read and understand the test at first sight because you lose lots of abstraction for Example or less documentation or comments, you make it more difficult to understand the test in the future or to maintain it. So this pattern might be conflicting to the Golden Rule and thus you should yeah. Rethink it and maybe use some documentation or some duplication and say oh it's okay if I can get the intent of the test better this way.

No 100%. I love this point because sometimes we make things over complicated thinking that like you said, we're not. You're not minimizing automation code. You should be following best practices and such. But the certain production code practices that don't come into play or shouldn't come into play with test code doesn't mean it's less than. Anyway, I just want to make that clear so I don't get any hate email on my end either. So yeah, he also made a good point of test only as much as you need. So I think sometimes people go to one extreme or the other. Either they don't do any testing or they go full blown. I'm gonna have 100% coverage and it's a nightmare. So where's that sweet spot? What are your thoughts on test? Only as much as you need actually.

I did a worksh workshop of this topic with some testers today where we thought about okay, especially when it comes to the end to end testing area, what should be tested as an end to end test, which is better written as a unit test or what should you not test at all? Especially people with qualitative background tend to test too much. So your test runtime gets horrible and yeah, you're wasting resources for some details you don't need to know and you wouldn't test manually at all. So my sweet spot would be two aspects. Consider how likely it is for a user to encounter an error. So see the happy path, the crud operations of a feature, all the things users tend to use a lot there an error would be quite obvious and quite annoying. The other aspects I love to use the word critical path in there. It's a workflow, it's a path where you can't afford any damage, where damage would quite expensive or have high risk for the user. In E commerce sector I always think about registration login or the checkout it was broken in a release our customer would lose lots of money. So having it tested in more detail is quite important.

No, absolutely. I see a lot of people don't realize risk is a really good I guess indicator of where to focus most of your time in it for sure. I think you also mentioned Once people know where to focus their time and once once they know the risk and what they should test. I think you have a framework almost a test should contain three different things. Can you explain a little bit how you see testing what those three things are that you think a test should have?

Yeah, this is a pattern I saw in the Internet. At first it's the triple A pattern but it's more suitable to unit testing because they tend to be more short. Of course I will get to these 3x bit but it's more difficult for the longer or taller nt test to see all the 3x things clearly. Abstract lecture but if you read through an end to end test you find those three aspects as well. It's about range, act and assert arrange should be. When you think about an end to end test, the first part of your test and in it you define anything you need to start your scenario. For example declaring variables or doing anything before a test. The second one being the action where you do the steps of your test. Anything you need to do to test your scenario. And the second and quite self explanatory one is assert. You assert on the status of the thing you test to get some information of the status of the things you test and to see if anything's okay or anything breaks. It's an interesting point because I had a discussion with my colleague shortly after the talk. When it comes to the unit or component tests we write at our daily business, this AAA pattern cannot be used 100% because sometimes we need to assert the state before we act before we test. So we had quite exciting discussion about that and came to the to another pattern called even when then pattern it's a bit refers to BDD behavior driven development where you define at first what is given so you can assertion before test because it can be counted to this given area. Then the when area where you say what you expect when something happens and then would be your assertion. So it's a bit another approach but aiming to this to the same result basically to have a clean test structure and to make it easier to understand your test at first sight.

Nice. So once you have like the pattern in place you have what you think is a clean test. I think another thing people get tripped up on is test data and that's a tough tough. A lot of people really find this as an issue. Any tips around test data? How to how to work with that when you're running tests in parallel especially.

Well, a little disclaimer first I came to one approach I I can accept. I'm quite satisfied with it but it's not perfect. And if you got any advices especially when it comes to anti testing, please contact me on twitter wherever because it's a topic I will yeah, I think I won't stop thinking about in the near future. You need to take in mind that if you use maybe two tests like test A and test B and they are referred to the same test data, that can be quite problematic especially leading to flaky tests if you screw it up a lot. Because if the first test changes something inside of this data and the second one has a dependency on this data, may it be by purpose or accidentally the last test, test B will fail and you need to take lot of time to debug that because if you see some test results you think the error is in the test, not in the data. A little real life example. Sometimes in my starting time I used MySQL dump test data for all of my tests and somehow some side effect came into place where the first or second test altered the data and all of those test following failed and you really don't want to have that. So so this is an example from end to end testing, but it could be important for unit testing as well if you take the same CDATA or something. So the solution would be trying to keep your test as isolated as possible. I use an end to end example because I'm most familiar with it, but if I write a test I take care that I'm always starting with the same basis, clean installation of my software or whatever and I take care of the test data for every test specifically and isolate. That means in my example I use API request to create all the data I need and afterwards I start to do my assertions or anything I want to test. So I don't. I know because of the cleanup there won't be any side effects from old data or whatever and I will always have the data I need to run my test smoothly and this way I avoided lots of wakiness Actually now that's a great tip.

I know when I worked for Enterprise Co. We had a data loader which would insert data beforehand with an API and then clean it up at the end and we it pretty reliable as long as we didn't create corruption in the database because some people weren't using all their fields needed for the statement. But anyway you also touched on two other things. I think a lot of people a lot of times make tests unstable. The first one was the locator strategy. So do you have any tips for Locator strategies when dealing with end to end testing.

Mostly yeah, that's approach to avoid testing implementation details because with most tests you want to ensure that the user has a wonderful experience with their software and won't encounter any bugs. So you need to think the test test of. Yeah, like an image of your user. And if you are testing implementation details, for example. Amazing. Writing your test based on CSS selectors, which might change in the future, you can get into a quite annoying situation. Imagine you refactored your application, did some more styling, anything like that and your test fails because of it. Your test is just outdated and there's no bug inside of your application, so your test will signalize there's an error if there is no error at all. And so in this case you won't be able to see the problem of a failing test at first sight, making it harder and more annoying to maintain. The solution would be searching for selectors which are less prone to change. My favorite ones are data attributes with nice yeah, naming like let's take data Cypress test or something which is quite obvious for the developer if he reads it. So he knows oh this is data attribute which is used for testing, I won't change it. And he can continue with his refactoring without any further notice and without breaking any of your tests.

Absolutely. It's a great strategy. I guess. Another thing though is sometimes people do an assertion on a locator for an element after they do an action and they don't wait enough or they have weird there's something off with their waiting strategy. Any tips around that as well?

I think my friends or colleagues will say oh you're so annoying in this point because I never stop telling it to people and they're getting on their nerves. Because if someone thinks it's a good idea to use fixed waiting times in their test, and I happen to be the reviewer of it, I won't approve at all. Because think about the best case. If you wait, let's say one second every time you do whatever you want to do in your test, clicking something or whatever and your application application is already there to be interacted with in a half a second or something, you wait too long, which makes your test inefficiently and basically too slow. But that's not the worst case. Worst case being you don't wait enough time. Imagine your application is under load and you wait too less time. Your test will fail in this occasion if they're slowed. So again there will be flakiness and and it's quite painful to debug that.

So Ramon, I'm not sure if you coined this term or if it's someone else's flat test design I have in my notes. What is flat test design?

Yeah, it sounds more difficult than it actually is. Try to keep your test simple. Don't use too much abstraction. I love to use the word minimalistic for it. As simple as possible and as pleasant to read as possible. Less abstraction, of course, if the application gets out of control. Of course you can use your custom commands if you take some attention on the naming of the commands and maybe some typescript definitions or any other possibility to strengthen developer experience here. But well, don't test too much even inside of your test, just to keep the memory okay and keep it clean and minimalistic so it's not that difficult to read on to understand your test later on or if someone other developer wants to work with it.

Awesome. Okay Ramona, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? You've given a bunch, but just don't know if there's one that you always go to and what's the best way to find or contact you.

There's one thought I want to remember you if there's one sentence you need to take away as a learning from this podcast episode, think about your test like a friendly assistant, not a hindrance or something you don't want to work with. It should, yeah, make your life easier. Not bad. So yeah, this is the most important thing to remember. I think it's. Yeah, it's in Uni Goldberg's guide as well. So it's actually quite important and I think it sums up everything in a neat little way. If you want to contact me, I think the best way is using Twitter or LinkedIn. Actually my handle is Leichtek, which is. I think. I hope it's good to keep in mind if you're not German. However, I think it's okay. So you should be able to find me with this handle.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.com forward/a359 and while you're there, make sure to click on the Try for Free Today link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.