263: So You Want to Be a SeleniumConf Speaker with Marcus Merrell

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to the show. Do you dream of becoming a testing conference speaker or not? Sure how to get your talk accepted at an event? In this episode, Marcus Merrill, Director of Technical Services at Source Labs and one of the lead organizers of the Selenium Conference, shares his experience with running a testing event as well as what it takes to become a speaker. He also shares his insight around some cool new features from Source Labs that helps with performance testing. You don't want to miss this episode. Check it out. Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the homepage Sponsor section. Hey Marcus, welcome to Test Talks.

Thank you for having me.

It's great to finally have you on the show. I've talked to you for a while, been meaning to get you on and today's the day. Before we get into Marcus, can you tell us a little bit more about yourself?

My name is Marcus Merrill. I'm the Director of Technical Services at Sauce Labs. I've been in the testing community for about 18 years. This career path after about a seven year career spent mixed between video games and some database stuff. I'm also the co chair of the Selenium Conference planning committee and I work tirelessly, hopefully tirelessly, to turn that into a really good event that encourages all sorts of people to come and talk about testing and Selenium as a community.

Awesome. That's one of the things I wanted to bring up was how you actually are one of the key contributors to planning the Selenium Conf, which seems like a huge project. So how much time does that take up? How many people are involved? I just curious to know how that works.

Well, there's five main members of the organizing committee. There's Manoj Kumar, Anand Bagmar, both of whom actually work for applitools at this point. Then there's Ashley Hunsberger and I, who are the co chairs along with Andrew Krug, who is also known as Lazy Coder. And the five of us work to, you know, choose the venue, secure logistics and do the call for papers. On average it takes about an hour a week, but when we're getting into the call for papers it tends to take two to three hours or more, depending on how many submissions we go to. It used to take dozens of hours just among the core five of us to choose the papers, but we've gotten it down now to where we have a bunch of volunteers, like probably eight to 10 volunteers for every conference and they each review 40 or 50 papers and each paper is reviewed at least three times. So we have a kind of spread the, we've spread it around and we've also got more diversity of opinions going into which talks should be selected, which has resulted in some good diversity of outcomes in terms of the kinds of speakers that we're able to put on stage at the conference. So it goes in bursts and then at conference time I'm basically 16 hours a day, you know, the day before until the day after. But it's great. I mean, I really, it doesn't really feel like work at this point.

That's a massive undertaking. So I just wanted to call you out for doing that. Thank you so much, Appreciate it. So along those lines, I think a lot of times people have questions, I'm getting off topic here, but for call for papers like what makes a good paper? Any tips for someone that wants to talk, first time speaker, what you all look for?

There's been a lot of talk about that just in the last few days since we closed out the London conference and people are kind of looking for feedback and trying to find out why didn't my paper get selected, stuff like that. And there usually isn't one straight answer because we get 250, 300 selections and we can only select fewer than 10% of those. And so sometimes it's a little bit of a numbers game in that, you know, one reviewer is going to have a certain context that they're coming with and they're going to dislike a talk for some reason. And then it's basically so it's supposed to be reviewed by about three people. So if one person votes it down.

Because they've seen something too similar in their other sift and the rest of the papers that they've reviewed, they've seen something similar or they've seen too many things that too many times this talk has been given in the past, they've seen it recently, or it's not a fresh enough topic, something like that, we tend to be able to select so few of the papers overall that it only takes sort of one person on that tribunal to, to say, no, I don't, I don't think we want that. Now what we do is we generally sort first on what are the highest rated papers. And we tend to go down that list in a group discussion. And we go until we have around 25 papers, which is more than enough. Usually we take 20 to 23. And then we. We tend to select a number of alternates and backups. And then what we do is we do another round where the people who are doing the call will say, here's a paper I really think we ought to consider. And then if that happens, then we have a discussion about whether we should move that in or move another one out, that kind of thing. So to answer your other question about what makes a good paper, it really is going to be around trying to address a real need from a user in the community that feels like it would apply to a whole bunch of other people. That's over generalized, but that's probably the most true statement I can make, because the truth is we usually look for five or six different kinds of papers. We look for case studies where someone has done this right or failure analyses where something has done this wrong. Usually we don't get too many of those. Then we try to look for cultural talks. We actually overemphasize those. We, I don't know, we overemphasize them. We emphasize them more than I think the community is overtly asking us for, because people need to hear that stuff. And we always get. We always get some people saying we're doing too much of that before the conference starts. And then after the conference is over, thank God you did those culture talks, because we really learned a lot from them. We usually like to do a few technical deep dives and we select sort of the most interesting newest topics on those. Lately we've been looking at a lot of AI and stuff like that in terms of what kinds of papers we really like to look at.

So there's a lot of different categories. And so it's like if we see four awesome papers in one category, we have a tough choice to make.

And that's just kind of how it goes, because we're not going to take all four of them and overdo that one category at the expense of another category that we really need to focus on. Does that make sense?

Absolutely. Yeah. So once again, it's another thankless job because the feedback you get is always, in my opinion, when you do a conference, one person says, I want more demos, and another person will say the exact opposite. So it's really hard to even decide. I guess at that point I've actually.

Gotten to the point where I almost look for that. If I see almost a bell curve in the middle of. This was good. This was great. This was medium like a bell curve. And then on the ends I get just as many. This was great. This was awesome. As I do. This was terrible. You need to switch course. If I get kind of a balance of those, I feel like we probably hit the segment we wanted to hit. I'm trying not to lose sleep over where I lose sleep in programming. A conference is there's two rooms. One is huge, one is small. We have to choose which session goes in which. And almost every time we have that situation one time per conference, we mess up. The huge room has 40 people in it and the tiny room has standing room only with 100 people out the door. We almost. We make that mistake at least once per conference. It's hard to predict these things. And that's where I kind of lose sleep a little bit.

No doubt. So how do you guys pick where it's going to be? Is that also just random or is there some thought process behind that?

It used to be based on a number of factors like where have we not gone before, what community are we not serving now? But as the committee has gotten, I guess, longer in tooth here, we all have grown in our careers. One thing I think you might have noticed as I listed off those names is that Manoj and Anand used to be quality assurance professionals working for companies doing the job. Anand was a consultant. Ashley Huntsberger has been a staff quality engineer at Blackboard for a very, very long time. I was a staff quality engineer at RetailMeNot. We were all individual contributors.

And three of the five of us, potentially soon more actually work for vendors now at almost like a director level. We all have these really big jobs now and it's really hard for us to. To spend the kind of time working out legal, logistics and stuff like that as compared to the way it used to be in 2013, 2014 when Dave Heffner was doing this and it was his full time job basically and I was devoting a huge percentage of my time to doing this work. We could choose a new venue and along with Bill McGee from Sauce Labs and a few other folks, we could work with Software Freedom Conservancy to work out the kinks in the logistics and the legal liabilities and stuff like that that we would need in order to contract a new venue and figure out what was going on, do all the stuff and then have the CFP and then recruit everyone. We used to have a Lot more time and energy to do that. And what has happened is what was also true at that point was that there were some things that we weren't doing very well at that point. We were not very good at getting women to speak at the conference, and we were not very good at taking care of speakers and doing the little organizational things that I think conferences really should be doing. That's where we started getting in touch with a group called White October Events who manages our big conferences these days, except for the one we just did in Tokyo. And they handle all of the hard stuff. And they actually had some very interesting, very concrete steps that we could take as a committee to encourage more women and more minorities, people of color to come and speak at our conference. And it's all kind of worked out really great as a result of that. We are greatly sort of reducing the number of different venues that we go to at this point.

It hasn't really become obvious, but we're sort of rotating between a few major cities at this point, and that's kind of where we're stuck at this point until or unless we get some people on the board who have way more time on their hands and a desire to move this conference around the world like we used to.

Awesome. Yeah. I've noticed a much better diverse group of speakers. Even a lot of times I see the same speakers over and over and over and over again. I love conferences that kind of showcase new speakers and people with different perspectives. For sure.

Yeah. A lot of credit to White October Events and Ashley Huntsberger for teaching us how to do this thing, because Simon Stewart has been saying for years that he wanted to do this, but Ashley and the White October group came in and really taught us how to do it. It wasn't just vaguely wishing it was. Here are five concrete things you can do that will, like, guarantee that you get a better speaker core. And it worked. I mean, we had, I think, 40% female speakers at.

Tokyo. And I think. Or close, maybe not Tokyo, but in wherever the last one was. My goodness, my brain is right. Chicago. That's it. And then I think we have better numbers than that in London. I think it's 47%.

So what are those three things?

So one of them is paying for speaker travel and tickets. We used to do that. We used to say if you were a speaker, you get a free ticket. Now we say if you're a speaker, we pay for your flight, your room, your board.

We're trying to be very careful about things like even as small as if you are checking into a hotel and you have a debit card, banks are going to hotels, are going to put like a hundred dollar hold on that card. Not everyone can support that, you know.

So.

We started to accommodate for things like that. And also picking people up from the airport, taking people back to the airport, arranging the transportation and making accommodations for people like that makes it to where more women will apply to speak at your conference. Because when we weren't doing that, pretty much we'd get 75 applications and they would all be people who could afford to just go on their own, which in retrospect just seems silly. So I was always concerned about budget, but White October came in and told us, here's how you tweak some stuff inside your budget. You don't even have to raise the ticket price as much and boom, you have your speaker travel covered. So that's one thing. Another thing is we straight up invite people. Now instead of just going through the submission process, we say we know that there's a topic that's out there that is, you know, covered by I think Abby Banks, a great example. Some of this shift left, shift right stuff that I've been talking about, she is also talking about. So we say, well, you know, in my case there's been a lot of times when I've said, well, I know how to talk about that, but I'm going to find someone else who knows how to talk about that. I'm going to put them on stage instead because people have heard from me enough. So a lot of it is handing the microphone to the people.

Who you're trying to help, you know, accentuate.

And one thing that we've also noticed is that this.

Work and making it visible like this and talking about it openly. Oh, the third thing is also adopting a code of conduct which I feel like is so obvious it shouldn't be said, but that's not true. I still get invited to conferences which don't have a code of conduct. It's ridiculous.

Once you do these things for a couple of years.

It seems like it self perpetuates people. The word gets out that it's a friendly conference, it's a safe conference and we've had some really amazing applications and we love putting new speakers on the stage. That's the truth.

All right, so Markus, you did mention recently that a lot of folks, high quality testers in the, in the community have been going to vendors. It's pretty much applitools or sauce labs gobbling everyone up.

Yeah, applitools. It's funny because Sauce Labs was well reputed in the early 2000 teens as gobbling up all the talent. But now it's applitools by far. I mean, it's. They're having an amazing party over there with Dave Hefner and Angie Jones and Manoj and Adam Carmy and Kanika and Justin Ison. I mean, the list goes on and on and on. Sounds like a great time.

Yeah, no doubt. So how does that affect.

You'Re working for Source Labs? Is this affecting what you've seen for people in their actual. When they do talk at one of these conferences, are they more vendor specific now or is it still an open source type of vibe?

There's just been a couple of instances recently where some people that I wouldn't think have actually brought more of the vendor in to their talk that I was surprised by. On the whole, what I'm seeing is that people that have worked for vendors for years seem to bring it in almost without meaning to. And people who are just coming over, they still seem to have the sort of objective whatever. And I know that I'm also having to sometimes be a little bit careful because, I mean, I will stand on stage and say things like, I will be very transparent and say, you are all writing too many selenium tests. And this is coming from a leader in the selenium community who works for Sauce Labs, which is essentially a selenium vendor. You're all writing too many selenium tests.

Write fewer. Make them good, make them stable, run them more often. And I guess my philosophy is, and nobody at Sauce has really, you know, had a bad reaction to this yet is if we at Sauce Labs help people be more successful even by saying stuff like that, in the long run, they're only going to need more of what we offer and they're going to be partners that we want to stick with. They're going to want to partner with us because we're interested in their holistic success, which we assume breeds more business for us.

Absolutely. All this knowledge going into the vendor now, I think would help influence making good decisions. So is there any projects you're working on to help maybe drive or kind of influence new products or what the community needs that maybe the vendor didn't know about? Because you actually have real world experience.

Yeah. And the good news is that my team who works for me, the solution architects, there's six of them, and a lot of them are people that are on your list of, you know, the best testers, I think three of them might be on your list, in fact, Titus Fortner, Nikolay Adbalodkin and Wim Sellis. So they already had this resource but now what they have with me is someone whose full time job is to help advocate for that group, which in turn advocates for the customer. And so there's a lot we're doing right now around education. We're trying to release some video series. I really also admire what Angie Jones has done with the Test Automation University. And we're doing stuff that's not quite a duplicate, not at all a duplicate of that because she's doing very comprehensive stuff with like learning Java, stuff like that. What we're doing is more almost targeted at the exact level best practices. Here's how to look at this organization. Here's how to measure this organization. We've released something, I don't know if you've heard of it, called the Continuous Testing Benchmark. Which is funny, something I said to Charles a long time ago and I think they've known for a while at Sauce Labs is you are sitting on an ocean of data about our customers and what kinds of things they're trying to do. And we are now, I don't take any credit for it, but we are now actionizing, actionable, making that data actionable. It is telling us stuff now about how as an industry we are executing our selenium tests. The insights we're gaining are tremendous. So that's one thing is trying to say that as a whole, here's where the industry is along sort of an imaginary maturity curve saying if your tests are failing more than 10% of the time, you have a big problem. If you're, you know, and that's kind of an arbitrary number. When we get customer specific with it, it's actually.

There'S ways we can talk about this number that are actually actionable. And so we're doing some data analysis around that. The other data analysis that we're starting to do, starting to start to start to do is around failures we're trying to look at. If a particular build sees a number of failures over time, what are the patterns we can pull out of those failures to give you insights. Which gets me back to this whole shift right thing that I've been talking about lately. I feel like I don't know if it's a growing movement or if I'm growing it. But basically I feel like we've been in the shift left world for a long time where we try to move testing earlier in the pipeline. It's all about product coherence testing, coherence process efficiency, making the pipeline smoother and giving you the confidence to release more frequently. Shift right is about how to get insights and wisdom back into your testing process and your development process. And so that's what we're trying to do here is say all the tests that these folks are running. There's a group of a thousand tests that are run every day, four times a day. Here's the patterns that we see in that build, and here's the two things you could do that would make that build green and fully actionable. I guess you'd say.

Yeah, and it's, it's awesome because I, I like this level. Like you said, some people do focus in on the tester level and that's cool. But a lot of times educating your, your executives, your management as well, is just as critical, if not as more important. So I'm glad that we have this nice balance now between.

Yeah, absolutely. I mean, we've, we've started to talk kind of openly those of us who have done this job before.

Everyone on my team has about eight to 10 years of test architecture experience. And so we've lived in usually multiple different worlds. And almost all of us have lived in a world where we had exec sponsorship or we didn't, or both. We went from one to the other. And just the amount of change you get when you can talk to a VP openly about, hey, our test automation program is really proving value because it's finding bugs in, you know, right before they go into production or right after they go into the pipeline. And just having the VP's ear and confidence is a game changer. Yeah. And so the other thing that I've been focusing on lately, which I think is going to be the subject of an article we publish at some point, is that we've been talking about how why in the world this problem, problem is so hard. I mean, maybe you can, we can play a little thought exercise with this, Joe. Like think of all the different areas you have to cover in order to be someone who does continuous delivery. Let's think about what they are. I mean, security testing, right? So performance testing, load testing, what I mean, but I mean, even it starts back in like the product in the development phase, the unit testing and the integration testing. What else am I not thinking of for continuous delivery?

Monitoring when it's in production?

Oh, man. Yeah, monitoring is gigantic. And that's where ShiftRight really comes in. And I'm trying to think about which of these aspects that are kind of a science. I Think of monitoring as being kind of a science at this point. You turn it on and HTTP is going back and forth. And you know when there's a problem. Right. I mean, you know, quantitatively. You don't necessarily know if user behavior is off, which is the next step. I want to go into load testing. You can hire someone to come and do load testing. They don't really have to know anything about your site. Right.

Well, to create a realistic scenario.

Right. Realistic set of scenarios. At some point it seems like it becomes an engine that you crank. But I've been thinking about, is it true that functional testing is fundamentally more difficult than any of these challenges? Because it seems like most companies do have a handle on monitoring, performance testing, security testing, even unit testing. Companies who choose to do it seem to have a pretty good handle on it. But this domain. Tell me what you think.

Well, I'll probably do this. Almost 20 years. I've never been successful 100% with UI automation. I started off as a performance engineer and writing a performance script which was headless, that didn't use the UI was so much easier than a functional UI test. So for whatever that's worth, maybe I'm just a hack, but even Google, I heard, has only a 95% pass rate in general. So something's off.

Something's off. And.

I think a lot of people don't even have the 95%.

Maybe not even close. And so I've been trying to play this thought exercise about why is it so hard? Is it because we choose to test the wrong things at the UI level? I think probably. Are we using the wrong tools at the UI level versus.

Another tool that might help? I mean, probably in some cases we don't even know. In some cases it's the tools are. But like, what I've come down to is that there are several people who know how to do this stuff really, really, really well. And in every org that I've spoken to who does this really well, it always seems to come down. There's one or two people who are at the center of it. And you got to imagine if they left the company, everything would sort of go to hell. Like it has happened to me not more than or more than once that when I leave the company, they stop using my test automation and they kind of throw it out after it starts breaking after the site changes enough. It didn't happen in my most recent one, thankfully, but what it's led me to is the fact that it seems like it's just a bunch of heroes and heroes who know how to do this stuff. And that's heroes just don't scale.

No, that's a good point because I'm also thinking when there used to be more waterfall and I owned the framework, I can rock it, I was able to get tests to run and pass. But without me, bam. You know, like once agile came in and sprints and you have your teams doing, it's a disaster in my experience. I don't know.

Yeah. And so I think it requires this weird combination of domain expertise, technical know how, desire to stay in testing, which is not a common thing. Then fourth is the ability to do all this and probably make less money and certainly have less attention.

Within the org. You know, you and I talk about this stuff all day long, so we feel pretty good about ourselves and our contribution. But people that are in qa, you know, they're very unsung, I think. So that problem just doesn't scale when you require that amount of heroism. So anyway, what that's leading me to is at Sauce, we are at the beginning stages of trying to think of how we can scale the heroism so that everyone become a hero, so that nobody has to be a hero.

That's the kind of thing I've been thinking about lately is how do we scale my six people who are all heroes at their previous jobs, that kind of thing.

Awesome. So is that going towards products or is that just education then?

Like starts with education, but.

Some way or another, I'm sure it's going to make it into the product. I think that.

Sauce is really. Seems like they've got their head on straight right now about.

How they want to change things they've done in the past and how they want to help influence the industry and help them move along into the future. It really seems like the conversations we're having right now seem very mature.

I hope anything you're really excited about, that's a new release in Sauce Labs that folks should know about.

What I'm excited about on performance side.

Is when we talk about performance, people like you said, I think of stress and low test, but that's. I don't think that's the audience. So what's this performance piece about, this new addition to Sauce?

That's the thing that I'm really excited about is the fact that it's specifically not related to load and stress testing. It is deliberately irrespective of load and stress testing. So I've had cases in sites that I've worked on where there's a performance degradation and we Start to feel it in our users behaviors. And the first thing people do is run load and stress tests and we see, oh yeah, there's a performance problem. But here's the thing, it happens when you're just warming up with the 20 nodes and it also happens at the 500. And it's all the same, it's not a linear growth, it's just the same. And we say, oh my. Well, that means there's something in the UI that's changed. They've introduced some DOM element that takes forever to run, they've increased the size of images that they're doing, they're not resizing images properly, something like that. We run to the load and stress test when we see a performance issue, not understanding that it's actually a UI problem. And this is the tool that we suggest using for diagnosing UI problems. And not only diagnosing them, but dialing in performance on a page that you particularly care about and then noticing over time if the performance on that page changes. The example that I usually bring up internally and externally actually is that RetailMeNot, that's what company I was at before. It's a very, very, very SEO intense B2C website that has millions and millions of hits per day. And I don't know if you know anything about Google search search engine optimization, but we had something like 10,000 keywords where we were number one. If you say blah blah, coupon, we're number one on the rank. And one day Google introduced an algorithm, I believe it was called Panda about 2014, that changed how they index the different things that Google looks at. So normally they look at a combination of bounce rate and relevance. However they measure that, it's magic. And then page performance, well, they changed how they indexed and they started weighting page performance way higher. And suddenly we started dropping in all of our rankings. And it's because for months we had been sort of once a month we would do sort of a lighthouse check to see how our page performance was. We kind of took our eye off of the page performance ball, so to speak. And I believe that if we had had this feature in place at the time, running alongside our Selenium scripts, we would have known instantly when our page load started slowing down and we would not have been hit by this massive SEO penalty. So to me, if you're in retail or banking or some B2C where SEO is an important thing, having this running alongside your tests is absolutely invaluable. So I'm super excited about it.

Yeah, I mean, this is critical because usually once again with performance testing, you're measuring the response of your backend systems. But this is the true because a lot of applications now are like react and more AngularJS front end JavaScript, the rendering really does matter on the client's local machine. That's what you're capturing. Is that how I'm understanding it correctly?

Yep, it's capturing the. Yeah, you're right that it's capturing all, you know, the JavaScript rendering. Time to first byte, time to first paint, time to first interactive. Just those kinds of metrics that they are definitely influenced by server side factors, but if you can isolate them and measure them over time, you can really capture those deltas and you can write code that asserts you don't exceed your benchmarks from some previous time. So that's my big thing that I'm really excited about with Sauce right now. Then also there's the continuous benchmark that we're releasing and there's a few other things that just Too early to talk about. I wish I could talk about it more, but yeah.

It'S exciting. Time to be here for sure.

Okay Marcus, before we go, is there one piece of actual advice you can give someone to help improve their automation testing efforts or even their speaking efforts and let us know the best way to find or contact you?

Sure, I'm on Twitter.

Just give me a follow on Twitter. That's the best way to get a hold of me for sure. If I had one piece of advice I'd have to give it is kind of going to be old school, which is keep your tests atomic and autonomous. If they are UI tests, meaning test one thing and one thing only. You can do multiple assertions on that one thing, but try to take try to get the data and the site into a state that you need open on the page that you want to open on. Take a single action, make a few assertions to make sure that that thing went correctly, and then close out the browser and start the next test. It should take under 30 seconds, probably under 15 if it's done right. The other thing is to keep them autonomous, which means that any test can run independently of any other tests. Idempotence is another way to express that, even though they're slight differences. So that's the one piece of advice is those two things.

Thank you Marcus for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com 263 and while you're there, make sure to click on that Sign up for a free trial link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test T Talks podcast. Head on over to triple W.TestTalks.com for full show notes, amazing blog articles, and other automation awesomeness.