Email Storms in a Load Test with Paul McLean

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Paul McLean all about email storms in a load test. Really interesting technique. Paul has more than 30 years of load testing experience in a large variety of technologies and markets with a strong focus on risk based analysis and technical delivery of test in challenging situations. Paul has manufactured data to support many of his load tests and set a separate business focused on test data and related services which aims to improve test outcomes while simplifying as many aspects of test reactions as possible. Really cool stuff. You don't want to miss this episode. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to Test Guild test.

And click on the link to learn more.

Hey Paul, welcome back to the Guild.

Thanks Jew. It's great to be here.

Awesome. Now you have been on the show before, but for the folks that don't know or may have missed it, is there anything I missed in your bio that you'd like to tell us a little bit more about yourself?

No. I think you did a great summary. Thanks.

Cool. So Paul, interesting topic you sent me. I saw this on LinkedIn on email Storm for performance testing. I thought, huh, I don't even know what that is. So just curious to know at a high level, what is an email storm?

So an email storm, like we're talking about viruses a lot right at the moment and a virus is a problem. When one cell is infected, it produces a whole stack of virons that affect a whole lot of other cells and before you know, it gets out of control. Well, in emails sometimes a workflow system, for example, might generate an email and then one person may respond to that email, but that response may trigger several other emails and those recipients of those other emails may open it, click on some on the link, and follow some screen flows and generate even more email. So you have this email cascade and it effectively is a bit like a virus. So if I have a test and email, email messages are the basis of Initiating workload. Then instead of just limiting myself to an application simulation model where I talk about how many transactions per second or per minute I'm going to do, I actually incorporate real emails into the test. An extreme example of that is an email storm where you have this viral replication of email. So on that article I put a video in there showing the actual storm. I actually create a storm and I show how it works, which I think is pretty cool.

Interesting technique. So do you think most performance scenarios don't include email and email is actually an aspect of performance that can cause issues with your application that you're not aware of?

Yes, I do. The reason I think it's a problem is that it's very hard to catch emails. Traditionally, you've got to do a fair bit of work to be able to catch them. And most of the time, most of our scripts are procedural from a load testing perspective anyway. They need to be event driven if you want to be able to catch and process emails. So without the ability to catch an email in a script, you have to log in as a user as if they had received an email, and then go through the process of doing whatever the work workflow is like activating some screen or approving something or rejecting something or whatever the workflow is. So you're doing it procedurally. Whereas in most of the systems that I've tested, the email generation itself in the back end isn't like real time. It batches things up so you'll have some process that goes through and every 1 minute or 2 minutes or 5 minutes works at all the emails it's got to send out, then it sends them all out as a burst. Now, if people respond instantly to an email that goes out, then their responses, that workload behind their responses, are also clustered together According to that 12 minute or 5 minute process in the backend. So if your load test actually responds to real emails and triggers activity based on those emails, you're more likely to see that nasty effect of a burst of responses every minute or every two minutes. Because you're actually replicating the real life scenario of people responding to an email. Not everyone responds. Obviously, if you send an email to 20 people, three might respond, but you just stick that parameter into your test and you say, Okay, I want three out of 20 people to respond immediately to an email. The other thing is, when you get an email link, if you click on the link often, the actual backend workload is not quite the same as logging in and navigating to that, to that part of the business process. And some apps it's actually more intense, it actually does more queries, and some apps it does less. So it gives you a more realistic workload profile, which, that's why I like it and that's why I now incorporate it in my test. If email is a part of the.

Business process, so how do you determine how important it is? For example, I have a conference, and in that conference people can buy a ticket, but once they buy a ticket, they'll receive an email. And in that email it says, click on this link, verify you're able to log in and check your account. Is that a process that you think would deem worthy to automate?

Well, in that case, the moment someone signed up, you know that they've signed up in your script and you could just have a 50% chance that they might go and click on that link. You can probably, you probably know what that link is. In fact, you could probably work that out because of the nature of most of those types of links. And you could probably skip the email in that case, because it's like a one for one. You have your initiating event and then you're only going to have one email as a consequence of that, and then you can manufacture that process. But if that email link did something fundamentally different, if it did a different query or a different set of queries, then maybe it would be good. But the area where it's most important is where you have this cascading scenario, especially for work workflow management, where someone puts a change request in and five people suddenly get excited about that change request and then 20 people get excited about it. You have that viral replication type scenario, but like I did one with pizza, and someone would get an email to say, or a message, not necessarily an email, but I did it. The email flow, you'd click on the link to see where your pizza is, and then you could refresh from there. And that particular business case, the process of clicking on the link was different to just going into the app and doing a refresh. So that different workload can be an issue, right? Like, we want to be as authentic as we can in the testing that we do and make it as real as possible. We don't want to miss a whole bunch of queries in our load test. So if there's a difference between what happens when you click the email link compared with what happens if you just proceed to that same business function through the app, then you probably should click the link.

So it sounds like it could be then if people are ignoring this it could run a query that takes a long time, that maybe creates a bottleneck on the database or your disk. Is that what I'm hearing?

Yes. So if you click on the link in an email, sometimes it might do 10 queries that are the same as logging in and then doing a few steps to get to that screen to be able to approve or whatever it is that you've got to do when you get to that link. So if that's a different set of queries or if the queries are processed in a quick sequence, that leads to a different backend workload which will put different stress on the database and then could contribute to performance database related degradation. Does that make sense?

Yeah. How would you trace that if it is the email that's causing it? Are there any metrics you would look at?

Well, that just comes down to how you're instrumenting the application under test. So whatever instrumentation that you've got in place when you're running a load test for your standard user interactions, you would then get the benefit of that when you add in email. Because all the email is doing is it's providing you a new URL or URI to activate. So your test is still doing all of the normal things, but it's just the way it's triggering it and it's the timing of it that's different. So what I'm really suggesting, I'm suggesting that we incorporate emails into tests is that we move away from a procedural process of considering emails and we have more of an event driven process. So you would have a separate script that just sits there and whenever emails come in for all of the users that you care about in a scenario, it actually is able to handle those incoming emails in real time and then cause workload to be done in response to those emails instead of having a bunch of virtual users sitting there all deciding, now I'm going to do business process A, now I'm going to do business process B in a particular timed process. So it's separating out the asynchronous event driven nature of emails from the procedural planned workload of a traditional test design.

Nice. Now I know occasionally I'll get emails from functional automation engineers and hey, how do I automate email? There's only a few solutions that help with that. So how would you do this in a low test?

Well, so you mentioned in the intro, my test data services. So I've got APIs that let you subscribe to an email address from a, from domain, so say from example.com and to an email at testdata email. So you can do a simple API call that says, okay, tell me about any emails that come from example.com and go to this particular email address and then you can do a get message call. And if there's a message there, it'll give it to you in the response, the HTTP response. That way you don't have to log into anything, you don't have to do any complicated coding, and you actually then just get the message as an HTML response that your script can then interrogate, fish out the link, and then follow the link or do whatever. And it's quite nice also because it actually tells you the latency. So if you reset an email address in these API calls and then three seconds later an email arrives for that email address from Domain, then it'll say it was three seconds in there. Even if you check it an hour later, it'll still tell you it was three seconds. So that's actually quite nice because then you get to have an idea of what was the backend processing time that it took for this email to get generated and received, in this case by aws. So I've done it all on aws, but what I've done is I've just used some standard serverless code on AWS to catch emails using AWS's Simple Email service. So what I've done in building those APIs is nothing rocket science, it's just that I've gone to the trouble of doing it, but anyone could actually do it. It's not that difficult.

So in this approach, you're not necessarily testing the physical email server in a company? Correct. Is it more what that email is doing for the load against the application, or are you doing both?

Yes, well, you could be doing both, but generally outbound emails are very easy for email service to do. Inbound is different because inbound you have a whole lot of security checking. You want to check that there's no viruses, there's no malware, and so on. So inbound there can be issues. But outbound, when you click on something in an app and it generates emails to five people, that's very easy for the email server to deliver those five emails. So it's unlikely there's a problem. But the process between when you click that submit button in the app and those emails being generated, that sometimes is a problem. So what this will do is that it lets you see that you're starting to have a queuing problem on your back end. And instead of those emails going out in a minute or A few seconds or two minutes, it might go out in an hour. So if you have issues with your backend process doing what it needs to do to deliver the email to the email system or the message to the email system so it goes out as an email, then this type of test will show you that. So yes. Is that answering your question?

Yeah, absolutely. I guess the follow up here would be this approach. Anytime you have an approach, sometimes people abuse or they don't know how to use it. Is there anything you could people need to be aware of when doing email type performance testing, what not to do or things to look out for that they don't put an artificial load on the application?

Well, if you're doing an email storm, you want to make sure that it's not going to go viral and use up all the email resources in the world. Right now, because I'm using Amazon Sesame, they have very strict limits. So My limit is 50,000 emails per day, for example. And if I send a whole lot of emails to addresses that bounce, and the SES servers on Amazon see my bounce rate as going over a threshold, they'll actually shut down my email service, which is very bad. So I did a very embarrassing mistake for a client and I accidentally, for this client, used an email domain when I was running a load test and there was nothing at the other end email domain. It was like sending an email to example.com and the problem is that, you know, I was running my test, everything was going good and then there were complaints that organization's email stopped working. Like all of their emails stopped working. And it was because I effectively caused a blacklist of the email. Right. So sometimes email testing can be a problem. So that's a reason why when you're having a test, you ought to make sure that everything is separate. Production and test need to be separate. Your email component needs to be separate as well. So if you make a mistake with something in email, you've got to make sure that that can't affect production emails. I did another, if you want to hear a bad war story, I did another test and we sent out millions of transactions and there was emails attached to these millions of transactions over a weekend. And I got special permission. It was using production email and this organization said there's no way on earth these millions of emails are going to cause a problem. We've separated everything out, it's all nice and safe. And what they didn't realize was that when these emails went nowhere and got a bounce back saying they couldn't get delivered the bounce got treated as a legit inbound request through their system. And their inbound email processing had to do all of its virus checking and malware checking. So on Monday morning, after doing this test on the weekend, there was a queue of 2 million ish emails on the way in to this organization. And they didn't get any emails, production emails for a day as they were waiting to clear this queue. So I'm not a cowboy, I'm not a bad tester really. But I have seen problems with emails. And what it's shown me as well is that email is not the amazing unlimited, ubiquitous service that everyone might think it is. There are actually substantial limits. A lot of systems that I've seen have struggles with more than five or ten emails per second inbound. So that's just their emails and that's usually the security appliances dealing with things. But I'm using AWS for my testing services and I can do like hundreds of emails a second in and out, and that's great. But my personal limit at the Moment is only 50,000 a day. But 60,000 a day is not bad.

Yeah, I would say this would flag a lot of security breaches as spam or blacklist your IP addresses associated with the account. So how do you set up to make sure that you're not. Do you mock it or you actually using a real email server or all this activity?

I'm using real emails, so I'm using Amazon SES. So some of the examples on my recent LinkedIn posts, I'll sign up to a website and I'll catch the email. They are real emails and I'm catching real ones. You can send an email from anywhere to my email addresses and I can only do 50,000 sends a day, but I can do millions of receives a day. There's no problems with how many I receive. So I can sign up, you know, 50,000 new users to a real production website every hour and catch all of those emails with those verification links in it and do the signup process as part of a test. The reason that that's important is even though we're testing test systems, often the test process still involves a real email that actually has the signup link in it or the secret or whatever it is in there. So these services that I've got actually work in production as well as test, and they need to, in fact, because often we have production components in our test environment that we actually have to work with. So even though it might be in a test environment for the application and the test that we're looking at, they might have a different email server. It's still an email server that works with production. You might need to ask them to whitelist the test data email domain so they can send it to it. But it's really frustrating if you have to run a test, whether it's functional testing or automated testing and they use a real email server. But you have to go to some person in the organization and get that email might go to a special address or a catch address or something and then they've got to send it to you so that you can get the code so that you can complete the process that requires that code or to get the link that was in the email. That's really annoying. With these tools that I've got, it's just a simple get. So you can just stick a shortcut on your desktop if you like and double click on it. And then you'll see if there's an email come in for that account. Or you can subscribe to sets of accounts, thousand email addresses. And when an email comes in for any of those thousand from a from domain, like from example.com, you can pluck them off one at a time and process them, which I think is really cool.

Absolutely. Now, Paul, is this a service you offer for test data services? Is it free? Like if someone wants to get started with an email storm, is this something you provide? Is this just a technique and they could use whatever they have?

It's three things. It's a technique. So in my white paper I describe how I do things. If someone wants to, they can go out and build this themselves. So I'm not hiding how I'm doing it. It's all very simple, AWS serverless code. I have a service which you could subscribe to, but I also have on that a free key that anyone could use so you could actually play with. If you go to the website, you could play with catching emails right now and you could use it and it won't cost you anything, but it's rate limited and you know, if you're going to use it a lot, if you were going to incorporate into your real test, it'd be better to have a subscription so that you could actually know that it's going to support you whenever you want to run it. I've really built this stuff for myself and I use it myself. And then I realized I should see if I can make this available to others. So that's been my mentality. I'm not very good at marketing, but most of the requests I get. So they're very easy to stick into any automated testing tool, any tool that can do a get, which is any tool will work. I know that some tools say so load runner with a true client user doing an HTTP post, a custom post within one of these true clients is possible with a Google Chrome session, but it doesn't work properly with an Internet Explorer session just because of some of the limitations on that technology. So I ch to make them more gaps so that it's easy to use from anywhere. But yeah, so I've shown how to do it and I've explained how to do it. So if someone wants to go do it themselves, they can. If they want to use what I've got for fun, they can. If they want to use it, seriously, they should just get themselves a subscription. But I'm not really set up properly yet to do that.

Very cool. So Paul, I just want to switch gears really quick. You are speaking at this year's Perf Guild conference going on June 1st to the 2nd. You have a really interesting session. I, I just thought maybe you'd give a little blurb about what your session's going to be about and what people will take away from your session.

Okay, so I really like music. I'm an 80s rock type music person. I love electric guitar. And the thing is that if you measure the signal coming out of an electric guitar and you stick it straight through electronically, perfectly record it digitally, you would think without really much thinking that that would sound great. Well, it doesn't. Because the thing that makes an electric guitar sound great is the distortion. And that distortion is effectively changing the waveform of those sound waves that hit your ears. And a skilled musician is able to express their emotion by pushing their hardware, by pushing the guitar. The speaker enclosures, the way they set up all of the settings on those enclosures, the combination of components that they use to have a particular sound, a fat sound they might call it, or a particular style of sound. And it's very distinctive. I mean some guys, some rock artists have mics at the back and at the front of the enclosure they'll have two at the front. They arrange them specially and it's all to get a particular sound. And I'm fascinated by that. And I really love the music. And if I look at load testing, load testing sometimes is like having a requirement for a load test where they say to you give me this many business processes per hour or per second and this many of another business process per hour. And that's like the signal coming out of the guitar. And if you do that pristine, you would think you get a great test. But it's the subtle changes to the way that you implement their specification of transactions per hour to make it realistic. Like the way that you have character spacing between typing. If it's a UI that might or might not cause auto completion to fire, or the way that your script simulates a real user with subtle variations. That's what I would say is like a distortion on the original requirement. But it's a distortion that gives you a high fidelity test. A test that's more realistic than the customer knew how to ask for. So it's a discussion of this metaphor of fidelity, of a high fidelity music scenario where it sounds great because it's got some distortion, to a high fidelity test where it's realistic because it's got distortion. And there's a discussion on some of the technical ways that you might be able to achieve that. Does that sound okay?

Paul, that's what I love about you. Your approach is. I think it would save a lot of people from finding issues once the in the wild. Because rather than focusing in on what you know, you kind of focusing on maybe what you don't know. I don't know if that makes any sense, but I think a lot of people just stick to a script of what they know and don't and try to ignore any variances or anything that spikes up their, the deviance or something, you know, that makes sense.

Yeah, well, we all get like excited by different things. And if I go to a seminar or a conference and I learn something new that is really exciting to me, I find that fantastic. And if, even if I travel around the other side of the world to go to one, if I get one really amazing insight from it, I'm thrilled. And that usually stays with me. So I'm always on the hunt for that. So I go to a lot of IT conferences. I'm obviously thrilled to participate in Perth Guild, but I also go to a bunch of different ones. I went to an immunology one last October. I've gone to various quality ones where they actually talk about industrial quality processes. I went to one where they talk about bowling pins and they talk about how you make sure that, you know, that the bowling pins will work properly so that if someone's doing, you know, great bowling that they get a 300 score because wouldn't it be terrible if your bowling pins weren't right? So the whole quality assurance, all of the six sigma stuff. So by going to conferences that stretch you in different areas. It rounds you out as a professional and I get thrilled by that. So as a way of giving back, I sought to put this music distortion idea into a presentation. So that's where I'm coming from.

Awesome. Okay Paul, before we go, is there one piece of actual advice you can give someone to help them with their email blast or email storm performance testing efforts? Or what's the best way to find or contact you?

So LinkedIn if someone asks me a question on LinkedIn and it's a sensible question, I'll answer them. I've got an open LinkedIn account so it makes it easy. I'll probably connect with them as well if they're a tester. And my suggestion would be to actually try. Go to testdataservices.com and there's a how to guide there and a quick Start guide. Actually have a go send an email to one of these addresses and then see that you can catch it. And then you'll realize, my gosh, this is really easy to incorporate into my scripts. And then the moment you've done it and you've realized how easy it is, you can then easily do it right? But if you think it's a magical mystical thing that's too hard, then you won't. And once you've used mine, then I'm not saying you have to use mine. You can go out and build your own if you want, or use someone else's if you can find someone else that does it. But try it and then you'll be able to use it. And that will mean that instead of saying emails are too hard, I can't incorporate them into my test. You'll say, ah, you want emails? That's easy, I know how to do that. And off you go.

Thank you Paul for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com.

And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about Smart Bear's brand new browser based performance testing tool Solution Load Ninja. And as I mentioned, you don't want to miss this year's Perf Guild Online conference. It's the third annual online conference dedicated 100% to helping you succeed with creating end to end full stack performance testing awesomeness. Our goal is you learn a team tip, tool, technique or best practice that's going to help accelerate your performance testing efforts in 2020. Paul's going to be there with an awesome session as well. And if you register during the week that you hear this episode, you're getting half off. Currently, we're offering a ticket price, the lowest ever at $97. I'm going to raise it up to 147 next week and then it'll be back to the normal 197 per ticket after that. So definitely head on over to perfguild.com register so that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.