Email and SMS Workflow Testing with Jack Lawrence and Brooks Nelson

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today you're in for a special treat. I don't know if I've 400 episodes, if I ever covered anything around email and SMS workflow testing type topics. So that's what we're going to be covering today with the folks from Mailnator. And I think you really need a lot of value out of this. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For every framework, browser, OS, mobile device and API, get a free trial. Visit testguild.comsourcelabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey guys. Welcome to the Guild.

Hey, how's it going, Joe?

Thank you, Joe.

Awesome to have you. I guess before we get into it, maybe we go around the room and just introduce each other. So Brooks, let's start with you.

Sure. Brooks Nelson, Director of Business Development for Mailinator.

Very cool.

And Jack, My name's Jack Lawrence. I wear a lot of hats here. My business card says coo.

Nice. Awesome. So I was really excited to find your solution. I know, like I said, I haven't had many. I may have had maybe one show around this topic. So just for the folks that don't know what is, you know, email workflow testing or SMS testing, maybe let's just start off really easy that way. Any thoughts on what is email workflow testing?

Well, basically anybody who has a business that communicates with email, what they need to do is make sure that those communications are working properly. Anybody who sends more than a couple a day, of course they've automated or in some way made an efficient path for them to do this, interactions with their customers. And they need a way to test that. You don't test email by sending them to customers, you test it someplace else first before you turn on your live customer interaction systems. And that's what we do. It started as email. If you need to in any way test the email that flows into or out of your product or your platform, we're a good way to do that. We also offer SMS now so you can test your SMS based interactions with your customers or your clients. And we've added a few other things to the product that sort of broaden the scope for that sort of communications workflow testing.

Nice. Brooks, were you going to say something as well?

Yeah, I think Jack summed it up pretty well, though.

Perfect. Good. Awesome. All right. So, you know, I think a lot of this activity is probably manually done right now from some companies. I know you have a lot of customers. Is this something you see as a gap in a lot of companies where maybe they overlook this type of testing or they have a lot of parts automated, and then the last mile they start doing it manually?

It's all across the board. Some people know right away that they need this sort of thing to happen and we're a solution for that. And they get in touch and that's how they get started doing it. Some companies, though, Joe, I think, like you mentioned, they realize, oh, no, here's this thing and we're not sure and we should probably be testing this. And Brooks can speak to. We had an example last year with. Was it hbo?

Yeah, hbo. I'm trying to recall their specific use case. Sounds like you've got a top of mind. Why don't you jump in there?

I think it was their Max. Somebody at HBO wasn't sure about how something was working and they tested it. But they accidentally sent hundreds of thousands of test messages to their customers, to people who are subscribers of hbo. I think it was hbo Max.

It was, you're right.

And I didn't. It didn't come to us in the business. I think Brooks saw it just sort of in the popular media. Somebody said, well, what the hell is all this from hbo? And it turned out that they sent out a test message through their live systems, and that was a little bit of a customer service issue. WarnerMedia is now a Mailinator customer. They saw their problem because it presented itself in a way that they couldn't really ignore, and they set up a testing flow for themselves and they use us for that.

Great. So, Brooks, anything. Any other examples you could think of.

You know, more generalized, but, you know, the common sort of practice if you're developing code is to, you know, create yourself an inbox so that you can send yourself a message and make sure that the workflow, you know, whether it's 2fa or something similar, you want to test that. So you have to create yourself, you know, Brooks Nelson, 1, 2, 3. @ Hotmail, but you don't want to have to use that. And if you're doing it for multiple different clients or different projects, you know, that's going to start to get pretty polluted so having something like Mailinator, where you don't have to think about it and you don't have to actually create any inbox that already exists, makes a real simple.

Solution. Very cool. So I thought we'd dive into maybe the history of Mailnator, like, what is Mailnator? Why did it start? And then maybe we could build off that to get some more discussion.

Going. Mailinator started. This is an interesting story. It started as a bar conversation. Me and my friend Paul, Paul Tema, who's the heart and soul here, we were roommates in grad school, and I had sort of a side job doing system administration in a different context. And I had this weird sort of nerd sanctimony about how email was perceived in the world. And I talked about how there shouldn't be any passwords and everybody should be able to see every inbox. And it didn't compute for him until after a few iterations or a couple of beers. And then he said, oh, yeah, I get it. And he built it, I helped. But it's mostly all him on the inside. He loves the engineering stuff. For me, it was an experiment in identity. For him, it was an experiment in what's possible on the Internet. And we put up this website that did this very bizarre thing, which it allowed you to send email to any inbox you wanted to with no signups and no passwords. Also no privacy. And we just did it as an experiment. And it got popular right away. We had hundreds of users and thousands of users and then millions of users within months. People just poking at this strange new beast that was out there in the world. And that's how Mailinator started. And it was always popular enough to keep itself going. Maybe you remember the days of banner ads, and that's how it paid for itself in the early days. And then it just sort of kept chugging along. And it was always a really great place for Paul to do experiments in architecture or in features. He had his own real career going that paid money. And Mailinator was a free service for about 10 years. There's all kinds of weird stories from that time. I'm happy to relate them, but that's where Mailinator started. Any inbox, we had, at one point, thousands of other domains. The system is still open. Anybody can register a domain and point it at Mailinator and then send messages to that domain. It'll show up in the real Mailinator.

Feed. Very cool. So over the 10 years, I assume you collected a lot of feedback and information, how people are using it. Was it hard to pivot to a more, I don't know, monetized product, or more of not professional product, but something that's more like something the larger companies make consumer enterprise version, I guess, of this type of.

Activity. Well, I worked on it with Paul for the first couple of years and then, I mean, my point was proved. He still had interesting architecture ideas, but there wasn't really anything left in it for me. And in the mid-2000s, we worked on a sort of side project that played off of Mailinator, its function and also its sort of position in the free Internet world. But then it was. I didn't work on it anymore after that, except until about four years ago. Paul called me up. He had just taken a new job and he said, I have no idea what's going on, but it's starting to make money. And I said, well, that's great. And he said, but I just took this new job and I don't have the time. Do you want to see if there's a business here? And I was bored with what I was doing and I said, sure. And that's where we found out this thing that we'd invented with no, it didn't solve a particular problem. We didn't make it to do a thing. We made it to be a thing. And it sort of found all these use cases in the free world in its first ten years of existence. And where the money was coming from was Paul had made a very basic API and a click to buy here for $29. And QA and testing people were buying it and we didn't even know. I knew nothing about the QA and testing world and neither did Paul. But we found out that that's who was interested and we found out why. And so that's where Mailinator as a comm product, as an enterprise product started. We sponsored some testing and QA conferences just so that I could go and learn who our audience really was. And I learned a lot. It's a very interesting world. There's an awful lot of different jobs that need to get done there. And that's the QA and testing world. That's the real core of the Mailinator product right.

Now. Great. So I know that there's some other options available for this type of testing, but you know, because you are using email, it is kind of sensitive. I'm sure there's some sort of enterprise level type of grade things that you want to have in place in order to make it. So it's security and everything's in place. How does this stand to maybe other solutions. Is this an enterprise type of product you think? You mentioned hbo. Do other large companies of banks or companies like that, Is this something that they could use and be assured that, you know it's going to be safe and.

Secure? Yeah.

Absolutely. It is something not only that they can use, but that they do. And Brooks can probably tell you a lot more about who's using it from those.

Fields. Yeah, you know, our client list ranges from everybody, from someone like a Fitbit or a Docusign, obviously Docusign, you know, that's important that they have integrity to their systems. But, you know, when we start looking at, you know, the use case for Mailinator, you're putting test data in there. You're not putting anybody's actual data. You know, you're doing stuff that if it did sort of go sideways, you wouldn't be worried about what was exposed because realistically, nothing going in there should be private data. It should all just be test stuff. So that element kind of is nice because we don't have to worry so much about that end of it. But from an enterprise standpoint, you know, Jack and I have had this conversation. Mailinator was built for the public use, and because of that, it's been open to, you know, every hacker in the world trying to see how they can exploit a free product that gives them anonymous email addresses. So with that said, you know, over the years, Mailinator has added some adaptive filters and limits that prevent abuse of the system and the public. Mailinator is actually a completely separate build from the private melanator that you're subscribing to and you're paying for. So the architecture is all conceptually the same, however, we separate what people are actually doing on their private accounts from the open Internet. So there's also that level of security, but also the systematic resilience that's built into the structure of.

It. That's a super good point. Mailinator is enterprise ready in terms of being robust because as Brooks said, people beat on it for years and it's grown to be a very solid product. We do take security very seriously. We do all the stuff you're supposed to do. We have a penetration test every year, and we've implemented encryption all across the workflow. But as Brooke said, Mailinator as a commercial product doesn't use the public system. Anybody who's a customer here gets their own private domain. So they send messages, not to inboxes ailinator, they send them to inboxes at the domain that they have. We route all that stuff through our system, through our private commercial enterprise system, where things are encrypted both at rest and in.

Transit. So I guess how do people consume mail? Nadir, like in their test flows or their pipelines, how do they incorporate it so that if they had an existing test and then they need to make a call to SMS or email, how does that work? What's that flow look.

Like? So to get started with the testing, you can get a trial or just buy a subscription and right away the system's active for you. You can send emails to your private domain and you can test that manually from your Gmail account or your work account. Send an email to an inbox at your private domain. That'll show up pretty much instantaneously in your storage. There's a web interface if you want to check things manually. You're going to send 10 emails and you need to see what they look like. You can script the web interface like you said with Selenium and there's some other products that do that. We have customers who use Selenium and you can also interact with the messages you send via our Restful API and you can automate that in whatever your favorite environment is. And so to get started, it's really simple and kind of the sky's the limit in terms of automation, how much you want to do or don't do. Some people use the product manually. They're sending 10 or 20 messages whenever something changes on their side. Some people use the product continuously. We have clients who send hundreds of thousands of messages a day and they use our API to make sure that delivery happens correctly. And they use our API to go through their storage and delete things they don't need and push along the things they.

Do. Very nice. Brooks, anything you want to.

Add? Yeah, it's worth adding too that there's some mail routing rules that are included in the system. So if you need to click all the links in an email, you can choose that sort of rule. There's a few others, like forwarding to your email address that you've signed up with. So if you want to make sure that the email did come in, it'll kick it back to the account email that you logged in with. But yeah, being able to use a system as a heartbeat like Jack sort of described, where you're ensuring that the delivery is happening and verifying that using the API, that's pretty much the typical use case. There's another sort of load testing approach that people take too, and we'll even set up A what we like to call like a black hole. So the messages will come in and the server will provide a red receipt, but they won't be stored. It's just to check, make sure that those things are actually coming through. And if you're trying to see if your emails are going out of your system at the right pace and cadence and amount, then that sort of approach can work for you.

Too. I was just going to say that's one of the real advantages of having gone and learned about the testing and QA community is we discovered that they really did have all these use cases and needs that weren't being directly solved by anybody else. And we got to implement some of those solutions. And so just like the original Mailinator wasn't built to solve any particular problem, people sort of discovered uses for it. The enterprise Mailinator kind of has the same feel to it. It doesn't fix one particular thing. It's a general purpose tool. And we'll even develop features for people on request in particular, if it's the sort of thing that we think, wow, nobody's doing anything like that. So Brooks mentioned what we call a little black hole. Sometimes you just need to test sending. You've got to see if your sending infrastructure is sufficient to the task at hand. We had someone who needed to send a million messages and find out how fast they could do it. And it turns out that 88 seconds was fast enough for our system to receive the million messages that they were sending. We have people who, as Brooke said, use this as a heartbeat. Sending emails reliably is the core of their business and they need to know that that's happening all the time. So they set a parallel sending infrastructure and they send to us and they can continuously check that those messages are being received, not just at all, but being received quickly because those messages have links in them that will expire. So they can be continuously checking to see that those messages are arriving and fast enough so that the links are still active. And if they ever see in our flow that something's not arriving fast enough, they know that they need to check their production system because it might not be sending properly to customers. There's so many other use cases and they're all new to us. In the past few years, we just discovered that people need to do this. We have a company who uses this as part of their continuous integration development. Every time they make any change to their code, everything gets up and runs. Once they use us to make sure that the outbound messages from their new build get Received.

Correctly. Nice. Yeah. Load test capacity planning seems like a great use case for this. I think I saw another one on your site. Verify the onboarding loop. What's that all about? I've never heard that term.

Before. Let's say that you have people signing up for your service or your business or clients signing up on your website. There's an onboarding process. Somebody needs to create a user account, they need to set a password, they need to get a link to their new to whatever service you're providing to them that's customized to their use. And the way that you verify that that's working is you sign up a test customer and somebody who's in the testing shop on the side of the developer says, well, what happens if somebody fills out this form and click Submit? The way where we come in on that is those messages, because they're test messages don't go to real clients. They come through our system. And then you can look and see that the email was received properly, that the email is formatted correctly, that the links are working. You can click on the links manually or you can have our automated system click the links and you'll see for every signup for everyone who's onboarded into your client or customer system that they're able to get and execute on the message you send to them when they first sign.

Up. Nice. So I may just be off the wall here, but just for myself, I have an edge. I have a case where I have a email marketing system. I send emails and sometimes people say, oh, I didn't get it. Or it goes into this spam or promotion folder. I don't know. Is this a use case where you can check the deliverability to make sure that it's going to the right thing? Or is that beyond your. What you can do? Because that's up to Google or firewalls or whatever that companies.

Have. What a great question. That's something that has recently come up. Somebody's asked about that. They don't want to test receipt, they want to test deliverability. We don't have the capability to do that right now, but it's absolutely on our roadmap now. We've gotten actually I think three, at least two that I can remember. And Brooks I think fielded one from someone who wants to test exactly that. How much does this look like spam? How likely is this to get through? And that's on our roadmap as a feature to develop and add into the.

Product. So when I looked at one of the videos for the mail Nadir, the API seems quite extensive. Is there anything that people can use Mail native for that they may not know about? Or like, how many things can they do with the API? It doesn't seem like just a simple one, two type of deal. It seems very extensive of the type of activities you can perform with the API. Is that.

True? I think that is true. I haven't ever thought about it in those terms. And we're on version 2 of the API. The first version of it was, oh my God, we need an API and developed it as fast as we could. And the version two, it's more like a set of Legos. So nothing about the API is designed to satisfy a particular task or a particular workflow. The API is designed to have all the tools you might want to have for interacting with your test flow, with the test messages. And we see people build a lot of different things out of those.

Legos. Nice. So I also noticed when I was researching is especially for developers and testers, they're always looking for something somewhat free. I think you said you were launching a premium version or a premium tier called Verified Pro. Is that something you could talk.

About? Yeah, absolutely. I'll field that one. After, you know, looking at, you know, who our end users were and really wanting to lean into the community, making sure that as we developed Mailinator it was done for the audience and what those needs were, we realized that we had a couple tiers of paid subscriptions that would work if you had a full time job and you needed Mailinator all the time. But for those individual developers or testers who might be contract employees or someone just likes to develop things on their own, we realized that there was a gap and those people might not have the budget for even a $30 a month sort of plan. We said, hey, you know, it would be really great if we could offer the level of performance that private mail nator, meaning like a paid subscription could give, but at the free sort of level. If you're using a public Mailinator, again, there are these limiters in place, there are filters that are adaptive so that we can prevent somebody from spamming or creating a thousand discord account or things like that. So we said, hey, how can we give away Mailinator with access to the API so you can truly get the full breadth of what the product can do, but also ensure that it's going to the right people. And we said, hey, what if we created a tier that was somewhat meant for those people and we can kind of verify it in a way by asking for either a phone number or a LinkedIn profile to say, okay, great, this person exists. It's not just, you know, creating some pseudonym and leveraging our system to, you know, span out some stuff. So we created Verified Pro. And Verified Pro really gives you access to the API and then gives you some limits that are commensurate with, you know, an individual who's doing testing. Now, the hope is that if you are using Verified Pro, you are exposed to Mailinator and then you decide, hey, like I'm going to suggest that we use this as part of our team and then somebody can get a paid subscription down the road. But really making sure the product was serving the needs of an individual.

User. Absolutely. And Jack, I guess this brings you back to your roots when you start off with the free version. This might be controversial. I like paid solutions because I know there's a team working on it and developing it and creating new features. So it's almost like the best of both worlds that you have going on right.

Now. That's absolutely true and I think Brooks nailed it. There's a real need out there for people who have small testing needs, or maybe they're a contract worker doing QA work. They can't justify a real subscription, but they're real professionals and we, like you said, going back to our roots as a free product. Brooks came up with this idea of Verified Pro. It was a winner with us from the very start. And people who are in the VP program have full access to the commercial enterprise Mailinator. They get all the features that anybody else would have. They just don't have the volume. Anybody who needs to do testing occasionally, a few here, a few there, they can't justify a subscription. And we still want to be able to serve the broader community. And Verified Pro is our first attempt at trying to do.

That. There are any new features that you all are working on in the pipeline that are coming out for.

Mailnator? Well, you mentioned about sort of deliverability, testing the spamminess. We've got something in the pipe for that. We also have talk about going back to your roots. Paul started messing around with Telnet as a protocol just because that's how somebody amuses themselves on a Saturday. If their name is Paul Tema. And he came up with basically a console application for the Mailinator system. It's live, it's not even beta, and it's probably broke at the moment because he's working on it nights and weekends. But if you're someone who used to live in what we would call text mode, It's a way to interact with the Mailinator system as if you had a Telnet terminal. He doesn't really use Telnet, but that's sort of his goofing around on the weekends thing that he built. But it's going to be a part of the commercial product, because what that allows you to do is that's the core of an integration. So what we can do with that once we've got the console working, is put that in something like an integration for Slack. So from within Slack, you could call to the system and say, show me this email, or you could post a message into the Mailinator system. And so if the way you do collaboration is in Slack and integration is one of the really useful ways to interact with other systems and we've got something like that coming for Slack. But because again, it's just a Lego we built, it's not for a particular solution that's going to allow us to do things like put a way to interact with the Mailinator system in a little window on a webpage. It'll make it really easy to do something interactive on a mobile platform. So this started with Paul goofing around with an old protocol, and now all of a sudden it's something that we think will serve another need that we've heard from the QA community, which is they don't necessarily want to have to be logged into the website. They want a way to collaborate in the platforms that they use for interacting with their colleagues. And that's something we're hoping to have implemented a little bit later this.

Year. So we touched on email the bulk of this interview, but I know you also do a lot of other things. Like you, you mentioned about sms. What's the new functionality with SMS that's built in that can help folks with that type of.

Testing? Well, the deal with SMS is that it's a mobile world now and a lot of people do interactions with their onboarding systems or interactions with their customers or clients via text messages, and those things need to be tested just like your emails do. So we have SMS capability now in the system for a couple of years now. We rent basically real phone numbers from telecoms and we integrate them into your Mailinator team. So if you're just doing email, your Mailinator team sees the email messages flowing in and you can interact with them. You can also now send text messages to the number or numbers that we've attached to your team and you'll see those flow into your testing space exactly like it was an email message. They're available in the API or through the web interface. And it's a way for you to test, for example, if you're sending one time passwords or if you're sending notifications to your customers via sms, you can send them to a test number first before you make that live in your production.

System. Is there a breakdown of how many people use the email versus the SMS feature or is it just across the.

Board? We find that all our customers use the email functionality. We've got probably a third to a half of our customers use the SMS function. You gotta need it. There's an extra cost to that. We have to rent the numbers. And so putting SMS on a Mailinator test team costs a little more. We also have a third way to get messages into and out of the system. A lot of times it's not possible to test a message flow directly. And so we've implemented webhooks both inbound and outbound from a Mailinator test team. So if you can't simulate exactly the message that you want to send, if you can't make the message you want to send happen the way you want to, you can simulate it with a webhook in and you can also get notification for when messages arrive by flagging them with an inbound filter and sending a webhook.

Out. Perfect. Okay, Brooks and Jack, before we go, is there one piece of actual advice you can give to someone to help them with their email SMS testing efforts? And what's the best way to find contact you or learn more about.

Mailnator? Well, the easy one is you can find us@mailnator.com that one's super easy and we'd encourage you to sign up for a free trial. There's one note to make. You know, getting a trial gives you access at a business plan level. So it gives you a team versus getting a verified Pro subscription, which you're welcome to get. But make sure that you realize that there is a difference. You can't have two different tiers on one account. So want to make that note clear for everybody. But from a best advice I can give somebody to use Mailinator, I don't know, I'll hand that one off to.

Jack. The best advice is what any tester knows. Test first. We get a lot of customers that are people who know that they need us for their testing. But just like the hbo, I don't want to badmouth them. They've been super great to work with, but they made a mistake. They didn't test something before they went live and so I'm the last person to give this advice because I didn't grow up in the testing world. But we've learned a lot about it and we've huge respect for that. I think that testers need everyone else to know you got to test these things.

First. Thanks again for your automation awesomeness. The links to evident value we covered in this episode. Head on over to testguild.com a409 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good.

Cheers. Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.