**MOOPs** Episode 8 - Marketing Ops Manager @ Autodesk, Weisi Kang Shares Her Marketing Mistakes and What She Learned
16 Feb 2022 · 17 min
Weisi Kang, a marketing ops manager at Autodesk, describes a Marketo disaster: an alert step sent a monthly newsletter to 8,434 people, which triggered alerts to 2,000+ employees thousands of times each, crashing the email server. She walks through what went wrong, how the CEO found out, and the governance changes that followed.
Chapters
Hi, I'm Michael Hartman.
I'm Naomi Liu.
And I'm Mike Rizzo. And this is OpsCast, a podcast for marketing ops pros and rev ops pros created by the MoPros, the number one community for marketing operations professionals.
Tune in to each episode as we chat with real professionals to help elevate you and your marketing operations career.
Okay, Andy, thank you so much. for getting on to share an episode of what we fondly refer to as Moops TV. I don't know if that'll stick forever, but for right now it's holding true. And we are really excited to learn about the marketing ops oops that you might have encountered in your career. But before we get going, we would love to learn a little bit more about who you are and where you're currently working. Obviously we see a nice little background there, so that's awesome. And then just a little bit about your background.
Tell us about yourself. My background, not my background, right?
Yes. Nice.
So I'm Andy Carone. I am the VP of Consulting at Revenue Pulse, and I've been there for about 2 and a half years now. Uh, I started out actually more on the sales ops side of things, uh, on Salesforce back in the day, uh, and became a marketing automation and then Marketo customer in 2012. Okay. Uh, yeah.
So you came in on the sales side of things, which is great.
Uh, yeah, I started out in Salesforce and then got to own both systems at my first company, which was— I was so spoiled.
Yeah, yeah, for sure. Yeah, I was gonna say arguably spoiled, right? Uh, we don't always— we don't always like owning both systems, but sometimes it's helpful. So where were you in that track record of You know, in your tenure of being in kind of a sales ops, marketing ops role, at what point in your career did, did this mistake you're going to share with us happen?
So this particular error occurred several years in, probably would have been after I'd been in Marketo easily 5 years at that point. I owned a system that was leveraged globally by multiple teams. A lot of end users, a lot of casual users, and despite many discussions about a potential center of excellence model, the best that I was able to manage was a loosely policed system with some approval processes in place and a lot of program templating to get people to follow just an ABCD step within the system on deploying their programs.
Gotcha, gotcha. Okay, so, so this, so this environment, lack of controls, you know, people can get in there and do— sounds like you tried to get some organization around it, struggled a little bit. So what was it that, that the system was trying to do for you or this program that was in place? Like, what was trying to be accomplished?
Yeah, so in participation with one of the marketing teams that was not a common user, of Marketo, they were actually more on the partnership side. There was a monthly newsletter that needed to go out, and there was a lot of back and forth on how the internal teams would know that that email had gone. They wanted to see it. They, they essentially wanted to be participative in it. But when you're sending a newsletter email and trying to gauge engagement, and then you're sending to a bunch of internal stakeholders, well, that's gonna, you know, massively inflate, uh, your, your Your potential impact, right?
Oh yeah, the click-through rates, the open rates. Oh yeah.
And so this like, I wanna CC, I wanna BCC sort of conversation was very debated. And of course at that time, that functionality did not yet exist in any incarnation inside of Marketo. So it was, we can send you the test, we can send you something, but we recommend not having you be part of the audience For this particular email deployment.
Totally. You know what, that even in my most recent roles, we would just copy paste the email into like a G Suite, right? And just hit send or like send a test email to one person and that person would forward it internally.
Or publish, you know, the code out and then put that on a page where people could go play with it or whatever. Right. There's like a bunch of different options, but they were just dead set. Like, we want to get the email at the same time. same experience like we want it. And so in an effort to be accommodating, the individual who was scheduling this email to go out had member of list, right? So there's my smart list as member of smart list.
Yes.
And 3 flow steps, which was send email, change program status, Send alert. Now, in your brain, if you don't really think through it, you're thinking, okay, send alert, that's going to send an email to an individual. Cool. Like, that's basically faking a BCC. And in a scenario where maybe you're doing that with a demo form request or something like that, that might actually work, right?
Potentially.
I could see that being Something you might do, right?
Yep.
The challenge with this particular setup was that because it was a batch, which you never want to schedule an alert against or put that in your flow, right? Like, that's probably not a great idea. And also that of the 3 emails contained in the alert send-to, one was an internal distribution list.
Oh, okay.
Yeah. So what that meant was some quiet, rainy Thursday morning that hit its scheduled batch time. And I was opening up my email, just getting to my desk and went, uh-oh. And then my phone rang and I went triple uh-oh. And next thing I go in and It's like, stop the execution of that batch. Like, there's no calling it back. These are out the door. Like, it's a done deal.
Oh my gosh.
Yeah. So the output of this was that for every individual who received the newsletter, an alert of that same newsletter was also sent to the individuals in the alert flow step, right? Because one of those was a distribution list that went to over 2,000 employees. And because the send list for the newsletter was 8,434 people, we ended up with a scenario where the company's server got shut down because over 2,000 people got the same email over 8,000 times.
Oh, oops.
Yep.
Ouch.
More like—
goodness.
Yeah, no, I know. Plenty of choice words actually came out of my mouth at that point. But also, you could see the intention. You could see someone trying to do something in this system to accommodate a request that never should have been made of that system, that there were other ways to account for.
Mm-hmm. Mm-hmm. So who on the— like you said, you got a phone call. Who actually noticed it first? Was it you or someone on your team?
I was noticing it as Someone else was calling me to say, "Why is my inbox blowing up right now?" And then very quickly, you know, IT was looped in. Everybody was looped in. You know, the CEO of this company was on that distribution list, right? Like this was front and center. No way to do anything other than be like, "Hey guys, this is very shameless to me, but I don't care." Hey guys, this is why we don't do CC or BCC. BCC on emails that are going out in Marketo. We can send you a test. You want to test next time? Yeah. Good.
That's a hard lesson learned. That's for sure. Oh boy.
I will say it had the outcome of definitely stopping any future requests to be CC'd on a marketing email. Not necessarily the way I would have liked to have gotten to that resolution point on that particular topic, but effective.
Oh boy. So that's just— yeah. Wow.
I— yeah.
Well, hey, you got there, right? So, so in terms of remedying this, like, is it at this point, it's just a fall on your sword moment, right? Where you go to the team or the stakeholders that received all this stuff. Well, first of all, your servers are shut down, so you got to boot that back up. And then what? You send an apology?
Everyone's inbox has to be cleaned out, right? Like there's a whole process of remediation organizationally that has nothing to do With Marketo or, or even marketing at that point. And it's essentially a process of saying, we know why this happened and it will not happen again. And these are the measures that we're taking to ensure that this does not occur, right?
Yeah.
Ownership of the newsletter and execution of said newsletter shifted and approval processes went into place that had not been there previously. That said, there's now 2 different levels of people that need to look at this from both an email approval, which is already in place, and an automation approval.
Right. Yeah, that makes a ton of sense. So, so during this remediation process, just unpacking that for just a second, were like, who got involved? Because this, this, like you said, it falls outside of just the system now. So did you end up calling a handful of meetings real quickly with— was it you or just like some other team members?
Like, Yeah, so I had my, um, direct manager who owned marketing operations as a whole schedule with teams to coordinate on that end so that I could focus on doing what I needed to do on my end, which was wonderful. Um, and we did sit down and basically have a postmortem about what had happened, why it had happened, and how we would avoid it future state. But at the end of the day, it really was Someone trying to accommodate a request that should never have been made or accommodated in the first place, right? Like, this is not— yeah, 2,000 people inside the company probably don't need to get the partner newsletter, actually.
Totally right.
Like, that, that's probably not something to be putting in their inbox either. Um, either they're not going to digest it or it's eating up inbox space or their time or what have you, right? Like, yeah.
And deliverability and sender credibility, right? Like your domain's rating is tremendously important. So if you just keep sending people emails that they never open, that's not going to go over well.
Right. And I mean, this was an alert, so it was a little bit different.
Sure.
But still, I mean, it yet again proved the point of why we didn't want to be sending those emails as part of the newsletter blast. Right. Also, right?
So so that you put in a process, you guys put in new checks and balances, and I assume obviously that mistake never happened again in the future.
Correct. So that's not not on my watch anyway.
Yeah, you know, like I've washed my hands of this now. So yeah, so this is good.
Yeah, yeah.
Well, cool. So all right. So final parting thoughts: What would you say to someone who's made a similar mistake, is maybe about to make a similar mistake, or? Just like kind of broadly, like your advice for marketing and revenue operations professionals out there around, you know, avoiding the Moops moment. Like, yeah, what are your thoughts there?
Um, you know, at some point in your career, someone's going to ask you to do something and you're going to be rushing, or you're going to be trying to be accommodating, or you're going to, you're going to be going out of your way to do something that maybe isn't something that you would normally Do, whether it's because you're rushed, whether it's because you're new in a role, whatever that might be, new boss you're trying to, you know, impress. And I think it's just, A, it's imperative to have oopses during your career because that's how we learn. That's how we grow. Trying new things and failing at least means that you're actually growing as an individual, right? If you just stick on the straight and narrow and you never make a mistake, then that means you never tried anything either, in my opinion.
Yeah.
So sometimes, you know, you fail, but if you're trying something new and out of the box, maybe try it with a smaller audience first, or not a distribution list or what have you to see how that plays out, right? Test before you deploy, especially new things or things that are a little bit different.
Mm-hmm.
But also be prepared to own your mistakes, right? To go in front of your boss or your boss's boss or your boss's boss's boss and be like, These things occasionally do happen in systems like this because they need to be this powerful to execute and do the job they're intended for.
Sure.
And this proves point A, B, or C potentially of something I've been advocating for, in my case, center of excellence, right? This reinforced that recommendation.
Right.
Unfortunate way to have that occur, but it did, right? So let's talk about not just from a system or user perspective on how we can ensure this doesn't occur again, But if need be, organizationally to support that this isn't even a potential.
Right. Yep, absolutely. Get it outside. And those conversations are tremendous, right? Getting outside of the tool itself and its kind of capabilities, whether they're limited or robust. It's taking it beyond that and saying, how do we ensure that across the organization we're aligned so that in your example, that request— everyone understands that that request isn't really a sound request and here's why, right? And creating that alignment across the org. That makes sense.
Exactly. And I think the other thing there is there's a lot of power in being able to say to your boss, I know what happened, I know what happened, and it'll never happen again, right? Being proactive about coming to them with, hey, this thing just went down and you need to know about it, has— it's invaluable, right? If you wait to be called into the principal's office and you're all cower-y in there, which is the natural state, right? Like, we don't really want to necessarily be in the spotlight on things in general.
Yeah.
Certainly not like this, right?
Totally.
But at the same time, back to life. It happened. We can't go back and change it. So what can we do?
Yeah, absolutely. No, all great advice. And I think the more professional we are about handling— excuse me— those situations, the better, right? And it just— we all want to crawl under a rock when stuff like that happens. But the reality is that hopefully you've met with a boss or have a manager who's either already managed someone like you who's probably made mistakes too, or has literally been in your shoes. So, or you just sit like, you know, say, I promise this happens all the time. Shameless plug, go watch Moops TV, right? Like, this is nothing—
this is real. To you, literally, that this is so common.
There's a whole thing, there is an online segment about it.
Here you go. Yeah, that's right.
Anyway, well, Andy, thank you so much for getting on and sharing your MOOPS moment with us. And, you know, thank you for letting us learn from you. And hopefully others who are watching this will create a center of excellence and maybe avoid those costly mistakes. So really appreciate it.
Yeah, it's my pleasure. Thanks for having me.
More from Mike


The Missing Layer Between Strategy and Execution in Marketing with Charral Izhiman



