**MOOPs** Episode 3 - Justin Norris & a slow-to-find mistake

Hi, I'm Michael Hartman.

I'm Naomi Liu. And I'm Mike Rizzo. And this is OpsCast, a podcast for marketing ops pros and rev ops pros created by the MoPros, the number one community for marketing operations professionals.

Tune in to each episode as we chat with real professionals to help elevate you in your marketing operations career.

Justin, thank you for coming Thanks for coming on to record what we fondly refer to as Moops TV, so sharing your marketing ops mistakes. As with all of our guests, we deeply appreciate you taking the time to share and have an opportunity for us all to learn from you. So why don't you kick us off, some introductions, tell us a little bit about who you are and what you're currently working on or doing in your role, and just a little bit about your background and years of experience in marketing operations.

Sure. Thanks, Mike. Appreciate you having me. It's good to be here. And I'm Justin Norris. I'm the Senior Director of Solutions Architecture at Percuto. We are a martech, Marketo, and Workato-focused consultancy. And I've been working in, let's say, marketing more broadly. It'll be coming up on 15 years now, but for the past 10 years, really focused on the RevOps and MarketingOps and kind of business systems space, first in-house and then for the last 6 years in consulting where you get a chance to learn a lot of things. You get a chance to make a lot of mistakes too. So I've done my share of both.

Yeah, yeah, absolutely. It's quite a different world being in-house versus in the consulting side of things. Lots more to interact with and touch and probably opportunities to break things. Yes.

Yeah. There's hopefully you don't break too much, at least not in an irreversible way. But yes, there's no shortage of either.

Yeah. Good stuff. Well, so at what point in your career When in your career did this mistake we're about to hear about really take place? Was it early, mid, yesterday?

As I say, there's no shortage. And I, you know, when folks get started in consulting, there's often a little bit of concern I find around, oh no, am I gonna— like, there's a hesitancy, am I gonna make a mistake? And we try to create a safe space for making mistakes, you know, put guardrails in place to limit the impact of mistakes, but also You know, have everyone understand that it's going to happen. And I often say I've made bigger and worse and more expensive mistakes than anyone. So you know, if I can can do it and live to tell the tale, and I've been around for a while, then you'll be okay. But this mistake was sort of midway through my time at Perkuto. It actually happened probably around 2018, although the impact of it was not detected for some time. And then the fix for it. Took even longer than that.

Oh, interesting. This is juicy. I'm excited. Okay. Wow. A mistake that actually took a little while.

This is good.

Yeah.

So the mistake was actually on the surface of it quite trivial. It was a Marketo and Salesforce integration. And for anyone who knows a little bit about the Marketo Salesforce integration, I know not everybody does, but When you're integrating those 2 systems, you know, you define what fields you want to include, you define permissions on those fields, whether they're read-only or writable. And then there's a final step that you can do in Marketo, which is called field blocking rules. And what those rules do is define which systems or actions are able to overwrite an existing value in a field, which is a useful feature. So in Marketo, you can say, all right, Marketo can write to the job title field for a new lead or the address field for a new lead, but if there's already a value there, I don't want a form fill to be able to overwrite it, or a form fill could overwrite it but a list upload can't. So you can actually get quite granular and decide which sources can overwrite. And sometimes this is, you know, not something that people are that interested in, or maybe they'll do field blocking for a few things but it's not a big concern. But in this particular issue, you know, it was a complex environment, very large company, and a complex project for that reason, with a very complicated as well, and way well-developed and established Salesforce organization. And there was a particular concern from this client, in particular their Salesforce team, that Marketo was not going to overwrite Salesforce data. So we spent a lot of time actually establishing what those field blocking rules should be. And in this case, they were actually very important. And the irony of the whole thing is, is that it was in many respects a model project up until that point. And we do a lot of testing on everything we do. QA is a big part of our process and probably went through at least 30 different individual usual unit tests and QA testing just in Salesforce, the permissions of the sync user testing in Marketo once the systems are integrated. We tested it in a sandbox environment, you know, we integrated those systems first and then we did it again in production. And then the last and final step was to apply the field blocking rules and it just didn't do it. I mean, there's no other way to describe it. It just got missed and specifically by me. I say we, but really I was the lead consultant on the project and was executing it. got. And, um, you know, we celebrated the successful launch of the integration. It had been a long road to get there, you know, with a lot of change management and different things. Um, and, and, you know, winning over the sales force team— there's a variety of aspects to the project beyond just your standard integration. So we launched, we're successful, and then I don't know exactly when it was, but it was some, you know, maybe 8 to 10 months later perhaps that it first came to light. that Salesforce data was being overwritten by Marketo.

And how, like, who noticed that? Like—

I think it was probably the Salesforce team that noticed it, you know, which is, you know, this is a stakeholder that we'd spent a lot of time building up trust with. And so, you know, definitely not the team that you wanted to have noticed that. But yeah, as I recall, it's a bit fuzzy from memory, but somebody brought it to our attention, maybe gave a report and said, hey, what are these values? How is this possible? So, you know, you start digging in and, uh, and I pretty quickly identified, oh no, we didn't, we didn't set the field block. We didn't set the field block. So what could we do? Uh, you know, obviously had to take accountability for it and, and just explain. So here it is, we didn't do it. It was a mistake. It was a miss. Um, you know, we're gonna fix it. And so the process for fixing it was, uh, number one, trying to identify records that were impacted. And we were able to do that, you know, through some smart lists in Marketo and some reports over on the Salesforce side. And the types of data being overwritten, you know, in some cases it might just be, you know, they had nice clean data for a contact and then that person filled out a form in Marketo and maybe they're not capitalizing their information in the way that, you know, we would have liked. Or in some cases they're submitting other data which was considered less trustworthy. There was also some list uploads which might have been impacted. So there was a soup of of changes that came in.

About how long did that go on though? Like, are we saying like a year?

Like, before we noticed it or before? I can't remember exactly at what point we noticed it. I was just looking back through some history to see how long it was till we actually fixed it. And it was about a year later when the final fix I'll describe went live. So there was a big history of data and it was fortunate, a big history of data that had gotten changed. It was fortunate that we noticed when we did, or at least that the Salesforce team noticed when they did, because Marketo has a, a window of data retention when the, the changes are stored.

Mm-hmm.

And if it had gone beyond that, some of those previous values could have been irretrievable, or at least certainly not as easily retrievable. So it was good, you know, that we found out when the history of changes were still extractable. So basically what we had to do is, you know, identify the records that were affected, and then extract from Marketo using the API the history of data changes. And fortunately in Marketo, when a data— when a change to a field is logged, it'll log the new value and it'll log the previous value. And those are both stored as attributes of an activity. And so you can through that take a particular change and essentially reverse it by saying, this is what it was before and I'd like to go back to that. Now, of course, things are not so straightforward because in some cases, there would have been a lineage of multiple changes, you know, various things overriding.

Right.

Maybe Salesforce was then subsequently updated again. And so we had to go through a whole process of defining what are the business rules for this, this change, like, all right, if Marketo was the most recent update, then revert back to the most recent Salesforce update, but not the most recent Marketo update. But if Salesforce was the most recent update, then just keep that value, don't do anything. There was also exceptions, for example, the unsubscribed field, we would allow Marketo to have priority because that was considered a higher source of truth, but only if it was true. But if it was, you know what I mean?

You start getting into these issues.

Yeah, you get into these weeds, right? But we had to. And, you know, so the documentation just of the fix for this could have been, you know, 10 or 20 pages long. And then we essentially had to build an application that would extract all these changes from the API and then process them. and produce a canonical value. So it would say for this record, for this field, this is the canonical value. And then the testing on that was extensive. We had to validate the QA of that solution was probably more extensive than the original QA.

I can imagine, right?

Because it's complicated. You have to go through each of those scenarios that I just described. And is it working? Is it not working? And in some cases it wasn't working. Go back, refine the logic. And then you run into very trivial but still important things like formatting issues or, um, non-standard characters that are being rendered improperly due to some kind of encoding issue. And so each of those things— so it was some months of, of working on this solution with a few different developers, uh, until finally we had it ready and could then take that data, import it back to Salesforce, bring everything back to its you know, its original condition and, and then close the books on that one. Um, but I tell you, I will never forget to do field blocking rules again. It's just seared in my consciousness. Yeah, I believe.

Yeah, it's the, uh, the eternal checklist is like ever-present in your brain now. You will always think about that.

And that's what it was though. I mean, we have a very detailed grid of QA tests that we would run through, and we just didn't have a row for that item. It was just kind of implicit or assumed that you would do the field blocking rules. If there's just a row, which there is now, that said, are the field blocking rules complete based on the spec, then that would be fine. And that's how it is today. But just the absence of that line, you're reliant on the individual's awareness or memory of the need to do that. In my case, I didn't remember it in that instance. And in 9 out of 10 cases, or even 90 out of 100— 99 out of 100 cases, let's say, the impact The impact of that mistake would have been limited or none. But in this one particular scenario, it was huge because of the particular requirements that the client had.

Right. Wow. Yeah. Well, hard lesson learned to make sure that the checklist has all of the right checklist things to check off, but hey, this is the things that we go through in marketing operations.

Well, it highlights the importance of QA generally, you know, and no matter how smart you think you are, how experienced you are, I mean, I'd already by that point been consulting for 4 years. It's not like I was a rookie in this game, and we have very good processes, and we have extensive QA. I mean, like in every area, we're already going meeting whatever the standard or beyond the standard, you know, that you would consider professional in that situation. But you can have gaps. And another thing, another issue there was that I was also performing my own QA, which is so actually it was another MOOCs. I would say we have a process that it's always a peer. that does the QA.

Yeah.

You know, like kind of a second set of eyes on everything. And in this case, if I'm remembering correctly, I believe the project, you know, we were in a hurry, we wanted to show value. And so I did the integration and I said, all right, I can test this myself a bit. You know, what's the harm? And had I involved that second person, you know, maybe they would've remembered. So there was sort of another compounding process error. So it also shows in some ways, I like— I don't like when we make these mistakes, but I like when the mistakes are resolved of breaking process only because it highlights how important that process actually is.

Absolutely.

It shows you that the process is working. So when you deviate from it, things, things go wrong.

Yeah.

Yeah.

I think so many organizations feel that process is a 4-letter word, right? It's a bad word. And in reality, in marketing operations, revenue operations, really any operational role, process is what creates scale and efficiency. Right. And when you follow process, you won't end up spending many multiples of months and time and budget on how Having to fix something that for 99% of other people may not have been a— clients may not have been a problem. And in this particular case, it was a pretty substantial one. Sounds like the outcome was make sure the checklist is updated to reflect all of the items that are necessary to go through on that QA checklist. And then in addition to that, like you, I also need a second set of eyes because Every email that I ever send to our community is because I didn't get a second set of eyes. Anytime I send a moves email, it's likely because I had no one read it.

Right. I mean, that's, that's, that's it. And yeah, it's funny what you say about process being a 4-letter word, and I can totally relate to that impression. And I think the issue is not so much about process. I think it's just that there's a lot of bad process out there. Yeah. And, and that's what gives it the bad name. And Our co-founder at Procuto, our chief operating officer, is a Lean Six Sigma black belt. And so process is very much woven into the DNA of kind of how we do things, even from a very early stage when it was like less than 10 people when I joined. And we still had a level of process maturity that was way beyond what you would expect. But what I kind of learned from that and what's helped is that, you know, a lean process, as the name suggests, is a process that actually makes things better, makes things more efficient, and makes quality more predictable. It's not there to get in your way. It's actually there to relieve the burden of having to think something through every time and to ensure, um, just a predictable output. When you see what good process can do and what goes into designing it, then you really, uh, fall in love with it, you know, not as something that's there to slow you down, but actually there to speed you up while preventing, you know, the sort of issue that happened in this case.

Yeah, absolutely. No, I love that. I love the idea of, uh, You know, this lean process, and that's, um, that's a great way to just, just play back the importance of process. Um, yeah, I appreciate you sharing that with us. So, uh, what would you say as, you know, final thoughts, um, to someone who's, you know, maybe made a similar mistake or, uh, is about to make a similar mistake? Um, and just general advice for, for those in marketing and RevOps?

I, I mean, I think the advice when you make a mistake is the same in RevOps as it is in life generally. It's, you know, take, take Take ownership, take responsibility, make it right. And the cost of doing that is always going to be less than the cost of not doing that, I think. And it'll be painful in the short term, but in the fullness of time, you'll look back, you'll be glad you did it, and you'll be able to talk about it on a podcast one day. You know, it'll all be fine, but it certainly hurts. And, you know, kudos to My my peers, my team, my bosses at the time. You know, nobody gave me a hard time about it. We all just buckled together and got it right. And maybe if I did it again, I would get a hard time. But you know, it's it's always okay to make a mistake once. Like that's understandable. If it's repeated, then there's maybe there's something there that really needs to be looked at. But things happen. We're we're human.

You know, we're human. Yes, absolutely. Well, great, Justin. Thank you for sharing your your moops moment, or at least one of them.

I got more. You can bring me back.

We'll do like a whole—

It'll be a series.

Yeah, it'll be a whole series of just Justin Moves moments. Yeah, we'll just keep going. But no, thank you again. I appreciate you taking the time. I feel like everybody probably learned something today.

Appreciate it. Thanks a lot.