How To Weather The Storms When Trials Hit
19 Jan 2017 · 1 min
Russell discusses two major ClickFunnels outages caused by database migration failures and Amazon issues. He explains how his team rebuilt the platform and shares what he learned about communicating with users during crisis: over-communicate, show empathy, and treat customers like family rather than transactions.
Welcome to Marketing Your Car, the podcast for funnel hackers everywhere. You're now part of a small group of underground entrepreneurs who are trying to change the world through the products and the services that we believe in. But because we don't rely on venture capitalists, we don't have financial safety nets, so we have to do things differently. We have to do things smarter. We do that through an art and a science we call funnel hacking. This podcast will give you a glimpse at the daily life of a funnel hacker.
So let's get started.
Hey everyone, this is Russell.
Hope you guys are doing awesome today. So today's Marketing Your Car is actually happening in my office for a couple reasons. So first, I was getting ready this morning. I was planning on waking up at 5 to start working on the book. I had the last edits of the book going, have to get done before the end of the month if we're gonna get published in time for our launch date of April 18th. And so, but then I was, only had slept a couple hours that night, so I just Every 9 minutes for 2 hours, kept pushing snooze, and finally it was like 7 o'clock, I'm like, oh crap, kids gotta get up. So my wife and I got up and started working on the kids, get everything ready, and then check my phone to find out that ClickFunnels was down. Now some of you guys probably had the same sinking feeling in your chest this morning when you saw that as I did. And so I just wanted to kind of walk through what's going on in my head, because one, so you guys know, but more importantly because there are times There's going to be disasters and problems and issues that hit your business. And I'm just going to kind of go through what I did and what our team did, and hopefully it'll help some of you guys when disaster hits for you. And so one good thing I have on my side is I've got the most amazing development team on planet Earth. And so by the time I noticed that, they'd already known and they were on it and they were working on it. And what's funny is The last time we had an issue, and I'm pretty sure, I paused and I did a podcast about this when I was in London, but it was over a year, a year and like 3 or 4 months ago, so almost like a year and a half was the last time we had any issues. And it's funny 'cause at the time ClickFunnels had been built at a database company called, I think ClearDB was the name of it, and we were kind of outgrowing them and we knew that and we were trying to move over to these other servers and we were trying to move and they wouldn't let us, they basically handcuffed us, basically saying like, You know, like, no, we don't want you to move your data. If you do, you're gonna have to do this other way, which is gonna mean, you know, downtime for 3 or 4 days and whatever. And so as we were getting closer and closer to what became the day— the, the first day our servers crashed, um, uh, things started working, you know, less and less and more issues, more hiccups. And it was just like their company couldn't handle what we were, we were putting through to them. And at the time we had about 10,000 members. And so We've been wanting to shift over, we couldn't, couldn't, and I was actually speaking at an event in London with my family, so I was in a plane flying to London, and when I landed in London, figured out how to get international phone, which is not as easy as you'd think. Look at my phone and I'm awake, you know, I basically get there and realize that ClickFunnels is down, and messaging our team and they're freaking out, everyone's freaking out, and I'm just like, huh, and we know the issue, but basically, Luckily on my team, Ryan Montgomery and Todd Dickerson kind of called an audible and they were like, hey look, we can either wait for ClearDB to rebuild or we've got to rebuild this whole structure over on Amazon. We want to do things, we want to move things up. And so they made a decision to start rebuilding. And so they jumped into it and rebuilt basically the whole database structure of ClickFunnels over here in Amazon and then started restoring entire ClickFunnels database from from an old backup— not old backup, backup that had been— I think we back up every, I don't know, 20 minutes or 30 minutes or 10, I don't know, whatever it is, and restore from the backup. And ended up, I think we were down for 12 hours, which was like insane. Like, people were ready to kill me. Luckily I was in London, no one could find me. Um, but it was, uh, it was, it was tough. Um, and what's luckily is because, because our developers had the foresight to like rebuild instead of waiting, it ended up taking 3 weeks for ClearDB to restore our servers. So we would have been down for 3 weeks if it hadn't been for, um, some amazing people. And when I remember when I got there, Facebook was blowing up, my message, like everything's blowing up and people were angry. And I was like, what do I do? And I didn't know what to do. Like I'd never been in this situation before where like thousands, like literally tens of thousands of people's lives and livelihood depended on us and what we were doing. And it was a scary, scary situation. And, you know, first I was like, well, maybe I'll make a video and kind of, you know, try to calm the storm and make people feel good. But I was like, you know what? No, like, I'm pissed, they're pissed. Like, this is not a good situation. It's not acceptable for us. It's not acceptable for our, you know, our upstream providers. Like, and so instead, I, in the hotel room, I had my kids go in the bathroom and try to be quiet, and I recorded— it's a pre-Facebook Live too— so I just recorded a video on my phone, um, and I didn't clean it up or polish. I said, look, this is what's happening. I was pissed, and people felt it. And I posted that in our Facebook group, and what was— what came from was amazing. We had hundreds of people come and comment and on our side and basically like, look, we're here for you. Like, and, and they fought hard for us and our guys luckily got things up and within 12 hours we were back stable. And, and what was interesting is because that we learned a bunch of things and, and, and because of that, you know, we were able to move the new server, which was awesome. Things became more stable and then we spent the next 5 or 6 months just working on stability, which if you remember, there wasn't a lot of new features that came out during that time because we're focusing on building this backend structure and making sure that that never happened again. And it didn't for a year and 4 months. And then today it happened again. And so the first thing's like, what in the world? Like, was it our infrastructure? I thought we had built things in a way, and it looked like, as looking at the server resources, like, we have the ability to probably get about 10 times where we're at right now and still be fine. And as of today, so this is a year and a half later, we're at 30,000 members right now. So, you know, 3 times as many people using the system and 3 times as many people's lives are affected when we go down. So we're going down, we're looking, and we're like, like, on— from a server standpoint, like, we were fine. Like, we weren't taxing the servers more than we should be. Like, and so trying to figure that out and dig deeper, and finally found out, like, it was Amazon's— the something to do with them. Like, they were the issue. And, and, um, and so it's like this really weird thing where we were kind of helpless. Like, we can't do anything. They have to fix it on their side. And, you know, their service isn't typically the best. We actually upgraded from the $1,000 a month service plan to $15,000 a month service plan. So now we've got direct lines to all the backend developers. So we were able to upgrade that for future. But like, we were kind of stuck, like we don't know what to do. So it's like, what do you do? And so while we're waiting for them to fix it, our guys are like, hey, well, let's rebuild it, you know, and they start going to work. And it was just so cool to watch them in crisis mode. They didn't just sit there and do nothing. They said, okay, all right, what are our options? How many contingency plans do we have? And they went and executed the first one. And I watched as they tried and failed, and they tried the next one, failed, tried the next one, failed, and they kept going, going. And finally they figured out something to get it live again. And this time we were down for, um, for 4 hours, which is the longest we've been down since, well, since the other one. Um, but they got it up and working again before the Amazon team was able to figure out the issues on their side. And now we're communicating with them and they're fixing the stuff on their side. Um, and, uh, this time it's kind of same thing. Like, what do I do? Do we, you know, I have all this fear. Like, I got all these, we have 40,000 people in our Facebook group. That's the only way, only means of communication I have with our audience. People in Facebook group are like, why don't you guys email us? I'm like, We use ClickFunnels as well. We can't log in and email you. We're in the same problem as you are. We have status.clickfunnels.com page. We've got the Facebook group, and most people in Facebook group, that's the only really channel we had to communicate. And so decided to quickly do Facebook Live and just let people know this was happening and be very aware. And again, not trying to sugarcoat things. In fact, people were messaging me like, are you pissed or are you sick? I can't tell. Like a little bit of both right now, to be completely honest. But did that, and then an hour later we had more updates, did another Facebook Live, walking through the updates, and then an hour later we're back live and kind of shared that as well. And what's interesting is you watch like people that are, the message, like the overwhelming thought was like the fact that people appreciate our communications. It was interesting, I remember about 6 months ago Leadpages went down, I think for about 18 or 19 hours if I remember right. We were watching their server logs and they were down for almost a full day, and they did the opposite, they went radio silence and didn't say anything to anybody, didn't message and didn't talk about it, and just like it was the opposite. And I know a lot of people left LeadPages that day because of that. And I'm sure, unfortunately, we're gonna lose people from ClickFunnels, but I had other people message me in the forum that said, hey, I haven't joined ClickFunnels yet, but after seeing your guys' communication throughout the day, I'm now joining. So I think we'll gain a lot of people as well. But it's one of those things, it's just, it's, It's kind of, you know, for a lot of us, for me, it's my first time being like a real CEO and trying to figure out these things. And I think what I found from this experience is a couple things. First one is overcommunication is key. Like people are okay if there's problems happening as long as they know about them and you validate it. Number 2, I think that me, my initial response when people would complain on Facebook, 'cause I want to be very defensive. Like I had people who were like, they're like, hey, ClickFunnels is down, like I lost hundreds of dollars, do I get a free month, or I want you to pay that money back, and all these things. And like defensive Russell, you know, wrestler Russell, who like wants to fight everybody he sees, is like, you know what, like, dude, like the second you start paying me, right now you're paying $97 a month to use our entire platform, the same thing that in the past would cost me $20,000 to $30,000 a month to run on our own. For $100 you're paying for that, like as soon as you're willing to give me 25% of all the money you make from us, then I'm willing to give you back money when we're down for an hour or two. You know, and like, I just like, in fact, I even started commenting multiple times like that. And then I'm like, no, like, duh, I can't, I can't, like, that's not the right response. And like, I think us as a team, we all want to get defensive because it's like, we're doing our best. It's not even our fault, it's Amazon's fault. Amazon's not refunding my money, I guarantee that. In fact, we're spending now more to be able to actually get answers for them. And it's like, gah, like, it's so frustrating. And like, multiple times I wanted to do that. So I had to message my team and like, everyone on team support side, everyone like, people are upset and they have every right to be. I'm upset, they're upset. Like, we cannot be defensive because that's going to cause— that, that's, that's just the wrong thing. Doesn't help anybody. I said we have to have empathy. Like, empathy is the only thing that, that's right in this situation. And so when people on Facebook complain, instead of me mouthing off and doing what I, by default, I want to say, um, I tried to go into everybody and say, look, that sucks. Like, I'm so sorry. Like, you know, that, that's— I mean, it's not okay. Like, we're not okay with it, and you shouldn't be okay with it, and we apologize, and we're sorry. And try to bring empathy to situations as opposed to where we would naturally go. And so I think that was a big part of it, communication. I think empathy is big. And I think the third thing is understanding a lot of what I'm talking to you guys about through the last year or so and what the new book is about is really when you build a culture of people who, where it's more than just, like, the best way to explain it, in one of our masterminds there's a dude, he's super cool, named Noah, and Noah, one of the things he said, you know, he said, because I bought into you Russell, and I bought into ClickFunnels, and I bought, he's like, I see all the competitors pop out, and he's like, I see them in my Facebook feed, I see the emails, he said, I look at them for less than a second, just long enough to laugh, and then I close it down, he's like, because I bought into you. And I think that, I think the reason why the community is doing okay through something like this and why we're able to recover from something like this is because we're not just a product. We've tried to become more than that. We tried to become a community. We tried to build a family. We've tried to not just give you guys the tool, but the training and the platform in the back, like everything else that ties into that. And it's because of that, like, again, on the same side that I've talked about before, like when we were building ClickFunnels, I didn't want everyone to say, oh, that's Russell's company. I wanted to say this is our company. It works both ways. Like when we have downtime, I don't want people yelling at me, I want people saying, look, we have downtime, this is something we're all struggling with. And we look at it more as a community as opposed to like, us, you know, as opposed to like, not us versus them, but you know, here's a service provider and here's us. And I think that for all of us, like the more you can focus on that, like, You've heard me talk about it a lot in this podcast, but you'll hear me talk about it more when the book comes out, because that's what the whole first section's all about. It's like, how are we building a following of people that are part of something bigger that you're creating, as opposed to someone who's buying your product? And I think that those are the 3 things I kind of learned and were reaffirmed of today as we went through this mini disaster. And so hopefully that helps you guys, because it may not be today, in your business. It might not be tomorrow, it might be a year from now, or maybe it'll never happen, but if and when it does happen, those are some of the tools that have helped us to survive now 2 disasters in the 2 and a half years we've been live, and hopefully if any more come in the future, it'll help us to wave the, you know, survive the storm as well. But I just wanna thank you guys all for being listeners to the podcast. I wanna thank you guys for being members of ClickFunnels. If you're not using ClickFunnels yet, Hopefully this is just one more reason for you guys to come test, to come try it out, because we really do care. We're here for you, and we're not leaving. We're not going anywhere. So if you're not a member yet, go to clickfunnels.com, get your free 2-week trial, test this out, and you'll see what everyone's talking about. This week, we, I said we passed 30,000 active members, which is a huge milestone for us, and our goal is to get 100,000 by the end of this year. And obviously the only way we're gonna do that on our side is to become better. Become better infrastructure, become better people, become better teachers, become better marketers, become better at the fulfillment side, become better at onboarding. And I think that this was a really good chance for me and for my team to look at that, look at what happened, make some really good choices based on that and decisions, and, uh, and build things stronger and better so that it keeps getting better for you guys every single day. You know, the one big difference I'd say between our platform and anything else that I've ever seen in our industry is just because of the success we've had so far, we're able to put a lot of time and energy and money back into the infrastructure to make it better and better and better. So that's why hopefully you feel safe and comfortable building your business and your platform here with ClickFunnels. So that said, I hope that helps you guys, and I hope that in your time of trial as the leader, the entrepreneur, the CEO, whatever it is that your role is, that some of these lessons will help you and give you that foundation you need so that you can wave the storms and that's about it. So that said, someone's ringing my doorbell, I'm gonna go check out who it is. Appreciate you all for listening and we'll talk to you guys soon.
Thanks so much for listening. We hope you enjoyed the behind the scenes look at our funnel hacking project. You can also watch the Funnel Hacker online reality show and watch us build and launch funnels live at www.funnelhacker.tv. If you'd like to come hang out with your fellow funnel hackers, come join our Facebook group at www.projectclickfunnels.com. Oh yeah, and don't forget, you can get a free MP3 player with the first 257 episodes preloaded at marketinginyourcar.com. Join us next time, and remember, you're just one funnel away from your dreams.
More from Adarian
Kaylee Chu’s Bold Experiment That Took Her From Introvert to International Speaker | #Success - Ep. 119
Struggling to Focus While Writing? Adam Leeb’s "Freewrite" is the Solution | #Marketing - Ep. 118
Transmuting Desire Into Drive: Napoleon Hill’s Controversial Claim | #Success - Ep. 117
How I’m Using AI to Outwrite, Outbuild, and Outlaunch My Competition | #Success - Ep. 116
Scaling Online Q&A: Funnels, High Ticket, Ads & More - Part 2 | #Marketing - Ep. 115
Scaling Online Q&A: Funnels, High Ticket, Ads & More - Part 1 | #Marketing - Ep. 114