Personal Note from Mike: Fighting Fires
Mike Andes describes Copilot's January 2024 outages caused by sudden load and vendor problems, and the shift from quick fixes to root-cause fixes. He fired 3 developers, hired 4 senior developers and a new head of operations, and plans to resolve issues by February.
Questions this episode answers
January 31, 2024.
Three and a half days.
Three developers were fired.
Four senior-level developers were hired.
All right, ladies and gentlemen, we have not had personal notes from Mike for quite some time. Had a little time away, did a bunch of turnarounds, been really crazy. It's raining outside, you're probably hearing the windshield wiper. So let's give a little bit of an update. Things I've been learning, things I've been doing. Alright, so went to Mexico for three and a half days with my family. It was really nice. I was able to get them a entire weekend away. For all my direct family and their spouses and kids, it was a great time away.
It was probably during the worst of the worst when it came to what was going on with the issues with copilot. And I was very thankful for the family being very forgiving of the fact that for the vast majority of the time I was working. And if you're like, that's horrible, well.
You probably not a member of Copilot.
Because you would have wanted me to be in the fire at that moment and making sure that things were being done right. So that was the focus throughout the vacation. However, I did enjoy my time away and I enjoy working, to be honest. And so just having a different scene scenery is actually really enjoyable for me. So whether I'm working in my office or at a standing desk or on a beach in Mexico, it's still a good life. Now, what are we going through right now with Copilot? What's the update?
Well, we're still in the middle of hell.
However, we have assessed the fire and we have a plan to actually get out of this. So I do feel like we are on the upswing. And the thing that I've been learning and trying to like focus on over the past few weeks is as stuff broke in that first week of January and now we're at the end of January, it was easy to see like, okay, this is what's broken, let's jump in and fix it. And that's my natural inclination, is to take action extremely quickly. And I think I would have made people happier in the moment if I did that. However, as things started to break and we started realizing that it was because of the load of all the new users and all of these integrations breaking and just a whole bunch of things happen at the same time. I really wanted to to step back and make sure we knew the root cause of these things, like why stuff was happening and then have a plan in place to actually fix them for the long term. And so that is not my natural mode of operation is sitting back, taking calculated approaches Taking notes, having strategies and meetings. I do not enjoy that. However, this was needed because if we're going to 10x copilot, which we are going to in the long term, we can't have this happen again. And so rushing in to fix the fires as they were happening in the first couple weeks of January would have led to a lot of duct tape solutions that in this moment end of January, would have made people happier. But I know that we would be kicking the can down the court, down the field for another six or 12 months when the same inevitable end was to happen. But yet we'd have more people inside the systems with more dependencies on our current infrastructure. And so over the past few weeks, we've created an IRC internal review committee for every single bug. So when it comes in, it's not being reviewed best by our internal team on the product side, but actual engineers are looking at every single bug. So that way if there's commonalities, we're able to attack them from a much more systemic point of view and go after the root cause of the issue instead of, oh, there's this one drop down that's not working, let's fix it. Yes, a junior level developer could jump in and fix it, but then it takes usually a senior level developer and some people thinking and strategizing to discover why did that drop down stop working and actually get to the root cause of the issue. And so things like the irc, we have switched over how we are handling our entire project management internal systems. And these are like the little things that do absolutely nothing for our users in the short term. And that's why it's so frustrating and hard sometimes for me that wants to see immediate results and we've always been known for dealing with the bugs very quickly. And I'm like, we want to uphold that reputation. However, at the same time I want to make sure that this is an issue, these issues we solve for the long term. And so I do feel like we have the plan in place now. Jumping into the fire so yesterday was actually the first time at the end of the day we had like a list of things that were actually now working and fixed, including things like one click estimate, emails, webhooks, push notifications on iOS. These are the type of things that all depend on our other vendors and when put under stress, all broke in the first couple weeks of January. We still have a long ways to go. We've just jumped into the fire. But I do feel like the thing that I've learned.
Unfortunately over the past five weeks is there are times that jumping into the fire is required. There are other times when sitting back and actually strategizing about where the fire is coming from and then attacking it correctly is worth the collateral damage of letting the house burn for a little bit longer. And so I guess in the long term, only time will tell whether or not I'm 100% correct on this. But based upon even the past 48 hours, I do believe that having the wherewithal the stand backs debt back. Create a planning strategy. We fired three developers, hired on four much stronger senior leaders level developers. We hired a whole team of developers that worked together in the past. So there's a lot of congruencies and otherwise they're very young team. At copilot of not working together, we have like a whole collective group that's worked together and we hired all of them. It's kind of like a talent acquisition. That's fantastic. They used to be a startup working together. Bring all those heads together and be like, let's work on this copilot thing together. Because we have such a young team, we've grown so quickly and so at the same time, we've had project management changes, we've had changes like the IRC.
Internal review committee for every single bug. We've also had to really create better systems and structures for our internal team. And that is really defining the product.
Versus engineering departments and defining the customer.
Support versus customer success departments. All very different departments of the business.
But when you're getting started, you wear.
A lot of hats.
However, wearing a lot of hats leads to things falling through the cracks when things scale very quickly. And that's the same thing, like the same way that our infrastructure and code had issues. The first couple weeks of January, our.
Internal team was not ready to handle three to four times the amount of.
Traffic overnight like we we had.
And so we now have head of operations Lindsey.
She used to be my manager.
She's worked with me for a couple.
Years at the gym.
But from an operational perspective, we'll be.
Able to handle a lot and making.
Sure people are onboard correctly. They have developer handbooks, they have everything ready to roll. They've already seen certain sequences inside the code to be able to recognize patterns.
And just taking a lot off my.
Plate and then being able to make sure that individual contributors inside the team.
Have a role defined much more granularly instead of broadly.
These are all like the fine little brockling, broccoli blocking and tackling. That is not fun. I hate every single moment of this sort of operational stuff inside businesses. I love marketing, I love sales, I.
Love pushing the brand forward.
That's what I, I love this whole last five weeks is shows just how important operations is though, and that any operator that can sell and market, they will eventually hit a cap because they operationally will not be able to have the skills and acquired to actually build the infrastructure to support their vision. And the same thing happened at Command center the past couple years. It's like, I don't enjoy the fact.
That like, okay, we have a good.
Thing going, but we need to literally duplicate this a hundred times. Like we need to hire 100 people.
And so I don't enjoy that.
But that is what is required for the business to actually do well and to be able to serve customers and users well. And so at Command center, they have like 74 people now, like going into the spring rush, very confident they've overhiered and they're gonna be able to crush it. The same thing is true with Copilot. It's like these are operational things that are in the weeds, in the mud. It's blocking and tackling. It's extremely important. And so if you're listening, the thing I would share is if you're more the visionary sales marketing driven, you must find someone that is operationally driven that will put the brakes on you occasionally. And obviously you can veto things. You are the owner, you are the founder. And I would say that typically I default towards taking action and moving quickly. But there are times you need to step back and take more calculated approach to a problem. And by doing that, you will retain your team better, you'll be able to serve and really put your operational people in a better spot by creating those systems. So that's what I've been learning. Next week is gonna be really interesting. I'm going to be meeting up with a whole bunch of tech founders next week from a whole variety of different industries. But there's several CRM founders there of companies that are very, very large. This is part of the mastermind group that I've been part of. And it's a group of 50 people. And I am probably one of the smaller business, like from Copilot's perspective, probably one of the smaller businesses in terms of that one entity in the room. And so that's interesting.
That's fun.
It's very, very rare I'm able to do that. And yes, I have to pay a lot of money to get inside that room. However, it is one of those things that is harder and harder to find those rooms where I feel like I am the novice and I actually feel like I'm decent at when I'm in those rooms shutting up and listening and taking notes, I feel like I'm actually pretty decent at that. And if someone asks me for something, I can definitely help. And being part of this group, they actually require you to tell what you're able to contribute to the group. And I said, look, I know for a fact I probably have more social media friends than everyone in this room combined. I can help people with that. I can drive, I can show how traffic works and social media and driving leads. I need help on these other things like engineering and how I as a non technical co founder deal with some of these problems and structures and having engineering leadership departments and that's what we've developed over the past few weeks too is like a whole engineering department leadership team with an engineering manager and a principal developer and a.
What was the other one?
A systems engineer. Like these are the type of things that are not fun or exciting but is required to serve the users the best. And so we can talk about vision in the tour of blue in the face, we can talk about what we're going to do in the future until we're blue in the face. But if we're operationally we're not sound, we're going to lose customers and we're going to let people down. So that's what I've been focused on the past four or five weeks. Next week I look forward to talking to some of these founders and these other business owners that have done, you know, built these nine figure businesses and have tons of experience in the exact issues that I am currently facing. And we look forward to seeing if I can learn more. And I do hope and believe and do trust that over February a lot of these things will be behind us. And I do hope, I do hope that one day I can look back and chuckle at what I'm going through the past couple weeks, even right now, it's not like we're done out of the fire. We have a game plan to fix the problems and I feel like I'm very confident in that game plan and that strategy. And so now it's time to execute. And so when I come back in a few days or in a week or two, hopefully I can have an update that the strategy is continually being carried out and is being executed flawlessly.
That is the goal.
Will it happen flawlessly? Likely not. Have you ever stepped into a burning house and everything went perfectly? Probably not. But at least we're now stepping into the fire fixing problems, and we actually have a plan in place to make sure that these things don't happen again. So fingers crossed.
Hope and pray depend on that strategy.
And that plan, and we'll see where things go.
More from Mike


I Sent My Head of Sales to Hormozi's First-Ever Offers Workshop | C.Suite Unfiltered 117