Emergency Outage Plan | DO THIS NOW (Updated)

Hi. So this video was originally published in March of 2022, but I am now reissuing an updated version because since the original release I've made some changes to the emergency outage dispatch report. And those changes meant that I needed to update some things about the end of this video. If you're curious about what exactly changed about the report and why those changes were made, I made a separate video about that which you can get to from a link right around here. Alright, enjoy the video. Past me. Take it away. In a perfect world, videos like this one would not exist. But this is not a perfect world. What's going on everybody? It's Richard Kohberger here, the Blue Clown nerd. And in this video we're going to be talking about building out procedures to prepare for various outage scenarios. Because while we can't predict exactly what's going to go out and when, what we do know is that eventually some something will go wrong. So it only makes sense to have a plan for when that time inevitably comes. So first, let's start out with a power outage. Now of course, the best thing that you can do to protect yourself from a power outage is to equip your office with a generator. Having a properly sized backup generator at your shop can make a power outage of little to no consequence to you in a lot of cases. But they're also not exactly a cheap solution. They are an investment. So if that's not an investment that you're ready to make, then there is something that you can do in the meantime, and that is to make sure that all of your absolutely mission critical devices are plugged into a UPS or an uninterruptible power supply. These are basically surge protectors mixed with big batteries that keep themselves topped up at all times. That way if the power goes out, the UPS can just automatically switch to powering your devices with the battery reserve instead of with the power coming from the outlet. Now, there are several downsides to this versus a generator. Don't get me wrong. First of all, you have to make sure that your UPS is properly sized for the wattage of all the devices that you're planning to plug into it. You'll also need more than one UPS unit in most cases. For example, a computer will typically need its own dedicated ups. And then maybe you want a separate UPS to plug in all of your networking equipment like your router and your modem. And of course the biggest downside of a UPS versus a generator is that with UPS, you are working with a Finite amount of time that battery will eventually. So if the power outage lasts for a while, then having UPS's set up isn't going to make that a non issue for you. It's just going to buy you some time. How much time of course, depends on the size of the UPS and the amount of power that the devices you're plugging into it use. Generally, smaller, cheaper UPS devices can get you a few minutes. Basically just enough time to like save whatever you were working on and properly shut down the computer. And larger upsets can get you up to a few hours. The benefit of course, is that this is way cheaper than installing a generator. You can get a pretty nice reasonably sized ups for around 150 bucks. Now, a way that you can stretch the amount of time that you're able to get out of a UPS is to combine it with devices that are already battery powered, such as laptops and tablets. For example, if you're running on a laptop that is plugged into a ups, then not only do you have the battery capacity of the UPS itself to work with, but also the battery capacity of the laptop. Laptop. So now you're easily getting a full day out of it, probably even more. Especially if you're also connecting to a battery powered hotspot, which we'll talk about in the next section. Okay, next let's talk about an Internet outage. So the bare minimum thing that anyone can do to make sure that they're at least somewhat covered in the event of an Internet outage is to make sure that their mobile plan supports the mobile hotspot function. Most phones today are capable of being mobile hotspots, which means that you connect to that as your Internet access point and connect to the Internet over the mobile network versus over your regular Internet provider. But not every mobile plan from the carrier allows for that, so make sure that yours does. Now that can get you by for a little while, but keep in mind there are some limitations to that method. First of all, you can usually only connect three to five devices to a phone acting as a mobile hotspot. So if you have more than that in your office, then it gets complicated and you have to start having multiple. Your Internet in that situation is also highly affected by how good of a signal the phone itself is getting. And phones acting as mobile hotspots usually have some sort of throttling throughput rate. So you're only going to get so much speed out of it. Now the step above that is to purchase a dedicated mobile hotspot. For example, this One from Netgear called the Nighthawk M1. I am not endorsing this particular product. I have never used this product. I'm just giving you an example. So with something like this, you can connect up to 20 devices versus just three to five on the phone. You'll get higher throughput speeds than on a phone. And because this thing has multiple antennas and is a dedicated hotspot, it will generally get better reception than just a phone. Keep in mind that if you do purchase one of these, you also have to get a mobile plan to go along with it. So you'd have to contact your carrier and get a data plan for it. All right, next, let's talk about phone outages. And this is where things start to get a little bit more complex. So first of all, as of me making this video, there was a fairly recent, a few weeks ago, bandwidth outage. I won't go too deep into detail trying to explain what that means. I did make a dedicated video about that. I'll put a link in the corner of the screen here. But the short version is that Bandwidth is one of very few companies where phone calls kind of originate. So a phone call goes there before it goes to Verizon or Dialpad or Ringcentral or wherever. So they're kind of like the provider for the providers. And they experienced something called a DDoS attack, which brought them down. In that very specific, very rare scenario, there's nothing that you could have done that would have kept your phones up. If the phone numbers originated in bandwidth in any way, shape or form, that's it. They couldn't make it downstream at all. Even if you had old school LAN lines in your office, those still would not have worked. Because even phone calls through LAN lines go through companies like Bandwidth. Now, it's not like back when an operator had a switchboard and was literally connecting one phone to another. All phone calls go through some sort of phone network now. And Bandwidth is one of those networks. All that to say, even with plans in place, things can still go wrong that there plan can't accommodate for. But of course it's still important to have a plan. You can still die in a car accident if you wear your seatbelt, but you should still wear your seatbelt. So first we'll go over our main fallback plan for phones and then we'll talk about some things that we can do in the very rare and unlikely event that something like that happens where there's just nothing you can do. The phones aren't going to work. So first of all emergency fallback failover numbers. If you're a service titan user not on Phones Pro, then you have this feature called emergency fallback that's found under Settings and Phones. There you can input up to 10 numbers that you want to use as your emergency fallback. You can adjust the ring duration for how long you want the call to ring before it assumes that no agents are available. And you can upload a voicemail message. And then at the top, there's a simple toggle switch that you basically toggle to turn it on or off to say like, hey, right now we're in emergency mode. Turn it on and use all of these emergency fallback numbers. And then once things are back to normal, you turn it back off. So if you are using this, a few things to keep in mind. One is that whatever phones probably like CSR cell phones that you're putting in as those emergency fallback numbers, when you actually toggle this on to go into emergency mode, you should have those CSRs temporarily disable their voicemails on those mobile phones. And that is because this system doesn't have any way of understanding whether a person answered the phone or if a voicemail answered the phone. So we have it set up to where we have our own company voicemail uploaded into here, but it's only going to go to that after the call hasn't been answered. And if the call goes to the CSR's personal voicemail, as far as that system is concerned, the call was answered. It doesn't know that it was a voicemail. It it assumes it was a person who answered the call. And for that reason, those personal voicemails should be disabled while they're actively being used as emergency fallback numbers. And once the emergency has passed and you turn this back off, then they can put their voicemails back. Another thing to know is, as of the time of recording this video, this is a simultaneous ring, meaning that, okay, so you can put up to 10 numbers in here. If you do put in 10 numbers or three numbers or five numbers, whatever it is, it's going to ring all of them at the same exact time. That can cause a little bit of confusion. Let's say two CSRs pick up at basically the same time. One of them will get the call, one of them just won't, and they won't really know what happened. So just make sure that everybody's aware that is how it works. There is a knowledge base article on this emergency fallback feature, as well as a frequently Asked questions document and I will link that in the description down below. Okay. Now, if you're on Phones Pro, then within your call flow, you have failover numbers. So. So that's something you set up in the dial pad settings that tells Phones Pro when no agents are available, where do the calls go? Now, it is planned to have this same kind of simple toggle switch fallback setup for Phones Pro as well. So that way if you have special emergency numbers that you want to go to, you don't have to mess with your normal call flow. Like for example, maybe your normal failover number is your answering service, but in this emergency you want it to be your CSR cell phone, which, well, instead of having to go and dig through the dial pad settings to change that, you could just toggle a switch that's not out yet at the time of recording this video. But I just wanted to let you know that it's being worked on for the future. Now, very important piece here and the piece that I think a lot of people miss. You should drill this. You should test it and make sure that it works. Put it on the calendar, do it at least quarterly. Monthly would be even better. Because what can happen is you set this stuff up one time and then you don't need it for a long time. And then when the time comes, you flip the switch for the emergency fallback and it's ringing. A deactivated cell phone for somebody who hasn't worked here in three years or something wasn't set up properly in the first place, or just whatever, just make sure that this stuff works on a regular basis. So if you're using the emergency fallback just toggle switch method, just toggle that switch every once in a while and make sure that everything's going as expected. If you're on Phones Pro, then have all of the agents go off duty and make sure that the calls go to the proper failover. Or if you're watching this in the future, when the toggle switch is available, then you can toggle that switch too. But the point is, test your emergency plan, drill it frequently, have it on the calendar, take it seriously. That goes for everything, not just the phones. Okay, so there's that. Now let's say we are in that really rare situation that we happened to be in a few weeks ago as of recording this video where something way upstream is the problem. And no matter what we do here, it's not going to put our calls through. Well, in that case, we need to have a plan in place for how we are going to communicate with our customers through other means to let them know about this problem. Now if you have Marketing Pro, then you know, have the ability to send out a marketing email immediately versus having to schedule it out like you used to have to do. So if you have Marketing Pro, then that would be one way to do it. Have a template ready to go. Have an audience of your entire customer base and have it ready to go at the push of a button. Now you don't exactly know what the situation is going to be. So when I say have a template ready, I don't mean just like blindly send the template. I just mean have something in place and then kind of modify it a little bit when you actually know what the exact situation is and then send that out. If you work with a chat service, like an instant message chat on your website, make sure that they are informed about the situation and that way they can communicate to any customers that are chatting in and let them know. Yes, yes, that's a known issue. Communicate with whoever it is that handles your website and make sure that they have some sort of graphic, some sort of like banner ready to go that they can throw up on your website to say, hey, we're having phone issues. Here are alternative methods that you can use to contact us. Now again, if this is a super upstream outage, then it's possible that it's also affecting cell phones. So it's helpful to have some sort of other Internet based communication method like something like Slack or Teams that you can communicate with your field staff to say, hey guys, phones aren't working right now so be watching here on Slack or Teams for communications with dispatch and everything like that. Have a plan in place for who's going to watch for abandoned calls slash unanswered call bubbles. Are you going to add those to some sort of spreadsheet to call back later? Again, that depends on the type of outage. If it's a super high level, like a bandwidth outage, the calls might not even make it to the abandoned call list. But again, that's super rare. It's worth having it in your plan to have somebody check those abandoned calls. Okay, last but not least, we're not going to skip it, don't worry. A service titan outage, I mean it happens. Like I'm not going to pretend like it doesn't. I mean as of recording this video, I think last week, or maybe the week before, Facebook was down for an entire day, Facebook outages happened to everybody. Now statistically on paper, service Titan has very high uptime. It's industry leading in that area. It's also been improving over time. But I get it. I know that sometimes it can not feel that way because it's mission critical software and when it is down you really feel it. So don't get me wrong. I'm not trying to excuse the downtime that does exist by saying that the uptime is high. It's mission critical software. Every second that it's not working is very stressful and expensive to you. I know that, the team knows that and I know that there is constant work being done to make sure that the uptime is even better than what it is today. That said, stuff does happen and we have to be realistic. So the number one best thing that you can do to prepare for a service titan outage is to schedule yourself a daily upcoming jobs report. An upcoming jobs report is just that. It's a report that shows you what the upcoming jobs are. That way, if you can't access your dispatch board, you still know where to send your texts. Wait a minute. Wait, wait. Hi. It's been like four months since I recorded everything you just saw. You see, I started walking through the exercise of scheduling an upcoming jobs report myself so that I could ensure that I was telling you to do it properly. And while I was doing so, I realized a couple of things. Number one, I realized that we've been giving this advice to schedule an upcoming job support in case of outage scenarios for a long time, since long before ServiceTitan had appointments. So if we don't have access to ServiceTitan and we need a list of our upcoming jobs so we know where to send our technicians, the upcoming jobs aren't really what we care about anymore. It's the upcoming appointments. We have to account for multi day jobs now, but that's not a problem, right? We can just use appointment specific data columns within the report. So we had this set of data points available. Next appointment Start date Next appointment Start time Next appointment Technicians and Next appointment Confirmed but the problem that I ran into was with how service titan reporting was defining next appointment. In this particular use case, it was defining the next appointment as the next appointment whose start time has not yet passed. So the next appointment scheduled for the future from the point when you hit run report. The problem with that being that a technician is not always going to dispatch to an appointment before the appointment start time. Sometimes they're going to run a little behind. So for example, if you had an appointment at 2:00pm and at 2:01pm, the technician hadn't yet dispatched to that job and you lost access to Servicetitan. If your scheduled report ran at 2:01, then that 2:00 clock appointment is not going to show up on that report because that start time has passed. It doesn't matter that the job hasn't actually been run yet. I didn't like that at all. So I reached out to the reporting people. I was like, yo, reporting people. And they were like, yo, bcn, what's up? And I was like, ah, we got a problem. And they were like, ah, dang. Okay, what's the problem? And I was like, uh, it's hard to explain. And they were like, well, that doesn't help at all. And I was like, I see where you're coming from. Let's schedule a meeting. So we scheduled a meeting. I walked them through the use case. I explained why these data sets do didn't work for this particular use case and they said, okay, we're on it. So they came back to me with these new data columns that are now available in the jobs data set. Upcoming Appointment Start date, Upcoming Appointment Start time, Upcoming Appointment Technicians and Upcoming Appointment Confirmed. So these aren't replacing the old columns. They are there in addition to those old columns called Next Appointment Blankety blank. But these new data columns, these upcoming appointment data columns, they just look chronologically at the next appointment that hasn't been run yet. So in that exact same scenario, if a technician has a job scheduled at 2:00pm and you lose access to ServiceTitan at 2:01 and the technician hadn't dispatched to that job yet, if you're using these data columns, then it's still going to show up on the report because the job hasn't been run yet. And since I was already this far in the weeds, I figured, what the heck, let me just go ahead and make it. Make this report so that everybody can use it without having to build it themselves. And so, making its public debut, I present to you the emergency outage dispatch report. So this report is live in your account right now. All you need to do is schedule it. If you already have some version of the upcoming jobs report scheduled, I highly recommend replacing it with mine just because of the reasons that I just outlined. Instructions for how to schedule this report are in the description of the report, but I'll also go over them here. So you're going to hit schedule and then you're going to choose flexible report. Then you're going to set the filter by field to jobs with Next Appointment Start date and you're going to set the date range field to today. All of the other fields here are up to your discretion. Then once it's time to set the scheduled times for it to run, I recommend running it at least daily but no more than hourly. So of course the more often you run it, the more up to date information you'll have. Just don't go too crazy. It doesn't behoove anyone to set fire to Service Titan. And when you get down to the filter step, I recommend making no changes and simply hitting schedule whatever email or emails you are sending this out to. I also recommend setting up a filter in that email client to take all of these report emails and just put them in a folder somewhere so that they're there if you need them. But they're not constantly just cluttering up your inbox. I'm not going to break down the details of how to do that here. This video is long enough, it's different for every email client and you've got Google. But anyways, that's all I've got for today. Hey, I think this video is super important. If you know some fellow titans who might benefit from this information, please share this with them. Be sure to hit like if you like this content and found it valuable, be sure to subscribe to Service Titan's YouTube channel if you haven't done that already and hit that bell icon so that YouTube notifies you. Anytime we upload a new video video and leave me a comment down below and let me know what you think I should make a video on next. Please remember that your engagement through likes, comments and subscriber numbers are the ways in which my success is measured. Appreciate it. Peace it.