Joe Sullivan on Navigating Complex Security Challenges

Informing your decisions with intelligence analysis and.

Insight brought to you by the team@oodaloop.com.

Hi, this is Matt Devoe and welcome to the Oodacast. And today we are joined by Joe Sullivan, who's had a very long career in the cybersecurity industry. Joe, thanks for joining the oodacast. And I'd love to hear a bit about your background and your pathway to getting involved in cybersecurity. Sure.

Thanks for having me. Happy to have this conversation with you.

Yeah, I.

So I grew up in Massachusetts in the kind of the 1980s, went to college in the late 80s, and I was focused on. I was always interested in technology, and I have a brother who's a software engineer and another brother who does product work. And so just we always kind of talked about technology at home and we're doing, you know, experimental things whenever we could. When I. But I'd always thought I wanted to work for the government. And so my focus was on trying to do government service. And my intention was to go to law school and then go right into the government. And I got to law school, and I was fortunate enough to get a clerkship, well, a summer internship with the U. S. Department of Justice. And then I applied. And when I got out of law school, I went into the Department of Justice. The only way you get in right out of school was something called their honor law grad program. I was fortunate enough to get into the Department of Justice and was doing a lot of immigration policy and issues like that around migration and border crossing. There was a new immigration law. I went to DC and helped work on implementation of in the late 90s. And it was during that time that I was. I don't know how I became the advocate, but I became the advocate inside the Department of Justice for using technology more, especially when you're dealing with things like immigration, you need to understand different parts of the world and what's going on if you take, for example, political asylum cases. And so back in 1995, I was working in San Francisco and I convinced my office to give me a direct Internet connection, and they wouldn't let me connect, like my Department of Justice computer to it or anything like that. But I just became the person who was doing, like, dragging a little tech into the Department of Justice there. And then I had the good fortune to become a federal prosecutor, and I moved to Las Vegas to be a federal prosecutor. And a couple of weeks in, my manager came into my office and said, you're the only prosecutor who has a computer on his desk. Can you be the, our. We have to have somebody be our high tech crime coordinator for the district. And you'll get a bunch of specialized training and equipment. It's this new program. And I said, yeah, sign me up. And so that's, I guess, 1998 and 1999. I started doing cybercrime case prosecutions in Las Vegas. And it became kind of like my halftime job there. And I just love doing those cases. The complexity of trying to understand the technology and explain, like the role of a prosecutor is often to explain the complicated area to a jury or people who aren't experts. And so I continue to be that kind of bridge between technology that was quickly changing and people who didn't understand it. And I feel like that's been the foundation of my whole career.

Interesting. It reminds me, I think I did the first ever kind of open source intelligence presentation for the FBI back in 96 or 97, and those were the first field agents. They brought in agents from every field office. They were the first ones to be issued laptop computers. And they referred to themselves as the cyber ninjas, but they had no idea how to use them. And in particular, I remember giving this briefing all day and then at the end of the day someone standing up and saying, you know, just remember, you can't use any of these techniques yet. Which struck me as kind of interesting. They had the laptops, had the access to the network, but they weren't allowed to, at least at the time, to engage in kind of the open source intelligence gathering. So you get involved at the Department of Justice from a computer crime perspective. And then what led to you moving into the private sector and take us through your private sector career component.

Sure. So I was living and working in Vegas doing kind of halftime cyber on the rest White collar. And Robert Mueller had been appointed as the U.S. attorney here in Northern California right around that timeframe. And he's the same one who started the CSIP when he was at Maine justice, he had started ccips, which is the computer crime section in Main Justice. So now he's the U.S. attorney overseeing Silicon Valley. And he decided I need a dedicated full time high tech unit in Silicon Valley of federal prosecutors. And so I saw I had moved from San Francisco to Vegas and the idea of moving back was definitely on my mind because if you, if you spend time living in Vegas, you realize you don't want to raise kids there necessarily, at least not back in those days. I think it's developed a lot differently as a community now outside of the Strip. But back then I was just thinking I wanted to get back to Northern California and then. And to see a posting that was just spot on, what I would love to do. I applied for it, got the opportunity to transfer back at the end of 1999 to work in Northern California. And so I started working at the U.S. attorney's office in San Jose, which really was the heart of Silicon Valley back then. And. And my job under Mueller was to go to the companies and say, hey, I'm with the government. I'm here to help. Tell us about all your cybercrime. And, you know, was great. I, you know, they were very reluctant to talk about anything bad back then. You know, there were. There weren't a lot of laws around transparency, and most companies didn't want to celebrate or draw attention to their challenges. But we did on doing a lot of interesting cases, and I built relationships with people in a lot of companies. And then after a while, in a couple years of doing that, companies started trying to pull people like me out of the Department of Justice and into the tech companies. And a good friend of mine, someone who's a big mentor for me, he had been a federal prosecutor. His name's Rob Chestnut. Rob was the Senior VP at eBay in charge of trust and safety. And he said, my job keeps growing. Can you come in and join my team at ebay? And he said something that stuck with me forever and that I share with people who've kind of made that transition from public to private sector. If you find the right role in the private sector, you can feel like you're having as much an impact on safety and security as you did when you were at the government. And the thing Rob said was, as a prosecutor, you deal with the bad thing after it happened. I really need someone like you who could come into ebay and help us design policies ahead of time to prevent the bad thing from happening. And so that was intriguing. I'd never worked inside a corporation or a company and didn't have any background in business, but I thought I'd give it a try. And so I moved into ebay and worked on the trust and safety team and spent a lot of time kind of engaging with law enforcement and thinking about cybercrime from the perspective of ebay customers.

So you're at ebay. I think it's possibly the first time that we met, although briefly back in the ebay days, and then from there, transition over to Facebook. Was that the next iteration?

Yeah, I was in this. I was in this situation where so I, you know, I'd spent eight years with the government as a lawyer, and I went into the private sector and I was doing a little bit of legal work and a lot of operational security work. And at a certain point, I thought, I need to pick one. I need to be either a lawyer or a security person. And I decided I should. I should try being a lawyer. And so I transitioned over to the PayPal side of the company. That was when PayPal was part of the eBay Inc. Umbrella of companies back Skype at the same time too. And, and I got to work on projects for all of those companies. But at a certain point, I said, I'll go over to PayPal. The general counsel for eBay Inc. Needed somebody to be a senior leader on the. On the PayPal legal team, helping their general counsel. And so I went over to PayPal and in about six months, I realized I don't want to be a banking lawyer. And I really enjoyed the security work a lot more. I ended up spending a lot of time, even though I was technically, you know, running the North America legal team for PayPal, I ended up spending a lot of time with Michael Barrett's team. Michael is the CISO at PayPal. Andy Steing and I worked on that. PayPal's responsible disclosure policy that we published back then. We were, I think, one of the first companies to do responsible disclosure. I got to work with Andy on that and just in general, couldn't. Couldn't stop doing the security work. And also the ebay security team kept pulling me back into dealing with a big project which was back then, a large percentage of our problem users, if you will, were in Romania because of the cybercrime economy there. And I spent a lot of time, even while I was at PayPal over in Romania doing training for the law enforcement fighting, meeting with legislators, trying to change their laws, doing actual enforcement actions, fighting with the ISPs that wouldn't cooperate in terms of turning over phishing data, etc. And so I just couldn't get rid of the operational side. Then I got a cool opportunity to go over to Facebook in 2008 and get out of being a banking lawyer officially. And in a very short time at Facebook, the security team ended up reporting to me. And then I was made CSO there and kind of gave up on that debate between legal and operational security and became focused on security full time. And that was. The rest is history.

Yeah. How many years are you at Facebook? And it was during a very, very impactful period of time for the company from a growth perspective.

Yeah. When I joined in 2008, we were smaller than MySpace. People still talk about MySpace as, like, the coolest thing. And by the time I left, we were a big public company with over a billion users and almost 20,000 employees. So got to see it a lot. Took over a very small, really high quality, very, very small security organization at Facebook and kind of built it up into hundreds of people and figured out how to. How to do that while I was doing it. I learned a lot during those years.

Sure. You seem to have a great knack for managing highly technical, kind of eclectic folks as well. To what would you attribute that? Your early interest in technology, your ability to kind of translate between tech and policy. How did you navigate building such a team and keeping them working efficiently?

Yeah, it's. I think, you know, from a manager and leader standpoint, one of the things we all struggle with at the beginning of becoming a leader is we assume that we have to be. We have to be as competent in each area as everyone we manage. And I guess, like, probably the. I have throughout my career been put in these situations where the opposite was true. I'll never forget when I became a federal prosecutor in Las Vegas, I had, like, two days of training, and then my manager pushed me into a meeting. He's like, go into that room. And I walk into a room, and there's three FBI agents who have 20 years of experience. They've got guns on their hips. And, you know, and I've never interacted with an FBI agent in my life. And they're expecting me to tell them what's okay and what's not okay. And that, you know, and they, you know, they, they. They briefly lay out in five minutes, all right, we want to go kick in this door tonight. And we need a search warrant. Or do we need a search warrant? Or can we just go kick in the door and, you know, and can we take their computers and all this stuff? And I'm just like, wait, you're asking me for, like, to oversee you doing this stuff? And, you know, I would just, like, listen, ask a bunch of questions, and run back to my manager and be like, what am I supposed to say? But you just. If you. You realize it's not my job to do their job as the leader, as a person giving them guidance. It's my job to figure out how to help them do their job as well as they can and. And course correct them when they are. Their vision isn't as broad as yours, because as the leader, you Just get to see how all the different pieces fit together. And when you have those strong people who are like in a narrower function, their, their biggest weakness is often their inability to see outside of their lane. And so I, I've tried to focus on more kind of like how do I figure out who are the strong high performing people and then I just try and get in front of them and clear the way for them to succeed rather than, you know, try and micromanage them.

Which also think that many would interpret as kind of loyal to their technical capability. Right. If you're clearing the way for them, most want of work in a kind of friction free, distraction free, point me in the right direction and let me execute. So to the extent that you keep that pathway clear, I'm sure that that instills a lot of loyalty from the technical teams as well.

Yeah, because especially in security, we're going against the grain of a lot of the organization that we're operating within. You know, it's just a reality that security involves asking people to slow down and think about risk and spend some time and money managing those risks. And so, you know, for, for those strong technical people, they got into it because they like the technical challenges and they find that interaction and this is, I'm stereotyping a little bit, but like a lot of those technical people on my teams, they would find that the debate over, you know, whether, you know, whether to implement the technical solution, it's just so draining for them because that's, that's not their passion, that's not their interest. And, and I happen to have a training in how to interact with people and use words to convince them to think about things from our perspective. And that's basically the definition of what you learn in law school. And so if I could bring that experience to bear and help them to be able to get their technical work done, like we complement each other.

Perfect. Before we move on, I did want to ask you obviously worked a lot in kind of these kind of multi jurisdictional international computer crime issues back then. It was certainly really the wild, wild west. There weren't a lot of mutual agreements or compatibility between the different nations. How would you interpret that situation as it exists today? I still know that some, some holes in the net. But what's your perception on, you know, international computer crime cooperation today?

Yeah, it's funny, I think, I think it's a lot better, but it still has a long way to go. Like when I was just going back to when I was at ebay, I spent like one Time I spent almost a month in India traveling around the country, training law enforcement and talking about the Internet and getting. Because, you know, the natural reaction when Internet crime started happening in India was we should hold the. Hold the platforms accountable. And so our country manager for ebay in India, he got arrested because somebody had ever put up a listing on ebay saying, I have a video that I'll sell. Now, he didn't actually put the video on, but the video was troublesome in the eyes of the government. And so they didn't know who put it on or who actually had the video, but they knew that, like, our platform was used to advertise for potential sales. So they arrested our country manager. And so I went and, you know, the rest of our VPs were flying away from India and I flew in to go and talk to the law enforcement and legislators and everybody and say, like, okay, how are you thinking about crime on the Internet? And do you think it is a good thing for your country to go after, you know, the companies that are trying to bring tech platforms to your country? And how do we think about crime? And how do you do international investigations and how do you gather the data? And so back in those ebay years, I think I did training across Eastern Europe, Asia, North America, just try and get people to think about the technology and how to do the investigations. And like, just concepts like MLATs, the Mutual Legal assistance treaties that the governments would have to use to get evidence across borders. So painful. And would take six months to get like one set of IPs. And so you'd have an investigation that would just sit on ice for six months while you waited for those IPs, then you would move on to the next step, the investigation. When you finally got the records and it was just like, so, like, that world is much more effective, efficient, but we still, like, one of the challenges is getting the country where the victim is always has an incentive to help the victim because that's their citizen. But the country where the perpetrator is doesn't have the same incentives. And like everyone, they have limited enforcement budget. And are they going to spend it going after their people to protect citizens of another country? It's a hard sell. Even. Even if it's a, you know, country that's fully in favor of the rule of law.

Is interesting is it almost dictates that the criminals will target people outside their own country just for that exact reason?

Yeah, I mean, a lot of us are watching and, you know, listening from a distance to the conversation right now about the MGM situation. And I keep hearing the speculation and maybe even more concrete than that information that it's that some of the people responsible are in the United States. And you're just like, wow, we haven't seen that in a while. You know, it'll be interesting to see if that proves to be true or not. I have no idea. But like, we just haven't seen that inside the United States at a scale like, or significance of an event like this.

I saw the TLP red presentation on that last week. The evidence seems pretty convincing for U. S. Citizen involvement, so it will be interesting to watch that evolve. Cool. So next, from Facebook, off to Uber as the chief security officer or chief information security officer there.

Yeah, so I was, towards the end of 2014, you know, things were going well at Facebook, had a really great team, and maybe it was getting a little restless for some chaos, But Uber reached out and the first, the first interactions we had with Uber was they actually, they sent over a technical team, their CTO and a bunch of people came over to Facebook because they said, we have a big problem, which is we haven't figured out how to lock down internal access to customer data. Well, and we had put a lot of time and energy on. Into that at Facebook on my team, because that's a fundamental trust issue inside a place like Facebook. You don't want your employees to go spelunking in customer data unless there's a really legit business purpose. And so we. And there's a lot of sensitive data that's very attractive. And so we've done a lot of work to lock it down there. And if you think about Uber at the end of 2014, it was like the biggest security nightmare, house on fire situation you can imagine. They had all these stories about insiders abuse, access to user data that have been in the news. There was a. They went public with this data breach from 2014, but like a while after it had happened, at least in the perception from the outside, you know, there was like, why'd you sit on it so long? And then there was a third thing. There was a really tragic sexual assault situation in India where a driver was accused of sexually assaulting a passenger. That was playing out in global news. And so the company was just like, their board knew and the CEO knew, we need to make a major investment in security. And so they, they pushed hard to recruit me. And they said, we'll, we'll basically give you an unlimited budget and you can go and tackle all these hard problems for us, you know, we think we're going to be a big successful company, but this is one, you know, this is one of the areas where we're, we're falling way short and it seemed like a, a really interesting challenge. So I made the move and I was fortunate. A bunch of people from my team, I think between people who'd worked for me at ebay and Facebook, over 40 people followed me there and, and we were able to pretty quickly build a team from, you know, again, I heard, inherited a security team of like less than five and built it up into hundreds and in a couple of years.

Excellent, I think, testament to your leadership skills that that many people would follow you. It is interesting though, with regards to that model. You know, you're at ebay and users are interacting with each other and obviously there's a transport for, of physical goods. But then you go to Facebook, there's not really a lot of physical interaction, maybe in marketplace, but then you go to Uber and you know, the two parties at Uber are interfacing with each other in the physical world. Right. So it adds a whole new element of, to, you know, how information is protected and how people are put on the platform. So I could see where that would be an interesting and unique challenge because it, it was something that you hadn't encountered before.

Yeah, that level, we had dealt with it at ebay. We acquired a bunch of local classified platforms around the world and we were a minority investor in Craigslist. So we, from a trust and safety standpoint, I had thought a lot about and worked on how do you deal with, you know, classifieds ultimately lead to an in person interaction.

Sure.

So we had thought about it in that context. And then I was an advisor for Airbnb while I was at Facebook. So I was thinking about and helping the Airbnb trust and safety team deal with, you know, their version of that digital platform brings together two people in the physical world. And so I had been thinking about it from the Airbnb perspective and honestly, it was what excited me the most about going to Uber and that the area that I spent the most time on personally was what we call trust and safety, which was focused on the rider and the driver and ensuring that it's a safe experience. And it turns out there's so much you can do with technology to impact that and reduce the risk of those in person situations going badly. And I don't think enough has been written about all the work that the trust and safety team did at Uber and has continued to do after I left in terms of not just analyzing the feedback from prior rides, but like we did things during my time there, like we integrated a selfie mechanism into the app. So like one of the trust and safety issues was, you know, somebody gets cleared as a driver, you know, they do the background check, they've, you know, they have insurance, all those things. And how do we know that that person's actually showing up as the driver the next week with the car when somebody presses a button in order to r. And so we set up a technology that randomly, every so often would ask the driver to take a selfie. Especially if we see like a driver all of a sudden putting in 11 hour days, seven days in a row, you're like, the math doesn't add up. Time to pop a selfie question at them. And then we can compare it to their driver's license that we have on file. And we used some APIs from Microsoft's photo recognition stuff and made it work great. We did a lot of things like that. We did, I think we call it, we looked at the data on rides. You can, you know, every, you know, the driver's phone and the rider's phone are sitting in a car that's moving through space and time. And so you can, we could tell if a driver had their phone mounted or they were holding it in their hand. And believe me, as an Uber rider, there's nothing more disconcerting than sitting in the back seat and having the driver having the phone in their hand. You're just like, it's just scary, right? And so we were able through telematics to figure out and, and tell those drivers, if you don't put it in a, a dock, we're not giving you any more rides. And so like there's so many little things like that you can use technology to make the, the rides safer. You can also, you know, map out like what is the most efficient route and if a driver starts taking them down a dark alley inside instead, then you can like pop up a question to the rider, do you feel safe right now? You know, like, there's so much you can do with technology to impact real world safety. That is just amazing chance for that company to work on.

Yeah. It was unprecedented too, right. With the compute power and all the capabilities on these mobile platforms gave those the opportunity to run applications like that or engage in those type of safety measures.

Exactly.

Well, so while at Uber, there was an incident that occurred that resulted in some legal issues for you. Can you step us through kind of the timeline with regards to what happened.

Yeah, it was a, it was a security incident that happened in the fall of 2016. We thought we resolved it well as a team by the end of that year and, and kind of just, you know, dealt with it and moved on and then kind of re. Reemerged in the fall of 2017. You know, what had transpired in the interim was RC so if the second half of 2016, the tail end of 2016, right around the. And after, during the time of that incident, I was working on a few other crazy things at Uber. As was always the case there. There was never just one crazy thing going on. But at the tail end of 2016 was when Donald Trump was elected president in November of 2016 and our CEO had agreed to be on, I don't even remember what a transition committee. That led to a lot of protests outside of our offices that our CEO was being supportive of the future Trump presidency was not well received and in our driver community and San Francisco proper. And so it was just kind of chaos around that. And then the company had some other major issues around the works, you know, the work environment and treatment of minorities and women. And, and our CEO just kind of kept going through these crazy things that were getting magnified in the media and put a lot of pressure on the company from the outside. And in the spring of 2017, he, he ended up taking a leave of absence and then ultimately I think was forced to step down from the company. The group of us who reported to them then became essentially co CEOs. So we were co CEOs of Uber for the through the summer of 2017, essentially nine of us running a $40 billion company and kind of trying to do it all together and in coordination with the board was definitely an experience I'll never repeat. And it was, it was hard, but it was also rewarding. But in the end we hired a new CEO who hired a new general counsel and the board was, had brought in outside lawyers. And it was kind of like, let's do a massive review of everything that happened in the last coup of years under the old CEO. And at a certain point the company decided we need to do like there should have been a data breach disclosure on that incident the prior year. We thought that technically Joe's team handled it perfectly, but like from a legal standpoint there should have been a disclosure. And then In November of 2017, the company chose to do that disclosure while I was out on vacation and chose to pre brief some reporters and tell them that I had done a cover up. And so I was I was up on vacation with my family in the mountains for Thanksgiving week out, you know, doing some grocery shopping, and I got a phone call saying that I needed to join an urgent meeting. And there was an email in my inbox with a severance contract that I refused to sign. And. And so I was terminated on that day.

So to unpack a little bit around the breach itself and kind of the way that the team handled that. Are you allowed to speak to that? Can you step us through what the situation was? Because I had a couple of threads that I wanted to pull on that possibly.

Yeah. So I still have an appeal pending on my case, so I have to be careful about how I talk about underlying facts. The way I've been able to talk. I have talked about it. I've given a couple of talks at conferences. And in fact, the talk I gave at DEFCON is public and it's on the Internet. I walked through some of the facts of the case, and the way I was able to do that and walk through it was I can share what I saw sitting in the courtroom. So the good news is that there was a lot of transparency through my trial of what actually happened in the underlying investigation. And so it's all in the public record now, in the trial record. And so I could talk about what's in that record. What I probably can't do is talk about, like, what I remember from back in 2016, during the actual incident, and 2016 and 17. So just to kind of, like, keep it safe from, like, a appeal record and what might happen in the future standpoint.

Yeah. So can you give the. The kind of. The highlights of the stuff that was discussed at defcon with regards to what was presented in the court work?

Yeah. So one morning there was an email in my inbox. I think it was in the morning, there was an email in my inbox saying, I found a vulnerability. Or I. I forget the exact words, but. And I.

And I.

It's the kind of email that I would get once a week where somebody says, I found, you know, I found vulnerability in your company. Pay me. And I always would just forward those over to the. The guy who ran the Bug Bounty team underneath the head of Product Security. So I forward. I just bought. You know, I usually would just forward them to him without even, like, saying anything, you know, So I just, like, literally forward it and then I delete it. And so I did that. And then the team had a playbook that they followed for those things. We had, you know, we had all the written formal Data breach response policies and you know, multi layers of those policies and then execution runbooks and things like that. And the team, the team immediately sent a response to the, to the person and said, hey, we typically handle these through our hackeron bug bounty program. You should be doing it through there, but send me like evidence that you actually have access. And they sent some stuff. The team, this is before I got looped back in. The team saw enough to be convinced that they thought there might actually be access. So they spun up an incident response. They looped in the communication and legal people that they, you know, that are dictated that they include as in, in any IR response situation. And then it kind of went from there. The people on the other side wanted to stay anonymous. My team wanted to figure out who they were, but also didn't want to spook them. My team figured out it was guessed that they were probably teenagers, young guys in the United States and Canada. Ultimately figured out it was a 19 and 20 year old one in Florida, one over the border in Canada, interacted with them, sent and then kind of went into full attribution mode. And I wish that this part I could have gone into in detail during my public talk. But like I think sooner or later someone needs to do a public talk on the details of the attribution side of it because it was really, was really well done by my team, you know, because of like the history of working at places like ebay and Facebook and do, and dealing with what we were talking about, about these international cases where we always valued the attribution as part of, of our work. Because if you, if you understand who's on the other side, you understand what, like what the risks are to your organization and you understand their motivation. You also, it helps you in your investigation to, as you dig deeper into your logs, like okay, if you know, if it's a nation state, you're like, I really got to go check in this direction and we got to turn over every rock. And if it's, you know, if it's a straight up bug bounty then you're like, okay, we should make sure we, we don't have lateral movement or stuff like that. But you just have different, the team will respond differently and also who else you'll bring in and how you'll handle it. So the team did attribution and was able to through kind of like some tried and true things that we've done in other cases in the past, essentially have a back and forth until the other side makes a mistake and exposes themselves A little bit. And that's kind of like the cat and mouse game in a lot of these cases that I've seen up close. And in this case, the thing that actually was successful in getting the first piece of data to help pin down the attribution was we sent them an NDA through a document signing platform that we thought would not allow them to obfuscate their ip. And sure enough, it worked. And so even though their origin IP was delivered to us nice and neatly, and then we were able to kind of combine that with some other data and do the attribution. And so then one of the members of my team flew to Florida and Canada and another member of the team sent a note to them and said hey. And we actually sent it to their real email addresses and we just said, hey you guys, thanks for working with us, but like we need you to sign everything in your real names and we also want to meet with you to make sure that you truly did delete the data that, that you had access to. And so my, my team, we had somebody who's a, who's a former CIA employee who is an expert in interrogation. He met, met physically with each of the two people, interviewed them, wrote up whole psychological analysis reports on them, concluded that, you know, when they said they deleted it, it's deleted, there's no risk that our customers data is going to get dumped in the wild. And they signed everything. We, we invited them to come out to California to meet with our team, meet with our company, you know, walk. We wanted them to walk through. A couple of members of my team thought that they were really good at obfuscation of their identity and like their stu. Like we could learn from it. We even referred one of them like from our standpoint, we concluded one of them was like the technical expert and the other was not so much. And for the technical guy, you know, he, we were trying to help him, you know, find a job in cyber security and you know, even referred him to some places, stuff like that. So you know, from. That's when I say that like.

The.

The way the evidence came in, when it got to my sentencing hearing, the, the judge said the end. Like the government and Uber had always presented the NDA as a cover up. And the judge said that's that my sentencing. He said that's not what the evidence showed because the judge understood it was about attribution and, and that, and he's, you know, and he said this is an unprecedented case. Like there's, the data's not out in the Wild nobody got harmed it and yet it shows up on the list of biggest data breach cases ever. But like there's no actual harm, which is very different from pretty much everything else on those lists. So the judge, you know, I was very grateful at my sentencing that he said that this was a really excellent technical job by my team. Even Dara, the Dara Khazrashawi that, you know, the, the subsequent CEO of Uber, when he testified, he said that the right thing to do was to pay the money and, and secure the data and that, you know, like technically everything was done really well by my team, world class. And it took six years to get that recognized. And I'm really, I'm really happy because I'm very proud of that team and how hard they worked and how they handled this case. And they had had this case hanging over them for, you know, the last six years and to have their side and they've all had to go work at other companies and have people, you know, give them side eye looks about this case and you know, I, I would much rather that the world look at them with appreciation for what they did to protect our customers.

Yeah. So at the end of the day and you know, it seems you identify, you do the attribution, you realize that it's, it's not, I don't want to say it's not a real threat, but it's not, not an entity that is kind of truly criminal. They more are on that kind of security research side of the framing maybe a little bit more from an obfuscation perspective, trying to protect themselves, which you know, maybe is, is right or wrong depending on, you know, how they're discovering vulnerabilities and you do the attribution. And I'm sure if you had determined that this was a criminal organization or they had criminal intent, the pathway would have been different. But in this instance you chose to, to mitigate and resolve and to extent it sounds like almost mentor, you know, somebody who famously had to fight those battles with CEOs in the 90s about why are you hiring this person who's a self declared hacker where you're making the judgment case? You know, they were, they were curious, they were looking for areas in which to demonstrate their expertise that maybe were looking for ways to monetize the expertise legitimately through some of these programs, thus the outreach. And you chose not to pursue the criminal realm to kind of treat this, you know, as, it was kind of a bug bounty. So it seems like there's a, there's a side tragedy here with regards to those who want to find this type of behavior and encourage them in the right direction. Vice always treat it as a black and white issue there, there is a little bit of shade of gray here where people are influenceable and you can, you can, you can do right by them by, by framing this in the proper way.

Yeah, that's why I did the DEFCON talk was I've gotten lots of invitations to meet and speak with security leaders about the risks and what, what stand for to that community. And, and I, and I do those conversations. I have a lot of one on one conversations with security executives when they're, when they're starting to panic in their situation, they'll call me and be like, Joe, this is what I'm seeing at my company. What should I, you know, how do.

I get a lawyer?

How do I get a PR person? How do I get myself out of this situation so I don't end up like you? I've had that conversation a million times. Fortunately, most of those people who call me in a panic, their situation diffuses and doesn't end up, end up in a bad place, but sometimes they do. And so like I've had that conversation with large CISO groups, small CISO groups, lots of different contexts. The conversation that I hadn't had that I wanted to have was this angle about the research side of it, because I had. And so that's why I did the DEFCON talk, because I wanted to look at the case through the lens of a security researcher, which is, you know, a lot of the people at defcon, that's their perspective. And that was the perspective, honestly of a bunch of members of my team who, a couple of them testified during the trial that they essentially related to these guys because that was like they weren't too far away from them when they started, you know, their Internet security career as teenagers before they, you know, found the right mentors and stuff like that. And so because it is a, it's a, the, the incentives, we, it's really important for us to get them right. We need to have better cooperation between companies and researchers. We also need to have better cooperation between companies and government. And that's, you know, that's a whole other category that I, I worry that my case. And you know, I didn't do the, you know, I didn't, I didn't think about this is what I said at my sentencing to the judge. I said I, I thought about this, this case during the case. I thought about the perspective of my company, my team, our customers and the researchers. But what I didn't give enough thought to was the perspective of the government and them wanting to be in the loop and them thinking that they have a right to be in the loop. And so that like I would like to better role model partnership between the public and private sectors. I mean the biggest irony for me is that I'd spent my whole career trying to build the dynamic between the public and private sector on cybersecurity. I was on President Obama's cyber commission at the time this incident happened and I was the person who was in Washington D.C. regularly championing. We have to get that like we need more investment in, you know, what's now CISA and, and organizations like that actually helping the private sector keep itself secure and that it be collaborative. And now I'm like the poster child for not cooperating. So that's, you know, that's something I definitely regret.

Yeah. And yeah. Which I guess becomes a disclosure issue but I think it's also kind of makes the whole issue of bug bounties a little complex because there are some bug bounties that by their nature are going to exp. PII or whatnot and has that become a reportable instance every time. Just kind of complicates maybe some of the approaches here. What I also found surprising though, and again we didn't, didn't clear any questions. If you can't answer, feel free. But it seemed like that your role was kind of on an island and when we think about companies and I think at one point I actually read early in the trial the judge made a comment about recognizing that one of the seats at the trial was empty and that seat should be occupied by Uber. So what do you think the lessons learned are with regards to kind of corporate accountability in this issue? You know, vice executive accountability. And were there any, any surprises with regards to kind of how isolating that journey was as the case progressed?

Yeah, it definitely. So I have two different kind of thoughts on this. One is just from be from the perspective of being a person going through was an incredibly difficult six year journey to, you know, to, to, to go from one day, you know, just seeing your yourself labeled as responsible for a cover up on the front page of every newspaper in the world to like six years later having the judge say, you know, it wasn't a cover up and I'm going to just give you probation despite the government arguing for prison. I wouldn't have got through it without like two things. One was having a really supportive family and two Having a really supportive community of people out there, people who understood that these are tough situations, and it's challenging and evolving, and it's hard to get it right every time. And the judge, you know, he joked about at the beginning of sentencing, he's like, I've never gotten so many letters before a sentencing in my career. Don't ever do that again. Lawyers. Because he had to read hundreds of letters. And a lot of those letters were people in the security community, people I'd worked with over the years. And that support really helped me get through this case. And not feeling like I was alone. I mean, I did feel like I was alone because the, you know, the company that had hired me was sitting, you know, they had their lawyers in the courtroom sitting, you know, next to the government lawyers, not next to my lawyers. And. And that side of it was hard, but the. The. I think for everybody else who comes after me. The judge gave two really important gifts in that sentencing hearing. The first gift he gave was he said, said, next time, the next person's going to prison, and that. That you'd be like, why is that a gift to security leaders? It's because what we didn't have was clarity. And what we have, like, what. What we want is our clear lines. And we can say to our companies going forward, like, okay, we don't have a choice. We have to be transparent. And if we don't be transparent, this is the risk. It's not a penalty, a fine, it's jail. And that'll get the attention of your company. And so that's a gift. The second thing that the judge said at the sentencing, and which he had, like you said, he had said before the trial, he said, there's an empty chair next to Joe. That's the company. After the trial, at the sentencing, he said, I don't understand why Joe's here by himself. His view, the judge literally said on the record, the CEO is at least as culpable as is the company. And so to me, that's the ultimate message that we, as security leaders can bring to our executive leadership and the board is the next time, the judge told the prosecutor, next time, you bring the company and the CEO, too. So I think those are, you know, those. Those are. Those could be conveyed as threats, but I don't think of them that way. I think them as gifts because they' they're moving us off of being on the island alone and having the rest of the leadership have to stand with us, which will make. Making them accountable means that they will Become more responsible leaders in these areas.

That's really great insight and it's good to know that A, I would not want anyone to go through the journey that you've gone through for all the reasons that you expressed how tough it was over the six years. But it's certainly have fear for folks who to take those roles as CSO or CISO and feel like they are going to bear the, you know, all of the responsibility for the actions of the company, which shouldn't be the case. I mean, you're, you're acting within and on behalf of and in coordination with other executives in the company. So it was definitely concerning as the case moved forward. And something that was kind of always on my mind was again, that isolation. Why is the, why is the CEO not there? Why is the company not held accountable as a company? Why is it a single executive executive in the decision making structure that is appearing here alone? So it sounds like the judge really took the time in the least to understand the case and the nuances of the case and try and provide as much value to the security community, you know, from this journey as possible. It seems like there's just a ton of lessons learned.

Yeah, I was, from the start, I always felt like the judge in my case was someone who really earnestly, like he really earnestly wanted to understand the issues he want. He's a passionate believer in the jury system and he wanted to get to the bottom of it all. Like, I never once felt like when we showed up in a courtroom, he didn't walk in as prepared as everyone else. He, you know, he did what we would hope a judge will do, which is, you know, try their best to understand it. And I don't, you know, and you think about like a judge, they every day they have like a completely different kind of case in front of them. And for, for, for, you know, he. A lot of people have difficulty understanding the concept of above bounty. Most people, when you explain bug bounty up, think front, kind of react like, I mean, I'm talking like not a gray area bug bounty situation, but like a totally like, yes, companies pay money to people who find vulnerabilities. A lot of people react instantly with like, that's extortion. If they know, if they, you know, if they can see that your car is a flat tire, they should just tell you, you know, not like, oh, they should get paid for it, you know, but we know like, if you're the company, the most efficient way to spend money on, on getting your security better is to pay researchers because you only pay when they find something real.

Yep. But it is again, the realignment of those. Sorry, that's the one of two people who has the ability to override silence on my phone. The, the bug bounty aspect as well changes the incentives. Right. So where somebody might be incentivized to pursue maybe more criminal endeavors or selling vulnerabilities on forums, it gives them an economic incentive to work in cooperation with the company. So I think that realignment of incentives is also important. I know we're pushing up towards an hour. I did want to talk a bit about. You also went over to Cloudflare, so by my count, you played a major role in some of the most important companies. Companies over the course of time. But now we're doing some important work in Ukraine as well. Can you talk to us about kind of the, the current initiative that you have and what you've been doing recently to help out the citizens of Ukraine?

Yeah. So after I was fired from Uber, I wasn't sure what I was going to be able to do next. And I was very fortunate that Cloudflare did give me a chance to go in and be their chief security officer in 2018. And you know, Cloudflare was a small pre IPO company then, but already I thought, having a material impact on kind of the Internet as a whole and Internet security in particular. And so I was very excited to join that company and I worked there for four years. Matthew, the CEO, and Michelle, his co founder, they were very supportive of everything I was going through and were comfortable letting it kind of proceed while I worked there. And I was able to inherit for the third time a very small number of security people that I could count on one hand yet again and build it up into a big team over four years. And that was a very rewarding work, work at Cloudflare because Cloudflare gives away a lot of its products for free to people in need in different contexts around the world, and offers a free product for basically everybody, but intentionally goes out of its way to help in difficult situations. Because that's just the nature of the founders. And they stood by me. Even when I got charged, they didn't. When I got charged with a felony, they, they let me stay on. And that, that was incredible, that level of support. And during 2022, you know, before the full scale invasion of Ukraine by Russia, the United States government, as everyone will recall, was warning, russia is going to invade. Russia's going to invade. And nobody believed it, including the Ukrainians. But the US Government, Cyber Command et Cetera asked. Asked Cloudflare to help out in getting Ukraine ready in case of an attack. And so a bunch of people at cloudflare worked closely with the Ukrainian government and the private sector there to get a lot of it behind Cloudflare. And so I got to work on that, and that was very rewarding work. Then, summer of 2022, I had. I had to take a leave of absence and leave to go do the trial. And, and, you know, we. We essentially agreed that if I won, I would come back. And if, if I didn't win, like, I shouldn't come back because it would just be too like that. I don't think you can actually hold that role and have a felony conviction. I think, you know, I had already seen customers refuse to do business with Cloudflare after I was charged. And, like, I know the names of every one of those companies. They're embedded in my head because I didn't think they should be choosing not to work with Cloudflare, but that, you know, that's their prerogative. But I didn't want to have a more and worse version of that after the trial. So we, you know, we, we reached agreement that if I, if I was going to lose, that I wouldn't come back. And so I just, after the trial and losing, just kind of was in a funk for a few weeks. And then I was very fortunate. A good friend and his wife just kind of took me and my fiance away for a weekend, it was the weekend of Thanksgiving, and just kind of like slapped me in the face and said, you got to get off your butt. I need you to come to my event and speak at it in December. And you got to get up and do stuff and you can't mope. And so after that long weekend with them, I was like, all right, I need to do something. And what was fulfilling? Where can I volunteer? And so I reached out to a few different people and said, I want to do something volunteering in a way that's meaningful. Can you help me out? And I, I'd done a lot of things in diversity and security over the years. I've been, and I've been an exec sponsor for our black employees at Cloudflare and at Uber. And so I was, you know, it was. It was. For me, it was either going to be doing some more volunteering in the. In the diversity context or, or if I could help out in some kind of cyber security way. And so I, One of the people I reached out to was the recruiter who had Placed me both at Uber and at Cloudflare. And so he's a good friend and like, you know, I was like, you owe me, why don't you place me at Uber? And he, I said I would like to do something nonprofit related to Ukraine. And he had his recruiter team, his sourcers contact I think literally every nonprofit in the US that was helping Ukraine. And he called me up a while later and he said, Ukraine friends needs a CEO. And I was like, well a. I was hoping to volunteer a few hours a week. I don't know about stepping in as the CEO. I've never been a CEO of a non profit. I don't know anything about that stuff. But he said, talk to the board, I talked to them and they said, look, we'll help you deal with all the things that we know and you don't know about non profit land. And so I jumped into it and this nonprofit, we do a bunch of different things to help the people in Ukraine survive the war. The biggest initiative that I started that's going really well was I've been getting different companies to donate their used laptops and we bring them over and we give them to kids who are stuck in remote schooling. One of the things I've learned from, I've been to Ukraine twice this year for trips and traveled around the country and met with people all over the country. Half the kids in Ukraine are in remote schooling. They basically were in remote school during the pandemic and then the war started. So they've been in room like, can you imagine kids in remote schooling for like four years straight? And then on top of that, the average income for a family in Ukraine before the war was like $2,000 a year. So they're not like going out and buying laptops and you, and also you can't go online and order a laptop and have it shipped to Ukraine. So they just, there's no access to modern technology. Half 40% of the kids in remote school in Ukraine are borrowing a family member's phone or other device to participate in remote schooling. So they don't even have their own machine. So I've been getting companies to donate machines. We bring them over. I've gotten companies as small as Tines, which is like a tiny little cyber security company. They're like, we have 10 laptops, can you take them? That's 10. And kids lives changed of course, all the way up to some big banks and everything in between have participated. And so that's been really rewarding work. I'm planning my Third trip of the year to go over the week after Thanksgiving. Because my team, when I was in Ukraine in July, they said, joe, you're getting the laptops. And then we're kind of handing off the laptops to other little non profits in Ukraine. How about we launch direct, direct interaction with families ourselves? And I was like, you guys want to take that on? And my team, who's all Ukrainian except for me, they said, yeah, we do. And now. So we basically created a Google form that anyone in Ukraine can fill out and nominate a kid. And I thought we could get swamped, so let's kind of have a minimum bar. And we said, let's, let's just start out trying to get laptops to kids who've lost a parent to the war and who might be in remote school or just struggling and doesn't have a computer. Computer. So we created a form for people to submit. And for the last two and a half months we've been getting applications and we're directly giving laptops out to families. And if it's near where people on my team are, will hand deliver it and get to know the kids. And if they're in one of the kind of too close to the front lines, which there are a lot of kids too close to the front lines, we'll actually use the Ukrainian postal service to deliver them. And shockingly, the postal service operates very well. Well, and laptops show up and function and we do, you know, we'll do video calls with the kids to make sure that they get upset up and things like that. So it's, it's been very rewarding work. And you know, when you go through something hard, all you need to do, like, the best way to reset yourself is go help somebody who's worse off than you, you know, and like the worst. And so during that period between losing the trial and the sentencing and not knowing what was going to happen at the sentencing, sentencing, I just threw my energy into focusing on like these kids, these families, these people in a war zone. They're living through something much worse than I'm living through. And, and they're, and they're showing so much grit, resilience, and you just like, you can't help but like, just get caught up in what they're doing and how you help them and you forget about your own problems.

Yeah. Gives you a sense of appreciation when you're confronted with somebody else's reality that it's so much worse living in a war zone, remote school for four years, loss of a parent. I mean is obviously all tragic for those kids. So thank you for that work that you're doing. Very important. They'll include a link to the nonprofit, you know, when we release this episode of the podcast as well. I do appreciate your time here today. I always ask one question at the close, given that I'm such an avid reader, and that is is there a book that you've read or listened to recently that really stuck with you that you might recommend to folks watching this podcast?

I've been, that's a good question. I've been doing a lot of reading, trying to understand history in Eastern Europe. And, and I like, I'm blanking on the name of the book. I just read this book that one of my daughters recommended, but it was about, it was, it was about, it was from the perspective, perspective of an Italian young man growing up during World War II and kind of the role he played blanket on the name right.

Now, but we can put it in the show notes. That's the beauty. The Internet.

Yeah, I can send it to you. But like the, it was I, I'm reading a bunch of non fiction books about like the experience. Like there's an excellent book that I'm reading about Russia at the time when the SARS kind of got pushed out, you know, at the tail end of World War I and how, how the evolution to, you know, the communist government happened. And then kind of like looking at like the, how the Nazi situation in Italy was happening, I just find like this historical context makes me look at what's happening in a place like Ukraine in a much more nuanced way because we, we always need to like think about situations not just from the perspective of, of the person we relate to the most, but like, I mean, I think about the Ukraine war as a tragedy on all sides. It's like the, you know, the Russian soldiers who are, you know, forced to march forward, if they turn around, they'll get shot and if they go forward, they'll get shot. And they, you know, they said they signed up for something different. Like I think that's a tragedy. I think what the kids in Ukraine are going through is a tragedy. There's so many. And so having some historical context but also thinking about these things from all the different perspectives is why I like to read and help me get outside of my own head.

Yeah. Interesting. Yeah. And there's a lot of, you know, I don't know how big a believer I am in kind of the cycles of history, but you know, to this date we certainly have seen some of those cycles. So it's good to understand where the cycles occurred in the past, kind of what the global context was, what were some of the catalysts and what were some of the outcomes so that we can avoid some of the negative cycle elements in the future. Joe, I really appreciate you taking the time to chat today and for being so candid with your lessons learned and your experience. Really appreciate it. We'll. I'll get those book titles from you and we'll include them in the show notes as well.

Okay, awesome. Yeah. And, and thanks for sharing the info about Ukraine friends, because if any listeners, you know, feel inspired to get their company to donate some laptops, they're, you know, odds are good they're just sitting in a pile somewhere and they've already been wiped. And if they're. They haven't been in, get them wiped and then let's put them to use.

Absolutely perfect. All right, thank you.

All right, thanks. Thanks for listening to this OODA Loop production. For the latest analysis on cybersecurity technology and global risks, please visit www.oodaloop.com.