Episode 49: Kathy and Randy Pherson, Authors of Critical Thinking for Strategic Intelligence

Informing your decisions with intelligence analysis and insight brought to you by the team@ooda loop.com.

Hello and welcome to the Oodacast. Today on the Oodacast, Randy Fearson and Kathy Fearson. Hello, Randy. Hey, Kathy.

Hey, Bob.

Hi there. Good to see you.

Yeah, good seeing you guys. You know, I'm really excited about this oodacast for a couple reasons. One is I've been in the intelligence community and around it for my whole life and I meet so many interesting people like you guys and I feel like here's an opportunity to share some very interesting people with my current network and community and I just can't wait to introduce you guys to others. So thanks.

It's a great way to end up after a wonderful Thanksgiving.

And so the briefest, let me just give my understanding of your bios. Both of you had careers in the intelligence community. Kathy, you were in the CIA. You did analysis and also resource management and were in for a career of over 22, 23 years, if I remember correct. 27, 27 years. Randy, you were also in the intelligence community, a deep background in CIA, a lot of Latin American studies, but then also at the National Intelligence Council as a national Intelligence officer and a highly regarded position where you help coordinate, collaborate and think on issues under your portfolio. Did I get that about right, Randy?

Yeah, you got it right. I worked mostly, I worked in Africa, Europe and Latin America. Also did work for the IG and was the deputy ex dir doing strategic planning for the agency.

And then I've known Kathy for a while through many of these groups that we interact on, like the AFC Intelligence Committee. And one thing I know about Kathy is she's a successful businesswoman who has created lots of value for others, which is what business is all about. And you too, Randy, a successful business leader. And, and I admire the way that both of you have helped create value for businesses and government organizations by helping think better, teaching critical thinking and best practices and writing about that. And I'd love to get more into that in this discussion.

Sure. If I could just raise one thing at the beginning, though, what not everyone knows is that Randy and I started our careers. He actually came in as a graduate fellow a year before I did. The second year when we were coming back, we were engaged to be married. We showed up at CIA that summer in June and we got married in August. Everybody thought that we actually had met in June and got married in August, which wasn't quite the truth. But the interesting thing about it is when you consider that we spent almost 30 years there. We started at just about the same grade and we ended up at the same senior service level grade. And to our knowledge, we're probably the only couple that actually did not meet at CIA to be able to do that. So I think that says a lot about the organization and the environment that we were allowed to grow and develop in.

That's cool. And that kind of leads into one of the early questions I wanted to ask you guys was early on, did you guys have mentors that helped, that you looked up to? Who were they?

I found that that was the one thing that was missing through almost all of my career. I came in actually on the ops side and I moved over to the intel side and actually then moved up onto to several other parts of the. I worked in four or five different directorates in the agency. And wherever I was working, people thought that I was one of them, which I thought was pretty helpful, but also meant that I didn't have an established network. So I got through with one exception. We had a division chief that gave us some advice when I was trying to decide to take a job or not take a job. But I pretty much got through the entire system without a mentor.

We've had some really good examples of people that we should follow. So oddly enough, as a married couple, we were hired into the same division of the Office of Current Intelligence back in the dark ages. And our division chief was a gentleman named Joe Ziering, who used to sit at his manual typewriter and type out perfectly reasoned, logical pieces about European integration.

We didn't need any white tape, and.

That was the example that he set for us. We were expected to be able to draft on the typewriter and to think through the piece at the beginning. And that was Joe's example. Carla Scopolitis, who was his deputy and may she rest in peace, also who worked for us when we first started our company. She was teaching writing for us for several years. So we've had some really great mentors that you'll find. We started in a division with John McLaughlin and Winston Wiley were our colleagues who later had very successful CIA careers. So we really learned with the best, right?

And let me tell you, my career started in 1982 and I got very lucky. I was a Navy intelligence officer and the training for me back then was here's how to spot what a tank looks like, and then what kind of tank is it? And here's every ship in the Soviet Navy and what are their recognition characteristics and what are the missile ranges. Of course, we had These broad concepts of the intelligence cycle, collect, process, analyze, disseminate, manage that kind of stuff, but no real analytic training. I got lucky. I went to Naval Postgraduate School In Monterey in 1984, 85, where I got to get lectures from this guy, Dick Hoyer. And he had written several articles by then that were required reading for every student in my curriculum. And I know he was a personal friend of yours and you worked together. Can you tell us about those early days working with Dick?

Kathy gets to start.

I do. When I joined CIA's Office of Security in 1991, right before the fall of the the Soviet Union, Dick was a gift to me. He was working as an independent contractor at the time. He had already retired, had been retired for some time and was working, doing some base rate studies on security issues and with the Personnel Security Research center in Monterey for dod. He was an independent contractor for me, which is when I got to know him. Randy had already met him as part of the training that Dick did on Denial and Deception, one of the first courses in the agency. And I will say I had him teaching my security analysts in 1991 the kinds of things that we're writing our books about now. And when I became an analyst, I mean, I was dunked in hot boiling oil working on Portugal right after the revolution, which no one had paid any attention to. And so literally I started writing articles right away. And it was only my training was OJT with my wonderful branch chief Bob English, who sat down and explained to me what leads were supposed to look like. I mean, I thought I wrote well in two languages coming out of graduate school and learned that intelligence writing is totally something different. And what's fascinating about it is that what we put in our books, it's what I wish somebody had told me when I was new.

I picked it up with Dick after I retired, got seriously involved with him. He had developed a methodology analysis of competing hypotheses. And I wanted really to expand that and its use across the agencies in the IC and did some work on the S and T side to get Palo Alto Research center to put together some software. And as we started to get people involved with some of those techniques, he wanted to write a book and I wanted to write a book. And we both wanted to write the same book. So after a little bit of negotiating, we decided to write the book and we got together and we pretty much Split it up 50, 50 and published these structured analytical techniques for intelligence analysis. And he went through, he did the hard research on that going through around 300 different techniques, and then we picked out around 50 that were directly applicable to doing intelligence work and knocked that out and worked on that. But it was a great partnership with Dick, and he was an easy guy to work with. And we always. We never got into an argument, and we always found the perfect way to say something, as best I could tell.

You know, one of the early takeaways I had from those papers, he began publishing in his first book, the Was it the Psychology of Intelligence Analysis?

Right.

The early takeaway for me was that I shouldn't always believe everything I think. You know, your mind can play tricks on you. We're humans and prone to bias. And he listed many of the key biases that are still at play today and methods we can use to help mitigate that.

In fact, that was a major. When we first published the SAT book, we got a lot of pushback from some in academia and some in the government saying that they thought, number one, that structured techniques was a fad, it would be gone in a couple years, like all those other things they heard about. And number two is that it was not really valid because it was not effectively predictive of outcomes. So Dick and I then wrote a chapter saying, well, this is what people have done. Very little research. It's very hard to do a controlled experiment on how you write intelligence pieces because you don't have any intelligence analysts that are available to do controlled experiments. They're too busy doing work. So as we worked our way through it, the light bulb went off when we want to write the second edition that, well, we're not trying to predict anything with the techniques we're teaching. We're teaching thinking tools and how to think about a problem and how to work through your logic. And that took us back to his original book on cognitive bias. And we kind of developed a methodology of this cognitive bias or misapplied heuristics. And we developed a new tech category called intuitive traps. These are all the things that can go wrong when you do an analysis. And we could teach you about that forever. Like, we could teach about denial and deception issues forever. But if you don't change your business process for how you do the work, you're going to get caught and you're going to be trapped. So that's where the SATs came in saying, we can reduce your susceptibility to error by maybe half. If you do learn to challenge your assumptions, if you consider an alternative hypotheses, if you look for inconsistent data, if you do those things, then it's going to protect you against all those biases that Dick wrote about so nicely in his Psychology of Intelligence analysis book.

Great, thanks. And that does lead to your more current books. I wanted to say I believe Dick's book on the psychology of analysis is of intelligence is canon and everybod should read it. It was an early book, but a better book is maybe this one, Critical Thinking for Strategic Intelligence. Now this one I'm holding here is the second edition. You have a third edition, brand new out. But I think although Dick was great at writing about these many biases, the world has changed over the last 20 years and there's a lot of other methods and research has changed over the last 20 years. Would you agree with.

Yes. Matter of fact, one of the things that we did when we went through to start scoping out the third edition, we realized that we wrote the first edition 10 years ago and the world is drastically different than it was then. I mean, just think about, think about the types of data that are available now that weren't available then. Think about the technology. Think about how the world of analysis and research has changed. Think about the speed with which everything comes at you. And what's fascinating about it is even understanding and going through the book with each of those kinds of differences key in our mind and trying to add in pieces about writing digitally and different kinds of thinking mechanisms that can come into play. Even with that, I think one of the things that we realized was that the basics and the structure of that book still stand up today. I think both of us were probably even a little surprised at that because we were ready to do a wholesale rewrite if it didn't work. Right, Randy?

Right. Actually, I'm sitting here now doing a final proofing of a book that will come out next month or in December, which is the second edition of our analytical writing guide. And one of the things that really got our attention was a lot of intelligence analysis is now being published on the computer in digital displays. And we learned as we worked on this the last couple of years that putting out a hard copy paper or doing a briefing or doing a digital display are all totally different. So if you're doing digital presentation of analysis, you have to go back and redo your research process, your organization, how you conceptualize your paper, how you deliver the paper. The entire analytical process that goes into that production from A to Z changes dramatically when you go digital. So we put in around a 15 page chapter on all the things that you need to know to get yourself up to speed for tomorrow's. World if you're trying to do digital analysis presentation versus the old hard copy version that we grew up with.

Well, let me get back and ask a couple other questions about this book. The title of the book, Critical Thinking for Strategic Intelligence. So first I wanted to ask, in your own words, what do you mean by critical thinking?

Well, one of the things that. One of the reasons we decided to name the book and to write about critical thinking is because as we were doing some of our classes, and this is from the very beginning when we were asked to come into the FBI academy and different places, that nobody had a definition that was a really usable definition for critical thinking. You would ask people, what do you think critical thinking is? And they would say, well, it's thinking critically. So a circular definition really doesn't work here. If you go read surveys of what managers want in the future, they want somebody who's a good critical thinker. Well, what the heck is it and how do you do it? How do you know if you're doing it? Well, nobody tells you that. And so our goal was again, to help people understand those basic. As I say those things I wish somebody had told me when I was a new analyst that would make it a lot easier to understand what those thinking processes are. When we put forward the outline for the book, one of the reviewers, the famous Jack Davis, who is well known as Dick Hoyer in the world of intelligence analysis, Jack came back and said, you know, guys, you got to have a. Gotta have a definition of critical thinking. I was like, oh, I was sort of hoping to avoid that and that everybody would make up their own, right? And he said, I'll tell you what it is. And we quoted in the book, and it is the application of the rigorous science of scientific methods to a world that is not scientific. So you can't experiment, you can't control your variables, but you can use those processes to think clearly and to diagnose, to reexamine, to challenge yourself to look ahead. And those thinking processes are what is critical thinking?

Great, thanks.

That's helpful.

Actually, I'll tell you, we received, friends gave us, as we were starting this, they gave us a bumper sticker that said, randy, make sure I get this right. Critical thinking, the nation's second largest deficit. Right. So that was one of our goals, is to overcome that deficit.

There's a lot of research that's been done on this domain of critical thinking, but I have not seen such a clear and concise definition of what it is. There's also a lot of Talk about what's needed in terms of teaching even at the kindergarten level of how to think all the way through high school and then of course into college teaching critical thinking. And amongst the people I dialogue with, it's almost unanimous. Everybody believes we need more critical thinking at K12 that's going to help the nation protect democracy and help us avoid all these issues with foreign influence. And the bias and fake news and other deep fakes is if we could all just practice more of this critical thinking. I guess what I'm saying is I would love A K through 12 edition of your Critical Thinking for Strategic Intelligence. Maybe it would be a book titled Critical Thinking for Life.

That's a great idea. We've been trying to do that and looking for K through 12 teachers to partner with. And our view was you could throw away around half of what's in the book because it's too esoteric for people those grade levels. But you need to know exactly how these children and the students think and how to resonate with what they need to know. But it's particularly important as we move into the world of digital disinformation. And there's so much out there. What's nice is that a lot of the high school age people are fairly discriminating readers. They've seen a lot of junk out there and they're more inclined than the old fogies to disbelieve what they get and what they see. But there's a recent article just came out of a survey of all of the European countries as to who's done the best to combat fake news and digital disinformation. And the answer was the Finns. The Finns have taken it. They start people at K and work through 12 and they have very rigorous programs where they create a culture of how to understand good thinking as opposed to bad thinking. So they've stopped putting their energy into into trying to keep out the bad stuff. And they've replaced it with a positive narrative of this is what you should be looking for. This is how to be a critical reader, a critical consumer. And they've been educating the population. And basically what they've done, the reviews I've been seeing is that they've inoculated them against a lot of the really the trash that we see all the time. Because they have an educated population that knows what to look for and what not to look for. And when I'm teaching college classes, I'm now telling college students they have a great future in intelligence analysis except what they're probably going to be doing is taking what we learned about sourcing and what we learned about rigor and everything else and applying it to dealing with the world of digital disinformation and trying to keep that under control. And that's, I think probably the biggest challenge that we face in the next decade is finding out how to know the truth and how to find and get access to it and to know what you can trust and not trust. And that has to be a long term effort to build a new culture. And like you're saying, we would love to start it off with another version of the critical thinking book. If there's anybody out there who wants to partner with us, give us an email. We're all set.

Cool.

Great.

And I think one of the things that is most helpful that we've realized in terms of working on critical thinking kind of exclusively and how it's applied is what is it that's most, what are those skills? What are those techniques that are most important? And this is something that Randy, after leaving the UK Cabinet Office at one point and they were saying, well, which of these techniques are most important? And so he thought about it on the plane on the way home and ran it by me and I added another one and came up with what we call the five habits of the master thinker. And so, as I tell classes, all I need you to do is think about five things. Number one, learning how to examine your key assumptions. Number two, thinking not just about one alternative, but multiple alternatives because there's always more than one. Number three is looking for disconfirming as opposed to confirming data. And this is really super important for, you know, instead of investigators just following a lead, you are always looking for how it is that you can be wrong. This is the scientific method in a world that's not scientific. Number four is looking for the drivers that are underneath your topic so that you can look to the future and use indicators. And number five, which is my all time favorite, which really should be first, is what is the context and how do you frame your issue? Because it is through framing that you actually understand the totality of your issue and you can actually share frames with other people and you can collaborate as a result.

So we did write an article on that and you can google that first in five habits and it'll pop right up nowhere else you're going to find it. Great.

I'll put that in the show notes, we'll link to that. That sounds really great. It's a good succinct way to put it. I Want to ask about something else that's in your book on critical thinking for strategic intelligence. You talk about something called System one thinking and System two thinking. And back to Hoyer when he wrote things like that didn't exist in the research literature at the. So for me and my story, I went to Naval Postgraduate School. I was lucky enough to be there. Then I go into these operational intelligence assignments where it's a really different world than the strategic intelligence world. It's very dynamic and chaotic, and there's all sorts of intelligence sources, a lot of it contradictory and conflicting, some of it deceptive, all of it hard to process. And the way you analyze what's going on is just by being immersed in it and overwhelmed with it and making almost instantaneous assessments. And it wasn't taught in any schoolhouse. It was taught by mentors and leaders. And no one was writing about that years later. Now we see there's this concept of System one and System two, and it resonated with me. It's just different analytical methods, I think. Am I capturing that correctly? Is that what you mean when you write about System one and System two thinking?

Well, we see that we focus on. Well, Gary Klein did a lot of work on intuitive thinking, and Kahneman did a lot of work on the deliberative System two thinking. Actually, what makes me think back, I wrote and edited and published a memoir of Dick Hoyer's life right the year before he died. And in the book, we have a picture when Dick went out to interview Kahneman and Tversky at Tversky's house in New Jersey, I think, and we got a picture of them in their 20s. And I got Tversky's wife to give me permission to publish it. It's great to see these guys before they had long gray hair and everything else, as young whippersnappers trying to figure out how to deal with cognitive bias in the System one, System two concepts. So we focus on. We'd say there's a place for intuitive thinking. System one, if you have a huge database or a population of information you're pulling from, you know, cops know what are dirty cars on the street, because they're looking at cars all the time, and they know what to expect. But if you really have a challenge, then you have to move to deliberative thinking, which is a system two side. And we make the point that the SATs are really the best way to take you carefully through an audit trail and collaboratively through your analytical process. So we are kind of apostles of a lot of what Kahnem has been teaching for a long time.

Great, thanks. Something else you write about in the book, and you mentioned it briefly, Randy, is the cognitive trap. What do you mean by a trap like that?

Well, intuitive traps is the phrase we use. And what we say is we have the high floating phrases like confirmation bias and anchoring effect, et cetera, et cetera. But when you get down to being an intel analyst and working every day, there are practitioners mistakes you make, like projecting your past case onto the next case. You know, other things where the little mistakes that you make, drawing conclusions from too small a sample and we can go through, actually we came up through 18 of them as little things that you can just do wrong, presuming there's a pattern, when there's not a pattern. Assuming that, well, we won't go through the whole list, but there are things that we kind of collected as the types of mistakes that you're not that you end up making. And if you can go back with a little checklist, say, did I fall into these traps? Then you can go back and correct them. And the idea was just to remind yourself that these are the things that you, by human nature, the way your brain is wired, you're probably going to end up doing. So go back and check yourself to see if you fell into the trap or not.

Great, thanks. And you also write about mental models in general, but I was wondering, do you have a good working definition you can give us on what is a mental model and how does that apply to critical thinking?

Me too.

You wrote that chapter.

Well, to me, it all goes back. It goes back to that fifth habit of the master thinker. It all goes back to the context and the way you organize your reality. So again, a lot of that comes from your experience. It comes from what you know. It's one of the reasons why, you know, people from Steve Jobs to Dick Hoyer will write about the importance of reading a lot and have and expanding your horizons so that you have more knowledge in your head about what alternatives might be. So your mental model is simply the way you are organizing the data and the problem that is set before you. And again, I think that this is something that you can learn how to do better, that you can learn how to do more broadly. One of the things that I've learned in a bureaucracy is that if you end up being in a situation where you're stuck on a problem and you're stuck in some sort of a controversy, one of the ways to break the logjam is to expand the Model, expand the framework because you then put it in a larger context and that changes the dynamics that you're talking about. And so your ability to expand is what's important. I did a paper, what was that? A couple of years ago, started to work some on creativity. And how does creativity fit in with critical thinking? You have classes on creative thinking? You have classes on critical thinking? Well, creative thinking is actually part of critical thinking or, you know, one way or the other. I mean they are inextricably linked because you can't be creative, you can't imagine alternatives, the alternatives for your critical thinking, unless you're thinking creatively and jump in.

There really critical is to know what your colleagues are using as their mental models. At a conference many years ago, I was Tedlock had just started doing his groundbreaking work on foresight analysis and I was looking at his work and we had lunch and I was saying, well, as people are trying to forecast, what are the mental models they're using to become good forecasters? I said, what I do is I build a model of how things work and then I fit the data into my model, like on political instability or what causes military coups. That's kind of my background. Other people I know are historians and they work off of historical analogies and they use those models to understand what's going to happen. Or you have like the empiricists, you just collect all the data and say, what does it mean to me? Or you have the algorithm makers. So what kind of a process are you using to understand the current, to predict to the future is really important. And if you don't understand where people are coming from, what kinds of models are using, then you can have a lot of disconnects and misconnects as you're trying to figure out what the future is going to be like.

And the thing that's super important about it is to, as Randy said, is to actually have a mental model. One of the things that really scares me about AI and I remember at NGA talking about, oh, we're going to live in the data, we're going to live in the model. Well, what's important is the model you bring to the data, not the model that the computer gives to you. You're going to inevitably be influenced by it, but it's your independence of thought and your ability to bring the human knowledge of what makes your model and where it might be different from the machines. As we write in the book, think about technology. Think of that AI partner as just another smart colleague down the hall. That does not obviate the need for you to do the mental work of doing your own organization and establishing your own models.

Yeah, I think that's very good guidance, very good context, I'd say. We named our company as an homage to a model ooda, the OODA Loop. Observe, Orient, Decide and Act. For us, it's an important model because we're in a business where there's a lot of competitive action. But we know that is not the only model. There are many other models as you pursue business. I like the law of the carpenter. Measure twice, cut once, sometimes that's the model. And there's just so many others that we all need to learn. And as you said, Kathy, I think the important thing is knowing which to apply to which situation.

Well, and what you're also talking about too is you're talking about the difference between the process and the substance. You talk about the difference between the qualitative and the quantitative. And again, this is one of the things in the world that we, that we're living in today where it seems to be so simple to go for the quantitative without really knowing how it works with the qualitative. And so I think part of what we're trying to do with our books is to show a very practical way of being able to use the qualitative without overdoing it. So recently there was a wonderful initiative that Tim Van Gelder from Australia undertook to have an international discussion, online discussion, on what is analytic, rigorous. And so if you approach that from a very quantitative side, then you get a certain view of it. But again, it's the mixture that counts. It's the mixture that counts. Because if you look at it, we have time and time again, if you only take the quantitative and you don't fit in the qualitative, you are going to miss the disruptions and the surprises that happen.

Right.

Hey, I wanted to talk about a couple other things in our remaining few minutes here. One is, I was looking over your website and many of the books that you list there are interesting, but one just really jumped out at me and Randy. That's your book on how to get the right diagnosis. And I read the introduction to that and I thought, fascinating. First of all, thank goodness that you had your wits about you and you were able to critically think and interact with your medical team to help drive the right diagnosis. But it's also, it just underscores that the things you write about in your other books are directly relevant to life and can help in many other areas. Would you tell us what is this book about how to get the right medical diagnosis?

Well, I went through around five years of having an undiagnosed condition. Basically I'm in a running club and I run every week. And over time I was running out of breath by the time I got to the end of the run. So my. Basically I would say my chest was getting a little sore and I was just having trouble keeping up with everybody. And over time it got worse and worse and worse. So I tried to get a diagnosis. I actually went through. And the problem was I couldn't get a diagnosis, but I could get treatment. So people would test me and they would treat me. And I went through seven different professions, disciplines in the academic world, I mean in the medical world, trying to figure out what was wrong with me. We got to the point that they submitted me for consideration by the NIH Rare Disease Program, Undiagnosed Diseases Program. And finally I was over in Barcelona with Kathy and I was having trouble going up hills. I was carrying around a 50 pound suitcase with all these books and all this other stuff. I get back to the US and I was going over the State Department for a meeting and I was having trouble walking to the State Department and we got a little bit winded. So this is not right. My doctor saw me on Monday and said that was Friday. Then I actually went off and set a trail that weekend for everybody to run a five mile run. And then came in, she saw me, she said, you're going to the ER immediately and you got something's wrong. And I get to the ER and they give me an ekg. They said, you're fine, you're discharged. I said, I'm not leaving. Said, what assumptions are you making about my case? You know, what hypothesis do you have about what's wrong with me? And they said, and they gave me a lot of lather. They discharged me the second time. I said, I'm not leaving this building until I get an answer. And finally I said, well, the question that got them was, if I die in two weeks, what are you going to tell my wife I died of? And they said, oh. And they said, okay, you win. And I gave them the note that we sent to the NIH 3 pager saying, this is what I've been through for the last five years. And I can't get an answer. And he read that, he kind of read it quickly. He said, okay, you win. Put me into the hospital, they did a catheterization, they checked out my heart and other things. And they said, oh, and Then the cardiologist comes in and he says, well, we need to apologize for the cardiac profession. He's 90%, 80% blocked at all of the arteries in his heart. And they put me in the next morning for a quadruple heart bypass. So the good news was they caught it. I didn't have any heart attacks and have any. Just shortness of breath was the only problem I ever had. And I recovered extremely quickly because they had an extremely strong heart. And they put in four new arteries. So they gave me good blood flow. I actually got on a plane and flew to Toronto 12 days later and delivered three papers. And I actually managed to stand up to deliver the papers, which was amazing. And we got to the ISA conference a year later. Nobody had the faintest idea that I just had heart surgery. So I wrote a book saying, you need to work with your doctor as a partner, particularly your nurses are your saviors. And this is how to work the system. This is how to use some of the techniques I teach to get yourself diagnosed, to get yourself healthy. And I kind of went through all the things that the insurance industry, everybody else has done to kind of conspire against people getting diagnoses. And so that's where we. So he published it, and it's a really quick read. You can read it in two days.

Well, Randy, I want to say back to this point of framing. When we're talking about your book on critical thinking for strategic intelligence. Of course, I recommend that to anyone who's going to stand up an intelligence program in government or industry. And there's a lot of people that should read that, or anybody that wants to be a better student or a better leader. But when it comes to your book on how to get the right medical diagnosis, that's one I would say I recommend to any human being that cares about health, which should be all of us. I think it's a great way to apply these concepts to what should be front of mind for everybody.

Well, one really quick tidbit, which I applied many times, was when you go into the doctor's office, watch how the staff interact with each other. If they're enjoying each other's company and they're kidding around and they're smiling and everything's working smoothly, you got a good doctor. If they're on each other's backs and they're dour or whatever, get out of there. Go find another doctor. And I found that very strong predictor of whether the staff cares or the staff is just putting you through the mill every 15 minutes. And ET cetera, et cetera.

Great.

What's great about the book is that there's stories from lots of different people, because everybody has a circumstance where someone can't get diagnosed, has passed away without being diagnosed. And again, the point is that this gives you something to do to help get that diagnosis.

Or the doctor understood them, they had a good conversation, they paid attention, and they're now alive. I mean, like, one of the things I like best in the book was we put together a list of how much pain do you have? And you go in and say, well, on a scale of 1 to 10, how bad's your pain? And so this is the most idiotic thing to ever ask anybody because you've got no point of reference, whatever. But. So I came up with a list of words, and I actually created a new taxonomy. I have aches and I have pain. And I think they're two very different things. Some doctors agree with me and some doctors don't. But if you come up, you can take that list and say, that describes what I'm feeling, then this is manna from heaven for the doctor. Because if it's a sharp pain versus a dull pain, it's an ache, it's a whatever. Does the pain migrate around the body? Is it on the skin, or is it deeper? All these things. If you can just use that one page to describe exactly what you're feeling, then you're making huge success very quickly with trying to get a diagnosis. Right.

Thanks. And, Randy and Kathy, I have, I guess, one final topic I would really appreciate your thoughts on, and that is any advice that you may have for us on how to stay current in this current age, where the pandemic has really changed the way that we interact with people and learn from people. Conferences, for example, it's totally different. People try to do online zoom conferences, and it's not the same. You're not sitting down with someone and interacting with them in person. And it's really tough to stay current. And so I would appreciate it if you guys have any thoughts or advice or tips or best practices you can pass along for staying current in the pandemic age.

I mean, I have to say that to me, staying current relies on continuing to try to solve problems. So, as you know, one of my big passions is security reform. And one of the ways that I stay current is what is it that we're going to do if to try to improve security processes in this new world with all the data and all the technology? By doing that and with increasing threats that Makes me try to stay on top of what's going on in the world of counterintelligence. It helps me try to stay on top of what's happening with technologies that might be able to be used for security projects, processes in terms of what's available, in terms of data sources. But yet the focus of it all is what does this mean for the individuals who are actually going to be carrying out these processes. So to me, it's really taking the problem, thinking about it in terms of what people can do about it, and then it's what are all the directions you need to go to keep up to date. From there, I often think back to, you know, some of the. Some of the futures thinking books from the 90s that always talked about reading fringe literature. You know, of course at the time, I guess Wired magazine would have been considered to be fringe literature, but again, it's lots of different wide variety of blogs and podcasts and I'm loving things like Blinkist and things like that that help you be able work through lots of material in a short amount of time.

Great.

I just threw in two quick ideas. One was it really helps to have a network of people who feed you data that you trust. And so build a network of filters out there who have already curated the information. They know what you care about and they'll send you something like, I have a colleague who's worked a lot with me on the Arctic, so I get notes from her once a week, a new article that just popped in from someplace just to keep up with how fast and radically that's going to change up there on the North Pole. The other thing that I've taken on is I like to hire a mentor or I like to mentor a grad student, a master's degree student. And I've got a couple programs that I've been involved with and I can use the master's degree student, number one, selfishly to have them do research for me if I want to get into more work on disinformation or whatever the topic is. But it also then gives me a fresh perspective on how they see the world and what they're doing. So I can get a lot of data and a lot of hard, difficult research done, particularly if I mentor them on their dissertation. Then we get into the substance of what they're working on is something that I care about a lot. And what I've done twice now is we've actually published joint articles with them as a master's degree student and me as someone who's been around a bit longer and we get them published in good journals. And of course it's all the world to them. And I like it because they do all the work and I get all the credit. So it works perfectly.

That's great. Well, Kathy and Randy, thanks very much for this time and insights. I really appreciate it.

Thank you. It's been a lot of fun.

Thanks for listening to this OODA Loop production For the latest analysis on cybersecurity technology and global risks, please visit www.oodaloop.com.