#0108 - Wiring the Winning Organisation with Gene Kim

This is the no Nonsense Agile Podcast. Join us for discussions on agile product development and leadership with world class experts who provide valuable insights and practical advice for industry professionals. Subscribe now to learn the latest trends and best ideas in the field.

In today's episode we talk to Gene.

Kim, leader of the Enterprise Technology Leadership Conference and author of the Phoenix and Unicorn projects. Gene describes how DevOps research shows that excellent socio technical leadership is critical to team success and failure. He explains how technical leaders can dramatically increase team effectiveness by improving the organizational architecture and wiring to support independence of action, rapid feedback and simplification while maintaining stability and security.

Welcome to the no Nonsense Agile Podcast. I'm Shane Gibson.

And I'm Murray Robinson.

And I'm Gene Kim.

Hi Gene, thanks for coming on.

I am delighted to be here.

Why don't we start off by getting you to introduce yourself to the audience?

Oh, for sure. So I've been studying high performing technology organizations for 25 years. That was a journey I started back when I was the CTO and technical founder of a company called Tripwire in the information security and compliance space. Our goal was to study these high performing organizations that simultaneously had the best project due date performance, the best operational reliability and stability, as well as the best security and compliance. By far the biggest surprise was how it took me into the middle of the DevOps movement. I got introduced to this community in 2010 and my area of passion has been studying how large, complex organizations that have been around for decades or centuries are using those principles and practices to win in the marketplace. So I got to work with Dr. Nicole Forsgren and Jez Humble on the state of DevOps research. And I spent the last four years working with Dr. Steven Spier trying to understand what is common between agile and DevOps and lean and safety culture. And that all went into a book called Wiring the Winning Organization that came out last November.

So did you start your career as an engineer?

I got my graduate degree in computer science focusing on networking and compiler design, and spent the better part of the 2000s, up until 2016, thinking of myself as an infrastructure and operations person. So the Phoenix Project was written from the perspective of an OPS leader, but around 2016 I've really changed my mind. I consider myself a developer. Hence the Unicorn project that came out in 2019 was essentially rewriting the Phoenix Project through the eyes of a developer. What does it look like when you're stuck in an organization where you can't get anything done? You can take your 10x developer, put them into the horrible Phoenix Project and suddenly the best developer can't build, can't test, can't deploy, can't get logs, can't get license keys, can't do anything without approvals, architecture review boards, information security, saying no and just showing how do we transform that organization where small teams can work independently and get amazing things done. And it has nothing to do with the skill of the developer, it's really about the wiring that they exist within. And that's really the job of leaders. Create the wiring that liberates everyone's full problem solving capabilities that enable this independence of action so that small teams can do what needs to be done and spend their best efforts on the technical problem versus the communication, coordination, prioritization, deconfliction, cajoling, politicking that's often required to get things done. So much of what I discovered through the stead of DevOps research show how this is universal, independent of industry, vertical, independent of the phase of value creation. I think there's really two constructs that really were this big aha moment for me. One was the observation that there's really three layers of which we do work. Layer one is the work in front of us. So it could be the patient, the middle that needs to be transformed, or the code, or the binary that's going to run in production. Layer 2 is about the technology that's needed. So it could be the press, it could be the MRI machine, it could be the code editor, IDE could be the Kubernetes platform. But layer three is the organizational wiring. It's the social circuitry that dictates who gets to talk to who, when, about what, in what format. When you take a look at any case study of worst to first, in general, the only thing that changes is layer three, the Mandarin system, whether it's Etsy, the transformation of Facebook, or the Nummi joint venture in Fremont, California. So that was the worst performing General Motors automotive plant around the globe and then became the site of the NUMMI joint venture with Toyota. And so within one year they went from the worst performing plant in North America to one of the best performing plants on par as any in Japan. And so what's remarkable is that it's the same people, same floor space, same capital equipment, same technology, same people. The only thing that changed was layer three. Layer three dictates whether you are high performing or low performing.

What does the research say about the success factors for more effective organizations?

So one of the key findings in the state of DevOps research was simplification, partitioning work to Enable independence of action. It was one of the top predictors of performance. To what extent can teams do what they need to do without a lot of coordination with people outside of their team? To what extent can they make large scale changes to their parts of the system without permission from anyone else? To what extent can they use their own testing on demand without the use of a scarce integration test environment which couples you to everyone else in the enterprise? To what extent can they do a deployment without needing to coordinate with platforms to depend upon? That's all about independence of action and contained blast radiuses and that everyone has what they need, when they need it, from where they expect it, as opposed to everyone being stuck. So that's a function of architecture and how teams are organized. The second key finding is amplification. And in the ideal we have really energetic feedback loops where even weak signals of failure are amplified. They can be acted upon quickly to ideally detect, correct and prevent. It's actually feedback that allows people to learn. One of my favorite examples is when developers develop things put into production and all the feedback goes just to the infrastructure and operations people. It's like, whoa, feedback is being funneled to the wrong people. And then the third one is really about slowfication. There has to be enough time to allow for planning and practice and experimentation and improvement. And there has to be a place where you can do your most consequential dangerous work. Not in the production environment. Can you do it in planning or practice? Can you do tabletop exercises? Can you do simulations, rehearsals? And if you can't do that, then that means all your most dangerous work is being done in production. When you can't undo and redo, which means that you can't iterate and learn. It's really about slowifying moving work and time. So it's not done in performance, but done in planning practice. There's simplification about partitioning work to enable independence of action and amplification, which is all about signals. How do we get signals stronger and going to the place where it needs to go.

Let's go back to the basics. What is DevOps?

Yeah, what is DevOps? It's a set of architectural practices, technical practice and cultural norms that allow organizations to rapidly experiment and iterate. It allows organization to deliver value quickly to the end user customers. And it's a way to do that while preserving reliability, security, stability and so forth. And so why do we do that? It's so that we can win in the marketplace. It's so we can achieve organizational and mission goals.

Does DevOps mean that we're going to integrate development and ops into one team sometimes.

I love the example of Facebook where They said in 2009, they're having all these incidents and outages. It was sort of like the downward spiral where code quality, service quality kept on getting worse. Number seven incidents kept on going up. They had this rule that in infrastructure and operations, you can't have your laptop open unless you have a live site incident going on in the meeting. And there was one meeting where all 15 infrastructure engineers had their laptops open, working on separate incidents. And that was a signal that whatever they were doing was not working. And the decision they had to make was that they were going to put dev managers and architects on page rotation. And it was the first time that many of these people saw the downstream effects of what they were doing in the daily work. And suddenly, within 6, 9 months, the number of 7.1 incidents went way down because feedback was now getting to the developers.

So I've seen people treat DevOps as the combination of DevOps. So you build it, you release it, you maintain it, it breaks, you fix it. If it doesn't work, then that feedback loop comes to you much quicker than going somewhere else. I've seen the whole SRE movement being attributed to DevOps. That idea of thinking about things that break early and then fixing the root cause, I've seen it treated as agility in terms of team design. This idea of how quickly can we move safely, how do we give feedback loops to the team, how do we allow them to experiment early, how do we bring the risk and the uncertainty forward? And so there seems to be a bunch of patterns that are all valuable that could be pushed into the DevOps bucket, but also could be product patterns or agile team design patterns. Is that how you see it?

Yeah, absolutely. I would say it's all of it and a bunch of principles. So let me give you an example about the data ops angle. One of my heroes is Mike Nygaard. He wrote the book Release It. That's a phenomenal book about designing more resilient services through things like the bulkhead pattern and the things that Netflix made famous. So he's now the VP of Data Engineering at Nubank, and he was talking about DataOps being the land that DevOps left behind. Nubank, one of the largest banks in Latin America, is actually one of the largest payment platforms in Asia. And the problem that he brought in to solve was our batch jobs are taking longer and longer to run to the point where they're taking more than a Day the storage and compute costs that execute the batch jobs are spiraling out of control. And one of his principles that he's using to take the organization forward is who should be responsible for it. So the whole notion of you build it, you run it, they're moving data ownership back to the product teams. So they have to be responsible for the storage costs and the compute costs and the SLAs in terms of how quickly data has to get to the people who depend upon them. So the signal's being sent to where they need to go. But it makes no sense to have a data team that's the dumping ground of everyone's shareable data. That takes too long to run, that's not fit for purpose, that's not what people are looking for. Instead you have to bring that responsibility to the service team, ideally a combination of dev and ops. And if you have a good platform team, you don't have to have a lot of infrastructure specialists there. They're providing shared services that are exposed so that other people can take advantage of great infrastructure that's tailor made for that organization. I think that's another marvel example of how you wire an organization one way and you end up with these runaway problems of batch jobs taking too long. But if you wire the organization a different way and now you end up with a self balancing pattern where the service teams know exactly what they need to do, they're being held accountable and they get the feedback they need to achieve the goal.

Yeah, so DataOps is really interesting. We had the whole data mesh thing come out, which was a great set of principles, but no execution plan. So now it's being taken over by the vendors. But if I look at the idea of pushing the data work back to the product teams where it should be, so we shouldn't treat data as exhaust, where we take the crap and then we have to clean it up and we don't control it. The analogy I use is you're a chef in a restaurant and somebody keeps delivering rotten tomatoes. You ain't going to put up with that shit for long. But as data people we do. But one of the pushbacks is that to get the data skills into a product team is hard. But we did that with DevOps. We took the operation skills that were hard and we actually pushed those skills back into the dev team. And so it is possible, it's just that we haven't focused on doing it yet.

Let's zoom out just a little bit and talk about why these patterns emerge and what I Learned Working with Dr. Steven Spear is that the more functional specialties you have, the more sophisticated your layer 3 wiring must be One of the best examples of organizational systems that make sure that no one has what they need when they need it are emergency departments in hospitals. Why is that? It's because the number of functional specialties have grown. They have so many different functional specialties which need to be integrated towards a common purpose. And so what was really surprising to me was that in the 1950s it wasn't so difficult. The hospital systems were safer, it was easier to get what you needed. And the reason is that in typical hospitals in the 1950s, at layer one, you had basically doctors and nurses. At layer two, you had very little technology. You had basically X rays which weren't typically complex or fast moving. So you could get away with a pretty simple layer three wiring to run the hospital system. Fast forward 70 years to current day, you have 20 different functional specialties. Just among the clinicians, you have nurses, supply chain, pharmacy. That's at layer one. At layer two, look at how much technology we have. It's not just X ray machines, it's MRIs, it's just cat scanners, blood tracers, computer systems, the electronic patient record systems. So imagine just how much more sophisticated the layer 3 organizational wiring must be to get people what they need when they need it. And so the same thing has happened in our space. It used to be pretty simple. In the mainframe days, you had your single cross functional team that was simultaneously responsible for mostly building and running. You had the operators, of course, but now you have platform teams, you have container teams, you have security teams, you have data ops teams. Just the number of miracles that technology affords continues to grow. But that's why we have to have more sophisticated organizational systems to enable developer productivity. All I want to do as a developer is work on my feature. I don't care about logging or the environment or kubernetes, deployment files or authentication or data masking or where my data goes, who I need to connect to. And what's marvelous is that it's the emergence of these development platforms that allow other teams to do that for you. And you don't have to open up a ticket, you don't have to chase them for six weeks, you can get them on demand. And I think that's what creates these miracles that make us so productive.

We've just interviewed Arnold Strobeck from Birtzorg and they have these cross functional clinical teams, a team of 10 managers themselves, almost completely. They're management overheads are 1/2 of 1%. So there is a industrialization scaling argument that comes out of Taylorism, which is causing the specialization and the silos and then the super complex systems, I reckon.

Yeah, well, what causes specialization of skills? In the 1950s you didn't have a lot of functional specialties. You didn't have neurologists and endocrinologists and so forth. But on the other hand, by the time you learned that you had cancer, it was basically a death sentence. You had mortality rates of like up a 95%. And so what science has done is enable this ever deepening functional knowledge of increasingly narrow domains. But the reward for that is that we went from 95% mortality rates for cancer down to if you detect it early enough, you have 90% chance plus of surviving. And so too in the worlds we live in. A couple holidays ago, I spent an entire week learning about Java logging. I do all my programming in Clojure that runs on the jvm, but I was never a Java programmer, so I never actually understood how it all worked. And one time it just became intolerable. I just couldn't get logs to show up where I wanted them. And so I spent a week learning about Java logging. It wasn't terribly rewarding because I didn't actually care that much. I got it working, that was important to me. But did I really care about the history of Java logging? No. If I could wave my wand, what would have happened? Someone on a different team just specialized in Java logging and I could just copy configuration file and I would just put it into my code and they would tell me where I could find my logs. And so this group has liberated me from worrying about things that I actually don't care about. Same for CICD Pipeline, same for how to get things running in production. And so I love specialization of labor so that they can do it. So I don't need to worry about it.

It's great if it works well to support you, but what if there is specialization of labor and you have to sit there for the next two weeks waiting for them?

Absolutely. And so I'm being held responsible for the quality of my data and making sure that the batch jobs run, but I don't know anything about it and I have to open up a ticket with a data specialist and I have to wait six weeks, eight weeks. It's terrible. In fact, let me give you an even worse example. A friend of mine was telling me that they're a mobile telco provider, 20 million customers and the number one initiative this year one of the top five certainly is to present a checkbox to all of our customers so that they can opt into a $5 a month service to get email or watch movies. The problem is that it requires work from 40 different teams across four different customer channels. This requires CEO minus one level support. Daily war room meetings, 28 million 12 months. And the bad part is that most people, when asked, give it a 20% chance of success. Why? Because it didn't work the first two times they tried it. And so this is not a complex problem presenting a checkbox to those customers. This is purely a problem of layer three, communication, coordination, and most importantly, architecture. Somehow we've created an architecture where it is impossible for small teams to get anything done. We want systems that allow people to work independently of each other. And sometimes when you have these complex dependencies, the best thing we can do is partition them so that teams regain independence of action. And functional specialties are given in a way that can be self service and on demand without creating the queuing problems, because queuing and prioritization and scheduling is so absolutely dangerous.

So you close that out with what I think is the key point and it's around self service. So the examples you used are where we've had specialization and then they've been codified as a machine that everybody else can use. And so the problem we know is humans take longer than a machine. Humans do it with variability. So we start creating these human bottleneck because we're trying to make the wiring of the organization act like a machine, but the humans are the cogs. Whereas if it's better technology, if your Java logging framework was set up and you knew that you push the config file to the machine and the machine will give you a standardized response and it can scale to one response or a million, you don't need to care. And that's a great specialization, but a specialization of the machine, not a human in the loop. And I think that's where we've got it wrong, is that we now go into micro specialization of humans, but they can't scale. We treat them like machines and they're.

Not the patterns and techniques to create wiring that works. One is modularization and platforms. Those are very similar, right? As we're enabling independence of action by making sure that work can be done in parallel. So what do you do when you have interdependent sequential steps, like an assembly line in the Toyota production system, where you deliberately sequentialize work so that they're tightly coupled and you want to make sure that they're not shared. Once you start sharing a critical resource, whether it's a machine, a person, this is where you get into contention and this is where wait time goes up exponentially. Team of teams is a great one where when they're trying to dismantle enemy terrorist networks, the value stream looks like US Army Rangers, intelligence agencies, and then US Navy seals. Those are all people analyzing things, trying to get data to where they need to go to synchronize a bunch of efforts. And you do that by not reorganizing the entire U.S. department of Defense, but making sure that the interfaces between US Army Rangers, intelligence agencies, and Navy Seals are well known. So that you can go through the OOD loop quickly, seamlessly, predictably. When exciting to terrorist, capture takes months or quarters, that will never result in a capture because people are trying to evade capture. My claim is you can do this both for mechanical systems and electron systems as you can for human systems. So how do you achieve that? It's really through partitioning. I'm just rereading Eric Evans book, Domain Driven Design. This is all about creating hard modular boundaries in the design process as well as the operations process, so that when bad things happen, they're contained to that domain.

Whenever you talk about domain boundaries, you find a bunch of people arguing where the boundary is, right?

Yeah, so much. One of the things I'm proudest of in the book is understanding the notion of coupling and what it means to be decoupled. Imagine two guys moving a couch. Let's call them Steve and Gene. Steve and Gene are coupled through the couch. What affects one affects the other, and vice versa. You might think two guys moving a couch is all brawn work. There's no brain work needed. And yet Steve and Gene have some problems needing to solve. Like where exactly is the center of gravity to get through a narrow doorway? Around which axis should they rotate the couch to get through a narrow, winding set of stairs? Who should go first? And should they face forwards or backwards? They don't need consultants, they don't need focus groups. Just by picking up the couch, trial and error, experimentation, communication, coordination, we can have some confidence that Steve and Gene will figure out how to get the couch to where it needs to go. But there are all these things that leaders can do to make their job more difficult. So one way is you turn off all the lights, and so suddenly it'll take longer. They may damage the couch, the room themselves, that's not so good. But there's another way that leaders can make their job more difficult, which is by introducing a lot of background noise or putting an intermediary between Steve and Gene so they can't talk directly with each other. Maybe they have to go through JIRA work tickets and that's their only mode of communication. Suddenly it's not so absurd to think that Steve and Gene will not successfully move the couch. And I think that's what DevOps is. They got to a point around late 2000s where the only way to do a deployment was through tickets. And no matter how many fields you added, there was just not enough bandwidth in that communication channel to enable them to safely quickly deploy code. And the answer is you let Steve and Gene talk directly with each other and co create the solution. So really it's a metaphor for joint cognition, joint co creation. To what degree are Steve and Gene able to act as a coherent whole? So in data Ops, if a service team is acting as their data is just exhaust, you're not going to get the right integrity of data and downstream batch jobs and so forth.

A lot of things you're talking about are the same things we talk about in the Agile community. I think DevOps is much more technically focused. I saw DevOps emerge out of the agile community in 2010. I was an agile conference in 2010 when Jez Humble spoke about it for the first time. So what I want to know is what do you think is the relationship between the DevOps community and the Agile community?

I've heard people argue that DevOps is a subset of Agile. Some would say no, Agile is a subset of DevOps, but I'm not sure if I care. There was an old bad way to do things, and what I've learned is that there's really three orthogonal axes of performance which completely describe all the ways that we can do things poorly or greatly. One was about independence of action. Are people configured and divided in a way where they can get something done? There's another axis which is that no one gets the feedback they need or the feedback's being generated, but it's going to the wrong people. And the third dimension is around slow vacation. Are you allowing enough time for improvement of daily work? So those three things I think completely explain whether a system will be well performing or poorly performing. And I don't care whether you call it Agile, DevOps, Lean, Toyota production systems, safety, culture, resilience engineering, whether it's Westrum organization models or Conway's Law, what Deming taught us, and technologies, all of these things are trying to describe these concepts.

I think continuous improvement and humble leadership is absolutely Core to all these?

Hmm, yeah, for sure. The larger the organization, the more the leader has to be the socio technical maestro. Dr. Westrom said there's five characteristics. High energy, high standards, great in the large systems thinking, great in the small because they have to know when things are being misrepresented. But most importantly, they love walking the floor, hanging out with developers, seeing what daily work looks like. That's the highest fidelity signal a leader can have about to what extent are they enabling people to do their work easily? And I don't care where you draw the boundary. Is it dev, is it Ops, is it DevOps? You need all of those five characteristics.

I see people complaining about zombie DevOps six months ago that DevOps team with an infrastructure team, they're working in a highly hierarchical organization with very siloed communication and very slow feedback loops and very long delays. And if you want the DevOps team to do a firewall rule change for you or get you a virtual environment, it's going to take three months. We've been talking about the destruction of the agile brand name caused by corporations slapping Agile word onto things. Is the same thing happening in DevOps and if so, what can we do about it?

Yeah, I think you could say the same thing is happening in DevOps and the same thing happened in Lean. Dr. Steven Spier was talking about how the Lean community got overly focused on the tools. It was not so much about a community of scientists learning together, but it became about the andon cord and the six S's and the standard work. Really missing the bigger picture without a lot of energetic leadership. That's just the way it goes. Somehow leaders think you can buy it in a box and that abdicates their responsibility for the overall performance of the system. I would say leaders responsible for the system level goals and they should never be a sucker to someone selling you DevOps in a box, Agile in a box, Spotify model in a box. You just hear all these heartbreaking stories of consultants coming in and saying we're going to Spotify you and they leave the system worse off than when they came in. You can say it's their fault, but no, it's your fault. You are part of it. You are complicit as leader. You are responsible for being the technical maestro. The opposite of leadership, enabling people to do their work easy and well is that they make everyone's work harder, terrible, miserable. That actually shows up in the state of DevOps research. To what extent do people feel satisfaction in their work, feel connection to the mission, to what extent can they recommend their organization as a great place to work to their colleagues and friends? That's all in there.

Gallup Polls have been doing this research on employee engagement for many decades, and they've said that only about 25% of people truly feel engaged, supported and able to do their best in an organization because of the system, the leadership.

And yet we know that in these great organizations people have a deep connection to the work. They have satisfaction in the capabilities they give to the customers. Someone's complaining about a button that's in the wrong place and hey, they deliver to their customer the next day and they get the deep gratitude and excitement from the customer because they're co creating solutions. How great is that?

Yeah, it is great. There's been quite a lot of complaints about SAFE in the Agile community. And I know, for example, the Team Topologies authors said the Safe people took their stuff and changed it all around so it wasn't what they said it was. And the same with the Scrum people complaining a lot about Safe institutionalizing Zombie Scrum. What do you think about the safe community and what's the relationship with DevOps?

Yeah, I'm a big admirer of Dean Leffingwell, the original author of the Safe Framework. I took his training and you can't sit through a week of listening to Dean talk about problems that he solved and not have a tremendous amount of appreciation for his experience. One of the things that really surprised me was how many high performing technology leaders had chief architect in their title. They were head of product engineering and chief architect, which makes a lot of sense because that's not all about organizational wiring and the socio technical system. But I've also seen organizations where it takes a year to ship a checkbox to the customer, and I find that very frustrating because that's an architectural problem. Another behavior that I've seen is that people treat the PI planning process as the only mechanism to do synchronization and coordination, which is absurd. And I've also seen PI planning sessions where it's too big. People sometimes describe these meetings as a total waste of time because it's about things they don't care about. They need to shrink it. I tend to one at Rally Software, which is where many of these safe practices were born. And the big room planning session is incredibly energizing. It's about dependencies that we want to detect now rather than later to enable better execution. When it's done wrong, it's about planning stuff that no one cares about, or they view it as the Only time you can talk to someone else, which is terrible. What you want is for teams to be able to identify who your dependencies are. Maybe even set up ad hoc cross functional teams so they can work together on a feature together independent of the rest of the release trainings. That's invigorating and it guarantees a way to communicate and coordinate with each other. So I've seen safe done well, I've seen it done poorly. And I put the blame again on leadership. And to the people who have the certifications who are doing it wrong, I.

Think that can be consultant. Companies are just giving leaders what they want. If leaders want some new words and some new processes, then that's what they will give them if it's the easiest way to make some money.

I was a card carrying member of the Institute of Internal Auditors for six years and that was amazing because they defined the International Professional Practices Framework, the book on how internal audits should be performed. I define what is an audit and what are the phases and how do you perform them and what are the standards. And most importantly they had what they called the quality assurance review process. Auditors for the auditors to make sure that what you do is in accordance with what's defined in the IPPF. And I think whether it's scaled agile or a McKinsey there should be some QA framework where people can objectively judge are they doing things in a way that is in accordance with what is prescribed.

What can we learn from military leadership?

Oh goodness, so much. General Stanley McChrystal and Lieutenant Commander Dave Silverman were co authors of the book Team of Teams. And Dave told me that they were shocked by the lack of formal training that leaders have in the commercial domain. And it really was dazzling just how much formal leadership training people in the military have. Both non commissioned officers, enlisted folks as well as officers. As an example, a friend of mine, he's a lieutenant colonel in the US Marines and he's going through graduate school to be an effective colonel and you have to go through another big jump in training to become a flag officer, a general and admiral. Some of it is technology specific or like about the domain but so much of it is about how systems work and what is it like to be a leader of leaders and how systems are wired. When you are having to achieve a mission where you can expect there will be no communication team with teams is so interesting because that was all about the Special Operations Command, about how the different functional groups, Army, Navy intelligence agencies need to modify the wiring to make sure that sighting could lead to capture. I loved how they described the notion of liaison office. And you wanted to have people within the special operations community embedded in embassies because they're the closest to where the intelligence is. And so what does that have to do with our work? I think if we look at the DevOps movement, it really came from having these people who could span boundaries. You had the best infrastructure and operations people, hang out with the best developers, hang out with the best architects, hang out with the best information security people. They created a team of teams that achieved a larger mission, more than just infosec, more than just dev and ops and so forth. And they describe so well the characteristics of that person that you choose to work outside of your functional specialty. They have high levels of empathy, high levels of ability to walk in other people's shoes. They're going to be very good at communicating. They know what the larger mission is. They're very good at making friends.

So for me, Phoenix Project was a seminal book because you told a story, and in that story were so many organizational patterns, so many technology patterns, so many personal behavioral patterns. What made you write that way?

So some of my favorite books are of that style. It's called the business fable 5 Dysfunctions of a Team by Patrick Lencioni, one of my favorites. I remember reading that on an airplane and I was just so stressed out. It was about a startup, an executive team that didn't trust each other and got nothing done. And I was like, oh my gosh, it's a book about me. But I think the most famous of that genre is a book called the goal by Dr. Eliyahu Goldrat, which is referenced in the Phoenix Project. For those of you who don't know, it's a novel about a manufacturing plant manager who has to fix his cost and due date issues in 90 days. Otherwise they shut the plant down. And I think that book sold 5, 6 million copies. It's integrated into almost every major operations, research, MBA curriculum, and so forth. My fellow co authors and I wanted to write the Goal, but for the IT context. Same number of pages, similar cast of characters, similar challenges that they had to face. It's been very pleasantly surprising just how relevant it is now as it was 10 plus years ago, is just a fantastic way to communicate problems where other people say, holy cow, this could be about me.

What interesting things are emerging in the DevOps community.

I've never had more fun than right now just experimenting on how can I be a better developer aided by AI like GitHub, Copilot Chatgpt, Claude. There are all these things that I dreamed about doing for a decade that I have suddenly started working on, because now it's within reach. I'll give you an example. For almost 15 years, whenever I've been watching YouTube videos, listening to podcasts, including years, I take a screenshot whenever I hear something interesting because it's my hope that someday I'll go back and look at the timestamp, go to the transcript and figure out what was so interesting and write something about it. I have over 1500 screenshots over the last decade, of which maybe a handful I've gone back to do something about. And yet I did it for a reason. So since January, I've been playing with, using ChatGPT, API, and more recently Claude and Google Gemini to give a screenshot. And I say, extract out, what podcast was it, what episode, what was the timestamp? Pull the transcript and find out what was being talked about. It's so cool. So in closure, write a function that takes a bitmap and march down the left hand side of the screen until you find a specific color, red, and then march right, and then calculate what percentage complete the video is. I mean, I could have done that, but that would have taken me two days at least. I haven't written any code like that in 20 years. I could do that in 30 minutes. So it's just amazing how much we can do. There are projects that take days, weeks, that you can now do in hours. A friend of mine, Steve Yeage, told me his company, sourcegraph, their cto, said we wanted to do an integration with Emacs. That would have been a summer intern project that would have taken eight weeks. He did it in two days solo. This is showing that the job of the technology leader is not getting any easier because you have these new problems. Like a friend of mine, he did a pilot within his bank, and they ultimately called the coding copilot a failure because junior developers were able to commit more code. But the problem is that it overwhelmed the senior developers because they had to review all of it, of which most of it was not actually deemed fit for production. And so that's a wiring that just doesn't work. Somehow the pilot as designed, did not achieve its goals, and so they're trying to rethink. All right, given the mix of junior and senior developers who should be using it, when and how do we still ensure that they're writing great code that can be reliably put into production?

All right, Shane, I think we better go to summaries, what do you got?

Excellent. One of the recurring things you've talked about all the way through is wiring. The structure of the organization, the social, technical architecture, system thinking. How do organizations go from worst to first? They don't change their people, they don't change the technology, they change their organizational wiring. So the more specialization you have, the more complex organization wiring you need. As soon as we have a shared resource, we introduce constraints. As soon as we introduce restraints, we introduce wait times. They go through the roof and now we have a system failure. And a lot of organizations don't look at their flow of work because they don't treat it as a system. They don't look at the handoffs and they don't look at the delays. They don't bring any of that lean behavior into their teaming. And how do people become unstuck? How do we free up that architecture, that flow across the organization? Three things, slow ification, simplification and amplification. So how do we slow things down? How do we remove the uncertainty and move it to the beginning and not test in production? What can we do? So, you know, a sprint planning exercise is about reducing uncertainty. It's around communication research. Spikes are about take the heavy, gnarly stuff that we don't know how to solve and do a little bit of work upfront to reduce the uncertainty. This idea of simplification, let's map out the nodes and links, let's figure out where the constraints are and let's unblock them, moving the data back into the product team. So how do you move the work to the person that actually starts it?

And that is about software architecture. How do we partition and group things to enable independence of action?

So we want to decouple. We want to decouple the work as much as we decouple the systems. And we want to introduce independence of action. We want to be very clear about how we're dividing that work and where those domain boundaries are. We want to make sure that there is feedback up and down or middle out. And we want to have people focus on improving their work. How many leaders actually stop and observe the system? In a manufacturing plant, they would walk the floor. How many leaders walk the wiring of their organization rather than sit in their little offices and wait for the problems to bubble up? So for me, wiring systems, thinking nodes and links, it's this recurring pattern and thinking of looking at it holistically, end to end. And yes, people and technology are part of it, but actually the system is where we get the majority of the failures. Murray, what do you got?

Yeah, the key thing for me is leadership. So leaders should be working on the organization wiring, which is to me the system. It's system thinking. In addition to that, the leaders determine the culture. All the problems we've seen with agile being turned into the opposite of itself all comes back to, to the leaders. And good leaders can make an amazing difference to an organization. And we've got some really good examples of good leadership from the military in team of teams and turn the ship around and mission command and all that stuff. There's a lot of really good ideas about empowerment, decentralization, rapid feedback, which are very helpful. So maybe organizations get what they deserve because that's what the leaders set them up for.

So whenever you have an interface, you want to have the clearest signaling so that the person on the other side of the interface will react in a predictable way. And so when you have no control over the other side of the interface, or when that other side of the interface is a very long distance away on the org chart, what's the best thing they can do? You can actually put a trusted person on the other side who can help translate, who can help communicate more clearly. And it reminds me of how the Apollo astronauts did it. So during an Apollo mission, the only people who were allowed to talk to the astronauts in space were other astronauts. And in general it was the people who trained them, it was a backup crew and one other person. So in other words, these were not just any astronauts, these were really good astronauts. And the reason for that is that when the band was between the crew in space and mission control in Houston, is that tenuous and potentially absent? The best thing you can have is an avatar of the astronaut on the other side of the radio because that will enable better communication than any non astronaut could. And so just another example of what great socio technical maestros do to ensure that the system behaves well as a whole despite very adverse conditions between two nodes.

If people want to learn more about what you're talking about, what are three books that they could read?

Yeah, if you like the Phoenix project and you want to get more of a dev centric view of that, I would read the Unicorn project. The Accelerate book documents the work that we did, the science behind DevOps and why it works. And if you want to open up the aperture to the real mechanisms at work, it's warranty winning organization. Anyone interested in AI I would recommend CoIntelligence by Dr. Ethan Mollick. He's an economist, he's not a computer science person and it's just this marvelous practical way of what is AI? Why should you care? What could it mean to society and workforces? And fun book I've read lately is Life in First Person. It was about the making of Doom and Quake and all the folks around ID software. It was fabulous.

So how can people engage with you? I know you run the DevOps conferences. Do you provide training, consulting services?

Yeah, I run a conference called the Enterprise Technology Leadership Summit. We're going into conference number three. We're going into year 10. The goal is highlight the most heroic technical leadership stories ever. And it's just been so fun and rewarding. You can reach me on Twitter, I'm real Gene Kim and on LinkedIn, so. So that's probably the best way to reach me.

All right, that's great. Hey, we really appreciate you coming on. Gene.

Oh my gosh. Shane, Murray, congratulations on the podcast and the guests that you've had. This is such a treat and thank you for all your thoughtful questions.

That was the no Nonsense Agile podcast from Murray Robinson and Shane Gibson. If you'd like help to create high value digital products and services, contact Murray Evolve Co. That's Evolve with a zero. Thanks for listening.