What is DaveOps? with Rich Jordan

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Test Skill DevOps toolchain. Today we'll be talking with Rich Jordan all about what is DevOps? That's right, I said it correctly. DevOps rich, if you don't know, is a 20 year testing professional with years of experience leading teams, creating organizational wide testing strategies within a large financial service organization, leading the test engineering community of practice and creating the enterprise DevOps strategy. Rich been a regular industry speaker including Gartner and DevOps Enterprise Summit, and is currently an enterprise solution for architect Curiosity Software. You haven't checked out our webinar with Curiosity software on model based testing. I'll have a link to it in the show notes. Don't want to check it out and you don't want to miss this episode.

Check it out.

This episode is brought to you by SmartBear. As businesses demand more and more from software, the jobs of development teams gets harder and harder. They're expected to deliver fast and flawlessly, but too often they're missing the vital context to connect each step of the process. That's how SmartBear helps. Wherever you are in the testing lifecycle, they have a tool to give you a new level of visibility and automation so you can unleash your development team's full awesomeness. They offer free trials for all the tools, no credit cards are required, and even back it up with a responsive, award winning support team show on your path to great software. Learn more@smartbeer.com today. Hey Rich, welcome back to the Guild.

Hey, thanks for having me, Joe. It's great to be here.

Great, I guess. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

I think you pretty much covered it. I've worked in test for far too many years and quite a lot of my kind of recent journey, especially in financial services, aligned to a lot of kind of transformations and therefore agile DevOps, a lot of those things kind of facing into those from a kind of a testing context and a broader kind of quality aspect. So yeah, let's talk.

All right. I guess right out of the bat I kept saying DevOps and it's a term I haven't heard before. So if you can explain a little bit, what is DevOps?

Probably not heard it because we made it up. It was a play on words. So within I worked in financial services, I led what tended to be the technical testing elements of testing within those large organizations. So DevOps actually had two meanings, okay. So the more PC meaning was that I led teams that was data automation, virtualized environments, operations, performance and security, DevOps, the other one. And it was really kind of an icebreaker in a lot of meetings, be that internal or external, which was within large financial services organizations, they're all going through transformations and they're all doing DevOps. And when you scratch the surface, what you find is you've got central DevOps teams that are kind of putting together tooling chains and not a lot else, all right? And therefore they've probably taken a lot of the processes they used to have, or they brought in a load of open source build tools and testing tools and things like that, and they're all clumping them together in a tooling chain and hey presto, they're kind of raising the we've succeeded in the DevOps kind of movement. And the reality is they'd happily kind of chug along for a couple of weeks or months and then complexity would bite, all right? And then their pipeline would come to a crushing halt. And you kind of get drafted in kind of as the kind of the technical testing element of the organization. It's like, hang on a minute, we want to do test automation. That's how they always get you in, right? And it's kind of, hang on a minute, your pipeline, you don't really know what you're putting into it and therefore it's rubbish in, rubbish out, right? And that if you, you know, think of it, you bend on a toilet, right? If you're chucking rubbish down that YouTube, it's going to become blocked, right? And so we get into a much more kind of fundamental conversation about what it means to face into complexity around organizations. And you kind of get into the kind of the three ways of DevOps, right? And not a lot of people actually talk about the three ways of DevOps. They just talk about these tooling chains. And so you got your flow, your feedback and your experimentation and you kind of go into flow and you start talking about, well, kind of explain to us what your flow is, right? And yeah, one of the subheadings, if you like, or the sub bullets of flow is teams shouldn't allow for local optimization. And you find that it's got within organizations Local optimization all over the place. Right. And they're all chucking rubbish over the fence to the next person and wondering why their pipelines are not working. Right. You know, this is where we would come in as Dave Ops as kind of this, you know, don't get me wrong, it's a play on words, a bit of a joke to then come in and kind of, hang on, take a step back, let's go a bit slower, work out what we're actually doing here, get some kind of a grounding in place and then we can start to automate that lean process once it's understood and once it's then executed. And that's not just a testing thing. You kind of get into the whole when does testing become quality? Kind of narrative and you get into the kind of, from an architectural perspective, for example, Right, Local optimizations. Actually we've got the monolithic systems here which we're trying to kind of crowbar through this pipeline and the two don't seem to go along. Right. And if you're starting to bring in a lot of new technologies, you know, a lot of these organizations will be doing microservices or they'll kind of have a solar architecture in the long term, or they'll be doing event driven architecture. Brilliant technology, but they've still got monoliths. Right. And yeah, hang on a minute, technology allows you to do things in a far more isolated way, but yet you're not doing them in practice because you haven't got the architectural disciplines to be able to make that technology work for you. And quite often we would find that testing as well. And the test approaches also were kind of agnostic or totally ignored the technology we're bringing in. And so quite a lot of test teams that were aligned to those DevOps teams. Yes, they had two testers, yes, they had SDETs, but they were doing a lot of the same tests that they would always do, you know, big end to end UI based tests and wondering why that's not working. Right. It's the hardest place to automate.

So grounding in place, getting the grounding in place. Are you seeing people just jump to like DevOps and they just try to ram things through without actually following what makes sense for their company?

Yeah, you've got this, I think, this interesting dynamic, right? You've got companies, you get this central DevOps team, right? And therefore they're building open source tooling to hopefully chain together these tool sets that other teams within the organization will use. Then you've got this other dynamic of Those other teams that got this kind of accountable freedom, if you like, or federation that they don't want to use, they're rejecting those tool sets and therefore they're starting to bring in their own chosen tool set. And therefore you've got this strange dynamic of tool proliferation. And you know, you get into the CI cd, right? There's an obvious tool chain kind of buzzword that goes about, and I remember having a conversation with an architect is to say, what is that team actually continuously integrating? Because yes, they've got lots of tools, but where are the boundaries in the system they're actually developing? And you kind of get a blank look all of the time. And therefore it comes back to this architectural kind of grounding is just not there in that, yes, you're using the technology, but you're just shoving the same old rubbish through new pipelines or new technology. And this is kind of the. You keep doing the same thing and expecting a different answer. Right. It's definition of insanity. And yet we do that a hell of a lot. And DevOps is kind of the next thing on the block. Agile before that, maybe. What came before that? Don't know. But we seem to keep repeating the same problem.

Absolutely. So I worked for a healthcare company and they had a very old application. So we're going to do cicd now. Even though we didn't release to the customer every now, we just released it because you had to be regulated. So it was every quarter or something. They still like, oh, we're cicd. So they try to force us to fit into a setting that our architecture, nothing changed, we just put in CICD and we try to ram it through. So is that what you're seeing a lot of companies doing?

Absolutely, to my point around, you know, people want to. They want to do DevOps and they want to raise that flag as quickly as they can because they brought in a hell of a lot of new tools. Right. And I've got very similar experiences of what you've. It sounds like you've got. Right. And that, you know, you've got a monolith there and, you know, you're carrying technical debt and your organization no doubt would have been quite siloed. And therefore Conway's law suggests that we don't really understand the interfaces particularly well and therefore, you know, we don't want to face into any of that stuff almost. We just want to shove what we're doing at the moment through this automated chain. And yet we were always struggling before that chain came along to actually automate it right. Or get a build or understand what versions we were trying to put in. So all of a sudden, because we've now got a name to badger up as, or we've got some open source tooling, what's going to be different? Not a lot. And this is where I come back to how many teams that are actually trying to do DevOps, whether you've got a central team or whether you've got these federated teams trying to do it, even appreciate those kind of fundamentals of DevOps, let alone kind of actually trying to face into them or what they've got today, the challenges they've got monolithic architecture, we don't really understand the interfaces flow, right. How are we going to unleash flow so we can overcome some of these challenges? And I don't think there is a lot of consideration in any of those things.

Yeah, once again, we didn't even have a build that was automated. We had to manually go in and put it together in the cicd. It was insanity what they were trying to do. So what would you recommend though? It almost sounds like you need to shift left some standards or set of things to be in place before you even get to that point. So what would those be?

So I think there is a fundamental problem around almost risk and transparency about what we do and don't understand about our systems today. And that includes kind of architecture, technical data around understanding interfaces. But then you've got a ways of working, right? And therefore how do you go slower to go faster? I think there is a very much a delivery or a feature factory mentality towards it today and therefore we've got to get the next thing out tomorrow or the company will die. Will it really? You know, but because of that attitude, we're almost paying a premium for those that problem. What we've got of trying to, you know, chuck a little bit more mud on that big ball of mud that we've got at the moment. And so for us, and you know this, I don't want it to sound like a sales pitch, but it's going to go into modeling a little bit, right? This is where we get into ways of working. And I know you had John Ferguson Smart on a couple of weeks ago talking about bdd, right? And he shared with me a white paper probably a couple of years ago now, which was called BDD at the heart of DevOps. All right. And fundamentally it talked about collaboration, right? And doing very early collaboration upfront to understand what we're trying to do. And for us this is what we started to do, but with modeling. All right, so where the kind of the BDD follows will go down the Gherkin route, we went down the visualization route in terms of creating models in our early conversations. All right. And actually in that white paper I mentioned, and I would encourage people, other people to go and look at it, there is a table that talks about almost maturity of the conversations around eliminating, I'll call it defects, but they were never defects because they found earlier on around getting consensus and eliminating kind of ambiguity in early conversations. And as you start to mature those conversations, things become a lot more clear, transparent, things become a lot more structured and robust around how things are supposed to work. Right. And it sounds very simple, but that coupled with the whole visualization of a model really works. Right. And I think come back to the complexity problem. There's a book that kind of percolates a lot around DevOps, which is teams to team topologies. Right. And a lot of that they talk about kind of platform team and they talk about how to structure teams in there. There's a section called Cognitive Load. And I think it's really interesting the way that if you imagine it, systems are very nature complex beasts, right? And therefore what mechanisms do we have to try and understand how those systems work? And they change, all right? And therefore our brains couldn't possibly cope with the kind of the change that's going on, the combinatorial explosions and all that kind of thing. And therefore, what living specifications or anything like that have we got to help us to kind of come over that cognitive load? The reality is a lot of teams are using kind of visio diagrams or word documents or head knowledge, right? There's a lot of old systems that have got 25 year SMEs that are kind of getting by on head knowledge. But when you start to have these structured conversations in early BDD conversations or visualizations, what you find is there's lots of gaps either in head knowledge and documentation, all of those things that today, if you're not doing any of them, they creep into development lifecycle or the testing life cycle, or worse still, you know, into production. And actually by having those structured conversations where you've got a living specification supporting you, you actually eliminate a hell of a lot of those things. And because you've got almost a lot of cognitive knowledge built into those visualizations, you find that actually your three amigos conversations, if that's what you're doing, or your feature mapping conversations, they don't need to go through a lot of the Fundamental stuff all the time. They. They can start to build on new conversations because, you know, everybody's already got this brilliant grounding through visualizations about what they're doing. And I think it's quite interesting. I always remember a conversation with. We talk about stakeholders and straight away we go to users quite a lot, I think, in that. But stakeholders have many. And you know, one of the kind of the areas that I managed was around security testing. And I remember sitting in a conversation with a security function that works like a function to us. And we started talking to them about this collaborative design that we wanted to do in visualizations and they remarked very quickly as to say, do you know what? We've never had this type of conversation before. We always kind of have to go back to the basics about how the system works. But because you've got all of these things, we don't need to do any of it. We know it's all there. And so we can start to talk about stuff that we want put in or stuff we know that you're doing these things. How does it actually work? So, you know, it's kind of a light bulb moment. And I think that happens a lot in terms of the kind of the collaborative sessions or the visualizations or maybe even, you know, you're following Gherkin in terms of. On the face of it, it sounds quite simple, but until you've done those things and you kind of lived it, you don't really get the kind of. The revolutionary kind of element to kind of the way of working.

Yeah. Because it almost sounds low tech. And when you think of DevOps, usually people don't talk about BDD or modeling. They think, oh, that's part of the development or the testing has nothing to do with me. But you just brought up two examples, DevOps and security, which I don't think most people think of conversations or BDD or modeling. Right. To be implemented there where it has benefits.

Yeah, absolutely. Just to elaborate on that a bit more. Right. Get onto platform as well. Right. And so we talk about the test pyramid quite a lot. Right. But I think, I almost don't think that goes far enough in terms of where does platform come into that. Right. And so compliance as code. And I think it's quite interesting, Right. We get into the stage of, you know, there's a lot of cloud, but whether it's cloud or whether it's kind of on prem infrastructure. It's infrastructure. It tends to be the same sort of stuff that you're either deploying on the cloud or you're deploying on prem and therefore how should you actually configure a Red Hat Linux server? Right? And you ask that question to a lot of people, they haven't got a clue, right? And they shouldn't. You know, that's the whole idea of the platform team is abstracted away from everybody else. But you ask the platform team, how should you build it, how should you configure it? Do they know? Well, chances are there's not, right? And there's not a great deal of standards built around it. Actually, there's an industry standard around how you should configure a hell of a lot of tooling that would tend to go in the DevOps chain. But the problem is, how do you take that? There's a hell of a configurations that go into configuring a Red Hat Linux server and therefore you get into the cognitive load problem, right? If you can get that into a model which then can generate a Python script to go and do compliance as code, you've got a conveyor belt there of platforms that you can just churn out like that. And actually you've got a brilliant kind of change mechanism as well, in that when Red Hat deliver the next version, I think they do it on quarterly basis. When they deliver the next one, there's going to be a standard that you can compare and contrast against, but you can also build that into your models very, very quickly, right? So that cognitive load and that cycle of change is fully managed in that flow process. And I think, you know, it sounds quite boring, but if you don't do that, how the hell are you going to keep that pipeline flowing?

Not even boring. It sounds like basic and not technical, but it's one of the hardest things to do, like the culture piece of it as it seems. So how do you get like buy in where like, hey, this is quality. Sure, it's DevOps, it's security, it's platform, but it's actually quality as well. I don't know, I find it hard. People are so siloed. How do you get everyone like, okay, actually this is part of quality. Let me jump in earlier than when it gets to me, near the end of the pipeline.

I think through experimentation, I think this is where. And I think there are many ways to do it. So the specific platform example is quite an interesting one, right? In terms of we couldn't convince the platform building adopt a standard, right? But we needed to test it, right? So we said, right, if you're not going to adopt a standard, we don't know what we're testing. Right. So we're going to adopt a standard in test. Right. And therefore, if we test it against the standard and you're not building against that standard, you're going to find problems. Right. And therefore it's back to this flow. Right. Or locum optimizations in that we're going to find stuff that's going to have to go up to the product owner to either accept or reject. And therefore you're creating this problem because we don't know necessarily what we're supposed to be testing unless we've got some kind of standard. And so we did that and we started, you know, it wasn't quite as kind of stick mentality as it may be, it sounds like, because very quickly we started executing things and it was automated. Right. Compliance code happens just like that. You just checking lots of configurations. Um, I think the actual benefit was where we could go back with a hell of a lot of intelligent questions about how something should be configured which they'd not thought about. Right. And therefore straight away, oh, I didn't even realize that was there. Let's. Yes, that sounds like a really good thing to start adopting. And it's like, look, yeah, we're already building these assets in test. Why don't you start to adopt them as well? Right? So there are other conversations where, you know, I'm thinking of another one where you've got upfront transparency about the risks or the technical debt that you've got. And I think another example that we've got was around implementing an event driven architecture with an organization using a DevOps delivery methodology. But it bolted onto what was a monolithic system at the back. Now everybody knew that. And so the conversations early doors got into, we're going to have a problem here because we know we've got this debt and therefore, Architect. Can we get into a place where we're talking very early doors about loose coupling? Because we want to get to a place where we can keep that loose coupling in place because if we don't, that new thing you're bringing in is just going to be monolithic and we're never going to be able to, you know, the whole kind of pipeline, the ferrum environments that we want to operate with is just not going to be a reality. So, you know, two very kind of polar opposites in terms of how we implemented a very similar approach through very different technologies. But it's kind of horses for courses, right? I think you need to use and abuse the kind of relationships you've got or the kind of the standards that are available that can. You're kind of your peers in kind of development or architecture are not kind of up for the best ways of working, if you like. There's always the catching them out in terms of. This is the bar that we're kind of testing against, but kind of falling below that. Absolutely.

So you talked a little bit about flow, you talked about experimentation. A lot of times I also see people resisting feedback for some reason. Like, this is our process, cicd. This is what the textbook says. I don't say anything about model or BDD or conversations we're going to, you know. So do you see that as a bottleneck as well, that stage?

Yeah, oh, absolutely. You know, I think I'm going to sweep in generalization here. Right. But we always used to work with quite a lot of partners and so we talk about bdd. BDD is synonymous with gherkin. Right. And therefore, to me, that's a misinterpretation of what BDD is. But you get the. Oh, we're doing this. We're gonna. We use BDD automation. All right. No, no, no, it's. That's not what it means. You're going to build up the same problem as you've always got. Right. And so do you want to have a conversation some of the time it's no, because this is kind of what we've got freedom to do. And you got to go and let them make the mistakes, to then go and then come in. This is where the DevOps bit came in. Let them make the mistakes to then come in and say, yeah, this is what you should have done, really. Right. This is how we get things going, this is how we increase flow. This is kind of the feedback you need. But I think it's interesting that you need the right culture within the organization to allow feedback to be productive, if you like, especially if you're working with partners. For example, within the organisation and the financial services organization I was working in, I was leading the community of practice for test and test engineering and actually we created three mechanisms to kind of increase collaboration and I'd say innovation. So, you know, to a certain extent you need to get a certain amount of momentum to kind of overcome that. It's diffusion of innovation, isn't it? The curve in that you get certain areas that kind of smash net, but then how do you make social proof? So we put kind of a cadence of sessions that we operated with, so one of them was a kind of a fortnightly show. And tell where we go and invite people to come along, whether they were doing it well or whether they had challenges, come along and present back. We had the organization strategy that talked a lot about kind of the collaboration, the model based testing. I'll talk a bit about kind of automation frameworks and things like that and freedom to use what you want to use and metrics that we would collect and things like that. But we invited them to come back and present around adherence to it, but also deviation from it. And so how did it work and why did it work when you deviated from it? Right, because we weren't kind of stuck in our ways in terms of how we wanted it to work. Actually. We wanted to keep evolving and be the best it could be because admittedly it shouldn't stay still. We should always be experimenting. That show and tell was called the Rebel Alliance. So again, there's a DevOps reference in there for anyone familiar with the Phoenix Project. That was what kind of the breakaway group in the Phoenix Project were called. And so I guess even in the name, the name was thought through in terms of being a bit rebellious to a certain extent, in terms of challenging kind of what the status quo was and you know, facing into some of the kind of the apathy that I think a lot of large organizations have. Again, we would have things like the Testing Times, which was kind of a newsletter that we'd send out. And again, it was all about them, the teams writing articles for us and you know, explaining what they were doing, but also raising their profile, encouraging people to speak out and be open about the challenges and what successes they were having. The last one was the Friday conversation, right. Which was within. We had kind of an internal Twitter kind of technology. And so it was a very open conversation that we would have around certain challenges that we knew we would have in the organization. So things like, for example, why don't people tend to care about non functional requirements? Right. And so then you'd have a very robust conversation about where functional requirements exists or what was the alternative for functional requirements or actually what was the mentality around even understanding the kind of the non functional elements of certain systems. So I guess we were cultivating a culture where you could have those kind of conversations. I think the thing is a tester almost, you've got to be brave, right. In terms of if you don't speak up to a certain extent, then as a tester who's there to challenge and give feedback, you're stuck. Right. Nothing's going to change.

Absolutely. So Rich, we Touched a lot on the culture, a lot of what people would consider soft skills. It is called the DevOps toolchain. You mentioned modeling. I don't know when you worked at other company AI machine learning was around, but I don't know how big it was. Do you see any type of tooling now helping with the DevOps process that you wish you had when you were in that role? Or you could see, is it growing? Is it just a buzzword still? Like it's still all goes to culture?

I think a lot of it is culture. I think the interesting thing, and I don't know whether you've seen any of my model GPT videos lately, but within Modeler we built something quite quickly which talks to AI and using generative AI to generate models. So, you know a lot of what I've talked about in terms of the collaborative conversation, you have the three amigos. Actually you could have the fourth amigo. With generative AI playing a role in terms of the brainstorming, I think one of the challenges we're talking to a lot of marketing euphoria I think is the term at the moment, isn't it around the possibilities of AI, right? And I think the problem that everybody will have is context and that ChatGPT can do some wicked things, right? But it's got the context of the world, which with the greatest will in the world, none of your organization's documents are not in the world. And Therefore how does ChatGPT or any generative AI or large language model get the context to be truly useful? And so we're doing a lot of work at the moment around how do you train up or fine tune language models. Now I always come back to the fundamentals, right, in terms of in order to train those things up in the right context, where you need a lot of domain knowledge that you trust and you know is unbiased and you know it's up to date, that's not going to give hallucinations, right? And so in an ideal world, the world will be modeled in Modeler, right? And therefore you would have a hell of a lot of living specifications that have got rules baked in and they go extractable from models that you could feed into those large language models. Otherwise you've got a problem that you've got a load of out of date documentation, you've got a load of out of date visio diagrams, you've got lots of things stuck in people's heads. How do you train it up to give it the context to be useful? You've Got some kind of declarative AI that all right, it does stuff, right? I guess there's, you know, you can find things on screens that have now changed, but, you know, is that truly groundbreaking? Actually, I think the kind of the moment, once we get over the how we give it context to kind of the domain of the organization and then we get into the cyclical cycle, right. In terms of almost once you give it that domain context and you keep feeding it that context in that cyclical kind of motion of change, continuous integration, each time that change happens, the thing should have more context. Right now, to me, there's an interesting dynamic going on here, right, in terms of how are you training it and how do you keep on training it? Because traditional ways. So if you're using video diagrams or word documents, unless you keep that document up to date, which you've never done in the past, that large language model is going to be out of date. And therefore you need to have that living specification as a core part of your kind of your changing living ecosystem. The definition of digital twin I think has been stolen for something else. Right. But I think it fits far better in terms of how do you keep this large language model up to date with the context that it needs to stay relevant, by which it can do generative AI, but it's still very aligned to the way the system should work or how the system should change.

Love it. Okay, Rich, before we go, is there one piece of actionable advice you can give to someone to help them with their DevOps efforts and what's the best way to find contact you or learn more about Curiosity?

So I'll recap on one of the bits that I mentioned earlier on, which is your DevOps test approach should really be looking at flow, feedback and experimentation. And so if you're still doing a test approach that you were probably doing 15 years ago, or you know, you've just adopted, you know, BDD Gherkin Selenium and doing UI based testing, only then you're probably doing it wrong. Right? Take a step back, have a look at flow, do a quick review of the flow that you've got in your team at the moment. So I don't mean test in that respect, I mean the whole flow. You've probably got some knowledge gaps in there which you really need to address. Otherwise you're going to kind of keep living with the problem and the problem's only going to get bigger the more you leave it. You need to make the change now. The longer you put it off, the harder it's going to be. It's going to get harder in terms of contacting us. We post a lot of content on LinkedIn and social media all the time. We've already always got videos coming out and say there's a video around model GPT which is kind of the latest thing we're exploring at the moment. But there's a hell of a lot of videos in terms of some of the content I've been talking about today, blogs and things like that. So lots of content. If you want to reach out, feel free. Hopefully I'm not that scary and I'll.

Be honest.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com forward/p119 and while you're there, make sure to click on the Smart Bear link and learn all about Smart Bears awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam attest guild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to TestGuild Info and let's make it happen.