Cloud Visibility for QA and DevOps with Greg Arnette

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Let me ask you, who manages the configurations for all your various test environments and other environments. When your teams don't have a good strategy, the resulting misconfiguration really can slow down the release process. And I've seen this cause a lot of things like downtime and a lot of teams getting frustrated. So what do you do? That's what this episode is all about. Hey, I'm Joe and today we'll be talking with Greg all about cloud visibility for QA and DevOps and a whole bunch more if you don't know. Greg is the co founder and CPO of Cloud Truth and Prior he was the founder and CTO of three cloud SaaS companies in the email and data production market. Greg has created cloud SaaS solutions for over a decade and is the author of the patent, method and systems for arbitraging computer resources in a cloud computer environment. So really knows his stuff. You don't wanna miss this. Check it out. Hey Greg, welcome to the Guild.

Thanks Joe. Great to be here. Hey everyone, awesome to have you.

So I guess before we get into it, you know we mentioned in the pre show something about cloud visibility, why it's important. Maybe you can give a little insight into why you think cloud visibility is so important nowadays.

Yeah, this is an important topic that all teams are facing right now and it impacts on several really defining layers in terms of this overall cloud visibility into the number of assets that you've provisioned from your cloud provider, which then in turn becomes your cost basis for using the cloud. It's your surface area for security and reliability. And behind all these assets are configuration parameters, variables and secrets to make sure they're all working properly across development, staging and production environments. So visibility into all these moving parts is a requirement. It's a challenge. It's where the DevOps tooling market is headed. And it's this combination of internally created tools by DevOps and platform engineers, as well as what third party vendors are bringing in a form of innovation into the space.

Nice. Can you talk a little bit more about the cost component visibility, why it's important? I've seen some articles recently where they said companies have moved to the cloud and then because of the cost they moved to in house Again is that to not maybe having visibility into the cost.

It's a combination. So there was a roughly around, you know, between 2007 and 2011, 2012, there was a huge initial rush into the cloud and cloud users tried to replicate what they were doing in their own on prem data centers like cookie cutter, what an architecture looked like in their say VMware system or Dell based or Equinox based system and replicate that into like an AWS or Azure environment. That's the worst case scenario. You're going to have spiraling cloud costs because you build out systems differently if they're cloud native versus on prem. Starting around 2012 and to where we are now, it's all about cloud native born on the cloud. Nothing's being designed that was going from say initially from the get go from an on prem system to a cloud, it's getting designed cloud native. So we need to be thinking differently about cost aware architectures in the cloud so that you're getting the benefit of why you went to the cloud in the first place, which is you don't want to pay for resources that you're not using. You don't want a bunch of pre provisioned compute standing at idle ready to take on a job if there's no job to be happened, if it's happening at midnight. And you also want to be able to burst up really quickly should you experience a very positive, a certain surge of usage, a lot more customers signing up or mobile users taking more pictures or whatever kind of app or system you're developing. So the cloud brings to get together on demand cost aware architectures. But if you don't pay attention to the signals, you're going to end up paying more money for your cloud bill than if you just stuck with an on prem solution. So the cloud allows you to be very flexible, experimental, but you have to really be aware of what your every line of code you're writing almost can be tied to how, how much that piece of code is going to cost you to run it in the cloud these days. So that kind of cost aware engineering is something new.

Absolutely. So who's responsible for this? It seems like a lot of times, is it the DevOps team, is it the QA, is it the developers? Is that one of the things that make it so maybe people aren't paying attention because no one knows who's responsible for the cloud piece or the cloud visibility piece.

Yeah, it's definitely a shared responsibility. Increasingly there's been this shift in the industry towards a new discipline called finops so everything today is AN Ops. It's DevOps, it's GitOps, it's DevSecOps. FinOps is the latest kind of incarnation of how teams deciding to manage their expense portfolio in the cloud. And it's a new discipline. It's carving together kind of like a slice of DevOps and a slice of like the CFO's group. More and more companies, as they get larger, are putting cloud cost analysis into the hands of the CFO or a business operations leader who really cares about tying in cloud expense with what it cost you to run, a customer, a tenant in your system, and also how it affects the bottom line. So it's this new hybrid approach that's taking effect right now?

Absolutely. I guess another thing I see people sometimes get hung up on is they don't know where to go to get all the information. They have all these platforms and you know, all these dashboards. So how do you address that for visibility to make sure maybe your CFO is seeing what he needs to see and be alerted when he needs to be alerted.

Yeah. So working backwards from say the CFO needs to be the recipient of a synthesized analysis that can be in sort of business friendly, not techno jargon understanding of how cloud costs are being managed. There is a typically a complicated process behind the scenes that's an aggregator of aggregators. So you might have a bunch of different dashboards that are helping the ops team run the system more efficiently. And then you have dashboards that help the revenue side of the organization understand how customers are growing. And marrying that together is what's the challenge. So you start to see tools like the datadogs and the aggregators of aggregators being able to synthesize different streams of data into a format that can be consumed by both non technical and technical people alike.

Awesome. So besides CFOs, I think you mentioned once again it's important for QA folks to be aware or have visibility into the cloud. So I'm just curious to know your reasoning for that. Why would you think that'd be important for someone that maybe has a QA or a testing role?

Yeah, and that kind of takes the conversation from thinking about looking at cost to configuration parameters, variables and secrets which are at the heart of doing everything well in the cloud. So every asset you have in the cloud, every piece of software that you're writing, every third party service that you're using has configuration behind it. And that configuration has to be the configuration for the development environment, the staging and test environment, as well as the Production environments. So application developers are giving tools that are typically called platform engineering tools that make their life easier. So an application developer who's writing some new features or creating a. Net new app has to specify the configurations for those that new feature, has to be able to communicate it successfully to the test QA stage as part of the SDLC process, and then to the OPS team that's going to operationalize that code. So it's that middle step that we see as kind of being underserved right now. Testing QA engineers are kind of borrowing from application developer tools and borrowing from DevOps tools to kind of get the information flow that they need to do their job well. And so QA and test engineers are managing a hybrid effect. They're taking software that was developed in a dev environment by an application developer. They're trying to simulate as best as possible the production environment that that software then will be living in, because they want to test it as close as they can to a production real world scenario to get an accurate result. But yet they can't actually simulate a full production environment because that would be too expensive. Production environments have a lot of provisioned resources for redundancy and reliability and the security posture. And it's not cost effective to keep a mirror image of production running all the time for test. So QA and test engineers need to be very adept at, call it spinning up environments that look like production, doing their tests and turning them off as quickly as possible to save money, and then spin them up again and do their test. So it's a delicate balance between an affordable environment that works for everybody as well as adequate test coverage for how that piece of software will work in a production environment. Because you don't want to get surprised when your test coverage and staging didn't find the bug that occurred in production because production was configured slightly different than staging, and that bug happened to exercise that difference.

Well, that's interesting because it's a container and you have config files that probably YAML files that spin these things up. If it's slightly different a parameter, like you said, if it's in staging, if it's not exactly the same, you may get different behavior. Especially I could see that like with performance being an issue, for sure.

Performance, that's where we see this a lot, right? Sizing the number of nodes in a cluster, for example, to handle the expected traffic load. Right. Sizing the number of cores in the memory in the database to handle the number of simultaneous connections that production sees. But maybe the staging environment doesn't see unless you can do an artificial load assessment of burstable traffic activity. So that's where QA and test groups need their own set of tooling that works alongside of the existing DevOps tool stack and often could be the same tool but just used by a different set of use cases but have an appropriate accurate way of doing it.

Awesome. So you also mentioned security. So secrets obviously if a secret gets into GitHub or something, that's a problem. So are there other security issues someone needs to be aware of that they may not have visibility into right now with their cloud environments?

Yeah, well certainly leaked secrets remediation is important and whether it's application developer incorrectly put a secret in a in a git repo and maybe it's even an open repo that forces a fire drill of rotating that secret because you just don't know who saw it, who got a copy of it. Also, leaking into log files is really a no no these days, and that's an area where both application and QA and test needs to be vigilant not to log a secret or redact it if it gets done or have a mask that puts all asterisks in its place. That's be more thoughtful. There's also a looming change in the industry that's being spearheaded by Google, which is going to change default lifespan of TLS and SSL certs. Now it's roughly currently it's roughly a little over a year and they're moving to 30 day or 90 day lifespans, which then will force all the consumers of certs, which is everybody, to have an easy way of changing those certs that affect the stability and security posture of a whole stack reliably, you know, more quickly than they have been accustomed to in the past.

That's interesting. Especially if you have an automated test and it expects the browser to be in a certain state.

Yes.

And you have to update hundreds and thousands of virtual machines. This is interesting.

Yes.

Great.

So it's going to put more pressure on teams, especially in the QA and test realm, to have the facility to duplicate that because you have to make sure the code works when your certs are being rotated every 30 days or 90 days versus once a year. Once a year it's kind of more ceremonious. But if it's happening quarterly, that's a lot of moving parts that have to be orchestrated carefully. It's in the same way as if you're going to change your database password. You have to make sure that the old password gets Cycled off, the new password comes online, all the clients are going to talk to that database, know the new password. So there's a, there's a bunch of patterns like blue Cream deployment patterns and so forth, which talk about this kind of concept as an abstract, but in terms of actually how you orchestrate the delicate balance of changing the server than changing the clients in the right order. You could have a period of time where clients can't talk to the database because they don't know the current password anymore, they weren't updated, that it's new versus the old one they have in their current config settings.

All right, so I think we covered a lot of pain points teams are probably facing. Obviously you created a solution cloudtruth. Maybe you could talk a little bit about what is cloudtruth and how does it help with some of these pain points we talked about.

Sure, yeah, thanks for asking about that. The genesis of cloudtruth is one of these examples of a solution designed to solve pain point that myself, my other co founder, had. In our real prior jobs of being SaaS and CTO leaders, we kept seeing that our DevOps teams were building these internal tools to manage what we call today config sprawl and secret sprawl. And it's a way of kind of putting the horse back in the barn, so to speak, around the fact that many teams start off with a very simple approach to managing their config and secrets. They typically stuff them in JSON and YAML files and git repositories and use a secret store either provided by the cloud vendor or say something like hashi vault from open source. And as their technology stacks scale, their business thrives. They start to incur very kind of incrementally this idea of config sprawl and secret sprawl to the point where it gets it becomes a tipping point and is now seen as an impotent impedance to fast release cycles. It's too much paper cuts on the DevOps release process and that's what we're here to file down. So Cloud Truth is a config ops platform that's designed to work alongside of GitOps and Kubernetes and infrastructure as code, so it doesn't replace any tools that you're already using right now, except maybe a homegrown version of something that we do versus some net new functionality that plugs in alongside your continuous deployment process and then becomes a single aggregated record of configuration and secret truth and makes it easy to inject configuration Everywhere it's needed. So if you're going to use Terraform to provision your cloud, we become the system that manages all your terraform variables across multiple environments and projects. If we're using kubernetes, we're the system that manages all your config maps and secrets and helm charts from a central authority, giving you role based access control, change management, the ability to compare settings across multiple environments. So it brings together a bunch of really wish list items that DevOps teams have been craving for the past five years that we've decided to launch as a separate product in its own.

All right, so I'm trying to visualize this. Help me out. So you have all these systems, does this sit on top of it then? And then when people see I'm a QA tester, I would log into cloudtruth and then cloudtruth would then have all the different areas where I could set configuration once and it would go out to all the systems and update them.

Yeah. So Cloudtruth is a robust platform. It's available as SaaS or self hosted and provides an API, a CLI, a web app, and then many configuration clients that can be used like a Terraform provider, a Kubernetes operator, GitHub Actions plugin. So wherever you need to consume config, Cloudtruth can be that system that injects it. And that's the idea of perfect config for every deploy, so that you're not focused on chasing down configuration faults as you're trying to get new releases out the door. So there's a web application that becomes your visual configuration designer. If you're familiar with tools like figma, which let you rapidly develop new user interfaces. We're like a figma for config. You can design your config in a graphical way using our tool set. So you have a hierarchy of your environments. You have your dev environments, your staging environments, your production environments. You can manage that hierarchy. You can set default values and then those values can be inherited further down in the tree. You can have overrides, you can manage ephemeral dev environments that are constantly being launched and torn down and launched and torn down. All the configuration settings behind all that become managed by our system. And then in terms of application config and microservices config, where the system will manage all the configuration for your, the software that your teams are writing, or your third party services, or your platform services, or your cloud pieces, all that has config as a requirement behind the scenes. So we become that central management layer, and that's what we call it, like a configuration command center with a very robust dashboard and visual layer, and then a CLI that allows you to take the concepts of these platforms and inject it into kind of any kind of workflow that you have, either through a command line interface or an IDE plugin or one of the clients that we provide or SDKs that we provide.

All right, So I would think. Well, I know it's probably overlooked a lot from teams, so it almost sounds like this would help with communication as well between the different teams within when they're creating software to say, hey, here's the config we have. Is this correct? And they can go to one place to go. Oh, no, that's not correct. This is going to cause an issue. So it's almost shifting it left before you get to an issue in production. Am I understanding that correctly as well as a benefit?

Yes, you're spot on this. It's a big source of release friction right now is this handoff from application developers to QA testers to operations on what are the right config settings for every module in the system or every component in your system. And they're going to be different across all these different environments. And then if you layer in a third dimension, such as tenants and customer configs, where you might have a certain customer in your production environment that's entitled to a certain set of functionalities, that's another layer of abstraction that we help handle as well. So it really makes the communications channel, or you don't need a communications channel anymore. This system becomes the communications channel. So it all gets correctly configured from every phase in the deployment cycle.

So how much effort is done from the, like the tester or the developer part, does it bubble up insights or is it all just the team deciding what's right and what's wrong with the configs? Is there any type of, I don't know, any type of analysis it does on its own to bubble up, hey, this config files looks a little funky based on our best practices or anything like that.

Yeah. No, you can implement your best practices as a set of rules in the system. So we call those guardrails. So every parameter, value or variable can have a guardrail, such as what type should it be? An integer, a string, a boolean. And then further, are there any constraints on what the value can be? So for example, you might have a parameter that's going to set the number of nodes in a cluster and there's A cluster that's going to be for production, a cluster for staging, and a cluster for development. In development it might be a single node cluster or two nodes because you just need something minimal to program against. In staging. You want that to look a lot like production, but without incurring the expense. So we can manage, say that's a very discrete parameter variable. We can manage how easy it is to flow between the environments and we can put guardrails. So developer environments can't be more than two nodes. Staging can be between 2 and 10 nodes. Production can be between 2 and 20 nodes, for example. So if you can't color outside the lines, you can implement your business logic and your technical logic as a set of rules and guardrails inside the platform. And then we'll assume you're supporting OPA open Policy Agent rules as well, which bring in a wealth of kind of external learnings that we can apply. So this becomes an expert system. You can set appropriate default values, ranges and very sophisticated rules on how you want to manage all this at scale.

Nice. How easy is it to integrate or implement the current system? Like you said, as long as it's not like a homegrown system. Is it like an API that people would just plug into?

And yeah, there's a number of integration points and that was designed from the get go to realize that Cloud truth will be installed in existing live environments. It's very rare. You get like a total greenfield experience these days to start off something new. So we know we're coming into the market, that there's existing workflows that are typically built around Bash or Python or Go code and scripts that are linking together different processes. We can interface at the right point, usually at the command line layer if you're chaining together a bunch of scripts in terms of the ability around the integration, as easy as possible. So typically we see customers adopting this project by project to kind of do it wholesale would be call it a boil the Ocean exercise. It's just not practical these days. So we sit alongside your tools that you might be using for cicd, for infrastructure as code, for container orchestration and for like the dev test stack kinds of tools. So we don't replace anything. So we provide automatic importers, we can scan your git repos, we can import all the config, we can integrate with services like AWS Secret Manager and Param Store and azure key vault, HashiCorp vault. So we can reference all that and make it immediately useful. So project by project, you can retrofit the config strategy in a way that makes sense for how your team is designed to work.

Nice. So I always found as an engineer, it's always hard to convince management you need a certain tool. So I think I was reading an article that you pointed me to that talks about how you can actually measure this by you can measure certain key virtual performance metrics like lead time and mean time to restore. How does that help? And is that something you can say, hey, look, if we use this, this will help us with our me time to restore by 10%.

Whatever. That's a great topic to kind of click into. So we're seeing that as teams are being asked to be accountable to delivering software new features for their respective customers and more so than they have in the past. And there's just a mandate right now to get more stuff done and to be more as efficient as possible given the economic climate that we're in. So these new kind of door metrics are taking hold and they're pretty simple to understand. It's like, what's your deploy frequency? What's your change failure rate? What's your mean time to recovery? If you peel away the layers behind all these important kind of goals that teams are striving to improve on, what you find is that there's a common horizontal layer that's how are you managing all the config and secrets for all your projects if you have an inadequate strategy that's going to affect your core DORA metrics. So we have testimonials from our customers that tell us that they have gone from release cycles that would take months to get new features out in front of their customers down to weeks. And they attribute that to the adoption of our philosophy on how to manage config and secrets at scale, which we've codified as a kind of a manifesto called the seven factor Config Principles, which is an idea of there's seven important factors that teams can adopt. They can use either cloud truth or build their own system. But if you adopt these seven core factors, you are going to guaranteed to see an increase in your release velocity and a decrease in downtime and a decrease in security issues.

All right, I mean, to put you on the spot, I don't have them written down. What are the seven factors? Really quick.

Let's see if I can remember them. I might have to look at a quick cheat sheet here. But they start off with the idea you want to keep your config dry. And that's the first principle. Keep your config dry, decouple and externalize your config from source, because we see that's one of the bad practices that teams are trying to work away from is, you know, nothing hard coded these days and you want to abstract your config. Like the idea of like, what's the log level from this service versus how do I fetch the value for the log level. The other, the fourth principle is centralized. That you want to centralize as much of this as possible. And centralized doesn't mean actually to put all your eggs in one basket. But you want centralized access, you want a single API call to fetch the config for a particular component. And that config data for that component could be spread across a couple of different interfaces. But you don't need to care about that as the consumer. You just want to be able to make one simple API call or CLI call. The fifth principle is keeping your config obvious and well understood. So right now many teams are suffering from the fact that your config and secret strategy kind of is a black box to almost everyone on the team, except for just a handful of people in the ops group which came up with this kind of initial structure and they've been kind of just kind of kicking the can down the road to evolve it more methodically. The sixth principle is keeping your config secure. So that's a no brainer for secrets. You mean secrets need to be encrypted. The crown jewels of like access to your database and your file systems and all the important interfaces are managing sensitive data. So having a very thoughtful strategy on how to keep your secret secure. But even your non secret config data should be kept secure. And just keeping it kind of in a plain JSON or YAML file could be a way that a hacker could figure out you're doing something and give them the attack vector that they need to understand. So even non secret data should be kept secure. And then finally, the seventh principle is versioning. You need to be keeping track of the changes of these values more so than just keep relying upon change tracking in your git repos. So a dedicated system that starts to look like a configuration management database or cmdb, which is a term from kind of the old school legacy it where cmdbs were popular for a while. There really hasn't been a modern kind of cloud based reinvention of CMDBS yet. And we're starting to see whether what we're working on might be going down that path in the future.

So Greg, could you just tell us a moment Little bit more about the free plan. What does it include? What can people get with that? Is it a good gateway to get them to know, okay, this is something we definitely need. Let us buy the more advanced paid option. Yeah.

The free plan gives you access to all the important capabilities that we provide in the platform. How we separate it from the paid plans is if you want to add a lot more team members or a lot more projects or a lot more environments. So it's designed to be the equivalent of us. We don't have an open source version. So our Free Forever plan is the way that hobbyists can get started. We also have special pricing for nonprofit and early stage startups. They can get access to the platform without being in a budget friendly way. So you can get a taste of everything in the free edition. It's when you want to scale or you want more of an enterprise support system or if you want to run the software in your own environment, we have price plans for those.

Okay, Greg, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps Cloud efforts and what's the best way to find contact you or learn more about cloud truth?

Yeah, the one piece of advice I would share is that from what we've seen after talking to thousands of DevOps leaders over the past few years as we've launched this project, is that there's a notion out there that people think their config and secret strategy is relatively simple. It's just flat files and repos and you marry in a secret store and that's all you have to think about. But the reality is that simple notion masks is a facade for something that's really complex, that touches multiple team members and it's seen as this kind of tax on release velocity. So I would encourage everyone to kind of take a step back, really think about how you're managing your secrets and your config at scale and take a hard look at all these different kind of scripts the that no one really wants to talk about anymore in your DevOps tool stack that become these weak links in terms of a reliable production deployment system that just operates smoothly. And to get more information around Cloudtruth, go to cloudtruth.com and there's a bunch of videos and documentation. There's a free Forever edition and a couple different price plans based on the kind of type of size and organization you are.

And for links of everything of value we covered in this DevOps Toolchain Show. Head on over to testguild.com p136 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.