Centralized Secrets Management Without the Chaos with Brian Vallelunga

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, welcome to another episode of the Tuskill DevOps toolchain. Today we'll be talking with Brian all about centralized secret management without the chaos and more. He's a founder, so he has a lot of experience on a lot of different things. Really excited about this episode. If you don't know, Brian is the founder and CEO of Doppler, which is the first secrets management platform for developers. Brian also has been featured in the Forbes 30 under 30 and worked at improving overall safety at Uber. So really excited to dive into that a little as well. You don't want to miss this episode. Check it out. Hey Brian, welcome to the Guild.

Hey, pleasure to be here. So happy, awesome to have you.

I guess before we get into it, it's quite a background you have. So I thought maybe you can maybe give the journey from Uber to founding your company. Like, what was the aha moment that led you to creating your company?

Yeah, happy to. So it all started. I was an intern at Uber and I had a very crazy career at Uber. I don't know if most people know about Chris Sacca, but he was an angel investor or is an angel investor and he had a crazy career at Google. And I like learning from these kind of guys. And back in the day with, with Chrisaka, he would just go into meetings that he wasn't invited into. They had an open door policy at Google and he quickly rose through the ranks that way. And as an intern at Uber, I very much did the same thing. I would just like go to the back of the room where the CEO and the VPs are in their meetings and I would just attend and listen. And eventually I started asking for my feedback and throwing projects my way. And while I was kind of going through all of that, at some point engineers started working for me as an intern or reporting to me. And as you can imagine, full time software engineers reporting to an intern wasn't a very popular thing. So they took that to my manager and gave my manager and my manager gave me an ultimatum. They said, you can or he said, you can stay at Uber and go full time or you can go back to college, finish your degree and then afterwards you'll have a job waiting for you. And I was like oh, I'm. I'm quitting school today. And after that I kept doing some more special products for Uber. And along the way I was also working on my own side project. It was a crypto machine learning marketplace, kind of like all the buzzwords in one. And it was one of those projects that very much felt like pushing a boulder up a hill. You move one foot forward, you slip five feet back from exhaustion. It's really hard to make any progress on it. And I'm about eight months in, really deeply frustrated at this point. And I decided to take a trip to Mexico in Cancun. And the one rule was like, I am not going to think about this project. I'm going to take a break from it all and just reset. And the first day there, I'm in a hot tub and my mind is racing about this company. So clearly did not honor my word about completely taking a break. And what I realized throughout that was that I wasn't going to be the one that gets this off the ground. There are just some companies that you're just not meant to build, and this one wasn't. But inspired by another founder, Stuart Butterfield, who I think is one of the best in the industry at failing upwards. A good example, he creates a video game out of that. That video game fails, but born out of that was Flickr. And then goes back and he's like, I'm going to do another video game, this time for real. And he creates another video game that fails, but born out of that was Slack and everyone uses Slack. And so kind of inspired by the same thing, I was like, what can I learn from my experience here and what can be born out of this failure? And not to say the failures are bad thing. I actually think failures are some of the best lessons you can have. And out of that was managing environment variables and secrets. And for anyone who doesn't know, an environment variable or secret is a piece of text that is used to configure your application or authenticate you to another service. So, classic example, you're on Netflix and the Netflix code needs to talk to the database that has all the movies in it. And the way that the code connects to the database is through a database URL. And that database URL says this is where you go to talk to the database and here are the username and password or the credentials to get access in. And so secrets are obviously super sensitive things. You don't want them out in the wild because if you do, then anyone has access to those underlying services they're kind of like the keys to the digital kingdom. And I was having a lot of problems managing these secrets in a lot of different ways at the time. I had a contractor and it felt very weird having a contractor have access to, like these hypersensitive credentials, especially knowing that they would eventually leave at some point and could take them with them. We also at one point dropped the crypto machine learning part of the crypto machine learning marketplace and was just a regular machine learning marketplace. And we had the Stripe production key and staging the staging one in prod. And so we were down not doing any transactions for about 30 days before we realized that was happening. And we were like, oh, why are we doing no transactions for a month? And then we eventually looked at the secrets and we're like, ah, that's why that sucks. And so coming back from Mexico, I go to this dinner called the Stripe Atlas Dinners. And it was a set of dinners hosted by Stripe, where a bunch of founders and developers would go to talk and chat about the industry. And at that dinner, I kind of stand up and I'm like, hey, I'm seeing this problem, I'm struggling managing secrets, and I need to know, am I a shitty engineer or is the world broken? You tell me. And a lot of people were having issues with managing secrets. And one woman in particular comes running up to me right afterwards and I thought she was going to trample me and she said, have me an outage by Sunday. And I was like, no, no, no, this is a Wednesday. That's not possible. And she goes, I don't give a shit. Have it by Sunday. And I was like, okay, you have a problem and that problem is real. And we did not have the solution by Sunday. But what we did do is we did some research and we found that individual developers were struggling, mid markets were struggling and very large enterprises were struggling. And they were all struggling for the same reason. And that's because the tools were not built for developers. And this is a tool that is often used by developers. Most of the time, the end user is the developer. And so this is something that we thought that we were uniquely qualified to solve. We had a little bit of a security background, we had a developer background, and we really understood how developers wanted to engage with tools. And so we built the first version in three weeks. The fourth week we got some customers through a method I call Chipotle sales, which we can talk about in a bit if you want. And then we got into yc, we raised a round from Sequoia and Now we're off to the rate, and then we did a round by Google, another round by crv, and we're off to the races now.

All right, all right. So you're a young guy, I'm an old guy. How do you get the, I don't know, the right word, politically correct word to say cojones to do this? How do you know to move forward without being scared? I guess. I guess a lot of people are scared to do things or to fail. So how did you get to this point?

Well, I have a lot of failures under my belt. Failures are very powerful lessons. So Doppler started number eight for me. And I think the thing that's always driven me to do this and build a company is that this fundamental principle of, like, you just need one to work, you can take 100 at bats, but if you get one success, you win. And like, you're going to have all the success you want in life, at least from a financial perspective, from a confidence perspective, from an industry impact perspective. And I also think that for the most part, people are pretty bad at estimating risk. And I think the risk of starting a company is quite low for the most part. If you do it in a venture capital way, venture capital away means you're not putting your money up front. Other investors are putting money in and buying equity stakes in your company. And if you look at that from that lens, well, you're not in any financial risk. You're probably actually going to end up ahead because you're going to have a far bigger network of people at the end of this, even if it fails or most likely when it fails, because you'll have recruited people, you'll have recruited investors, you'll have built hype, which means you now press contacts. You'll know a lot more about how companies work, which will inform how you think about finances, how you think about the industry, how you think about making critical decisions. You will probably, at some point, if you do like a Series A, do some secondary, so you'll pull some cash out on the side, so you have a couple hundred thousand to a million dollars. Like there's, if you get just any level of progress in the game, you'll be orders of magnitude better than if you didn't play the game at all. And so I think the risk is quite low. Like a lot of times they see it as like, oh, if you don't go ipo, it's a failure. And that may be the case for investors, but for the founder, that is not the case at all just by playing the game. Most of the time, most founders will end up ahead.

All right, so I always ask, I think that's when do you know, like, to play small ball. Like, I would be content with my own company solo, solo founder. I'm making money and I have money coming in every month. And, like, that's it. I don't need outside investors. Like, how do you know? Like, why I always curse to know when I meet someone like you? Like, why. Why did you choose the other way?

For me, I can't speak for everyone else, but for me it was a calling. Like, I've. I think if you, like, went and interviewed my parents, they would say, Brian wants to build a company. Like, since he was like, two or three, like, probably some of the first words that came out of my mouth were this. And I know a lot of founders say that as, like, a joke, but I mean that, like, literally. Like, I built my first company when. When I was in middle school, no, elementary school, as a way to. It was a gift wrapping shop. I found that a lot of people buy video games right before Christmas. And so I convinced my parents to build a gift wrapping. Help me build a gift wrapping stand right in front of a popular video game store. When people bought video games in person. And I was like, in an elementary school, but I saw a market opportunity there. I was like, that can pay for a trip to Australia. And it did. I went to Australia, New Zealand from it. So I've always been obsessed with building something highly impactful. And I think it really comes down to, I think most companies are going to be hard no matter what, right? Like, it's going to be painful. It's like the classic line from Elon Musk is like eating glass and staring into the abyss. And it's absolutely true, right? There is no handbook here. You have to find out how to be successful. You have to figure that out. You have to deal with that through a lot of failures. Like, failure is the best lesson or the best teacher. And so if that's going to be the case for both a small company and a big company now, some small companies have more proven track records, right? Like, if you're building a restaurant, like a franchise, it's a pretty easy playbook, but there's still going to be challenges there. You're going to have to learn how to manage people. You're going to have to learn how to recruit. There's a lot of things you're going to still have to solve. And so if it's going to Be hard no matter what. Might as well go for the biggest impact you can have on society and for yourself. And that's always been. My viewpoint is like, I want to maximize the impact I can have.

Love it. So I know a lot of developers are listening to this. They probably have a lot of ideas. You said this was your eighth go at something that seems to be really sticky now. So for this one, do you do experiments? Do you, like, throw up a page and see if people sign up? Or like you said, you try to get your first customer and then just build off that. Like, how do you know once you have that first customer, this is definitely something that's needed at a larger scale?

Yeah, great question. So we didn't do any of the throw up a landing page thing, actually. If anything, we didn't have a landing page for like the first, like, year and a half. We just had like a login form and I would just chat with people one on one. But I think the underlying pinning of that is do your market research. And so for that, I. We, like, rigorously chatted with anyone we could possibly get our hands on, like individual developers, hobbyists, large corporations. Like, we chatted with a lot of engineers at Airbnb, at Pagerduty, and so, and at Uber. And so we just try to figure out, where is this problem? How important is this problem to them? Is it going to get prioritized or not? Like, the people that are going to be buying whatever tool that you're selling probably have a list of things that they're responsible for in a given year, and they have a priority list. And if it's not, if you're not selling the product that's at the top of the problem list that they have to solve, then you're not going to. No matter how good your tool is, it's not going to get traction. And so we really tried to answer those questions as early as we could, and that's largely how I would even do it today. I would just have a lot of conversations with people and really probe deep. I think there's a good book called the Mom Test that can help get you started in the way to think about things. Like, a good example is it's very easy for someone to say, hey, I would use your tool. It's much harder for them to say, yeah, I'll give you my credit card. And whenever you have the tool live, just go ahead and charge it. Right? If someone's willing to do that, it's probably solving a real pain point for them. Another way to think about it is vitamins versus painkillers. If you have a vitamin, I can skip my vitamins every day. But if I have pain in my back or I'm on the deathbed about to die from cancer and someone says, hey, it's a million dollars for this cure for cancer, I will sure as shit find the way to get that cure right. I will be deeply, deeply motivated to get that cure. And ideally, the best products are cures to some of the hardest problems your customers are facing.

Love it. And what I love about your idea that what your space you're in is regardless of the economy, people are always going to be concerned about secrets, I would think.

Yeah, I would say so. Now I think there are. And partly one of the reasons why I'm on this show today is today I'm here as a psa. Like we are starting to see a lot of people wake up to the real issues when you don't manage secrets. But I think a lot more people need to know. Like secrets have been one of those things for a very long time where people undervalued them and so they just kind of like mismanaged them and it felt very risk free to do so. And what we're seeing now, with the rise of really sophisticated hackers that are very capable of hacking most institutions, small companies, large companies, it doesn't really matter that secrets are becoming ever more important because at the end day, like the game of defense and the game offense are very different. Different, right. Like as a defender, your job is to build your walls super freaking high and make sure that there's no cracks in the walls, right? Nothing. No no chinks in the armor. As an offensive player, like an attacker, a hacker, you are really looking for that one chink in the armor, that one crack in the wall that can bring the whole thing down. And if secrets the literal keys to the digital kingdom that every company has, like every company. And it's not one secret that most companies have. Most of the companies that we work with have thousands of secrets, right? So like a lot of different chinks. And if one gets out, it ripples out to the rest. Microsoft's a great example of this, right? Their email system got hacked and now Russia is deeply is deep in Microsoft infrastructure and they still reported that, at least the last article that I read said that Microsoft has admitted that they can't get these Russian attackers out of some, some critical sets of infrastructure. And so one, it's a domino effect. If you can get to one secret, you can Most likely get to all the others, especially if you get like an AWS credential. And so I think people really need to be managing secrets correctly now. I think the risks are really, really high. It's not just high for the business of like, oh, there was this data breach and now the company's going to get slapped with a bunch of fines at the end day. Most companies on this planet are here to service customers. And so that means that the data that they're protecting is not just like this arbitrary terabytes of data, it's real people's data. Right? And so when a hack happens, even though it looks really shitty for the company, the customer, the end user is getting deeply impacted by this. I've been impact, I've been in like eight or nine data breaches and I have like, I have felt the impact of that. I have stories that absolutely would terrify you of how my data was weaponized against me because it got leaked in a data breach. And this is real people getting impacted by these data breaches. Like we companies are tasked with protecting, keeping people's private data private. And if you don't protect the keys to the kingdom, you're not going to be able to do that 100%.

So I've written down something, a secret sprawl, maybe Q. Explain that a little bit more. And like, who's responsible for that? Is it the whole team? Is it the DevOps team? Is it the security team who needs to be aware of secret sprawl and how to like contain it?

Yeah. So I'll first explain the problem, then I'll explain who's responsible for addressing it. Think about secret sprawl as an equivalent is like imagine the days before Notion, Google Docs or any of those, any other online editors. And you had a word doc, I had a word doc and everyone in the company had tons and tons of word docs sprawled all over the place. Some were in email, some were in Slack, some were on your computer. And there was no source of truth. It was impossible to answer. Where are all the company's documents? Who has access to those documents? When were those documents last accessed? Where are all the important information in those documents? Like all those questions you can ask or answer because you didn't have a central place where all the documents are managed. Now we have things like Google Drive, we have things like Notion, where it's very easy for an organization to point you and say, all of our documents live here and we know exactly who has access to them when they were accessed. Last and what's in them. And so sprawl is the same thing with secrets. And remember, one secret is the key to an entire system, right? And companies have thousands of these secrets sprawled across developer, laptops, infrastructure, email, Slack, a bunch of insecure places like email and Slack, where they really shouldn't be code as well. And it just makes it really impossible to answer these like fundamental questions. Like if you have a bunch of keys to your digital kingdom, you better well know where those keys are, right? And who's accessing them. It's like very important thing. Like a good example of this is like if you walk into a building and you have that fob or that badge that scans you in the building that's there on purpose. Because now you can gate access, you can audit every single time someone scanned in for work and you can say this is an intruder, this is a, an authorized employee. The same thing needs to happen in our digital systems. Now who should be responsible for this? Typically for companies that are series A and below it is a CTO. It's a small company, you probably don't have a DevOps or security lead. So it is a CTO's responsibility. Once you get to Series B plus, usually it's owned by the DevOps team. And there are some companies where they have DevSecOps, which is a mix of DevOps and security and that team will own it. It is usually the team that has central authority over infrastructure because this secrets are critical infrastructure. They are the keys to that infrastructure. And so those are the ones that are strongly responsible. And then what they do is they, they set up the tooling and then they roll out the tooling and they set a policy that say you must use this tooling. And a really easy way to do this is if you build it part of all, if you integrate a secrets management part of all the other infrastructure you have, then there's only one right way to do that. All the infrastructure that you have is reliant on that system.

Love it. So when do people start seeking out a solution? Is it after they find they have unmanaged secret sprawl, or is it after they've already done it, they've already been burned? Or like, how do they find that, oh, not only is this a problem, but there's actually solutions out there that I should seek out?

We usually see three separate cases. So the first case, the most obvious one is a data breach just happened and they had that oh fuck moment where they realized that like a data breach is going to be far Far more expensive than whatever we charge on an enterprise license, which is like sub like fifty to a hundred K. Somewhere in that range. Like when you have a data breach, it's gonna cost you millions minimum and maybe even a lot more if you get a class action lawsuit against you because of straight up negligence. Right? Keys to the King. You're not managing the keys. Clear lawsuit case. In my mind. The second is customers that are on AWS Secrets Manager. So they basically have like put all of their most important secrets in AWS or GCP and then they have secrets for all for everything else. And at some point either two cases happen, they realize that, oh shit, we actually have production secrets on developers laptops because they're testing production in development. And so their whole security posture of the most important secrets are secure is not true anymore because now they're sprawled out everywhere and they realize that their solution actually wasn't a viable solution. That happens very, very frequently. I'd say that's a big, big bulk of our customers. The other is productivity, where they realize that because secrets are floating all over the place, they don't really know where all the secrets are. And so they can't centrally manage them. They can't say, engineer, you just started working, you get these set of secrets. Now they have to go hunt down those secrets. Or an engineer adds code that requires a new secret and they have to go hunt down the person who added that code and make sure that secret gets into the infrastructure before the code gets rolled out. Or else you have an outage, you have a missing secret. And so that's another one where like the operations around Secrets management, we call Secret Ops is missing from their workflows they need. Just like how you have workflows for rolling out and managing code, right? You have a pull request that says, we've reviewed this code, it looks good, I'm going to approve it. And then when you click the approve and merge button, it gets rolled out into your infrastructure and your infrastructure restarts with that code. All those exact operations from approval flows to rollouts, all of that needs to happen with secrets and actually needs to happen ahead of code, right? It needs to be there before the code lands in the infrastructure. And so they start seeing those pain points too. And they're like, yes, this is a real problem that we need to solve.

Nice. So how many of these data breaches or secret leaks are preventable? I always hear the saying, hey, if you're going to be hacked, if someone wants to hack you hard enough, bad enough, you're going to be hacked. How much are these preventable, though?

I don't want to say 100%, because that's just not true. Like, if Russia and China dedicate all of the resources they have, they're incredibly competent at this, they will be able to breach you in some way. But I would say for 99% of the market, that isn't a big target. If you have the walls up just decently high, you're going to be preventing most attacks. And so I'd say for most of the market, it's like 95%, 90 to 98%. It is very, very preventable. And there's a very specific set of things you need to do. And if you do those things very well, which most really good secrets managers like Doppler, others. And when I say Secrets Manager, I don't mean the cloud provider only Secrets manager like aws, GCP and Azure. I mean a holistic secrets manager that manages secrets from development to production, every environment in between on every project, and every engineer. If you are using one of those types of secrets managers, you will most likely prevent most hacks.

All right, so we're over 20 minutes into it. I don't think we even mentioned what Doppler is. Maybe a little plug, like how does it work? All right, so someone knows they have an issue, they find us, they do research like, oh, Doppler is the solution. How do they then incorporate it into their organizations and workflows?

Yeah, great question. So at the most pivotal level, or foundational level, Doppler is the central source of truth for all your secrets. It's where across all your projects, all your infrastructure and all your developers. It takes most companies that are small, like I would say seed to Series A, one to three days to install and get rolled out. For medium companies, a couple weeks. And for enterprises, we usually see even the largest of enterprises a couple months, unless they and I that I'm saying of implementation time. Now, of course, it may take a little bit more months longer than that to get prioritized and all that stuff, but the actual implementation time is pretty quick. And so what does it look like once you have the Secrets manager? It means that you know where all your secrets are, you know who has access to them, you know who has accessed them. Last. You have log forwarding into like Splunk, Sumo Logic, gcp, aws, so you can analyze those logs for anomalies and you have a change request flow, which means that whenever secrets change, they get approvals and then they get deeply integrated into Your infrastructure, meaning that they'll get rolled out to your infrastructure, just like code gets automatically rolled out to your infrastructure. And that is like the nuts and bolts. And then one big layer on top of that is any good security professional that's like worth a grain of salt knows that a data breach or a hack is coming. It's not a matter of if, it's a matter of when. Because they are a target. And every day that the company gets bigger, that gets more valuable, they become a bigger target. For a hacker, it's just like simple math. And when that day happens, the most important thing if you cannot stop a breach is to quickly remediate the breach, which basically means stop the attack from happening and then prevent all future attacks from happening. And the way to do that is you swap out the keys. So imagine someone's able to pick this lock or even worst case scenario, they have the key to the lock, right? They've stolen the key. What you want to do is you want to make that key not work anymore. So you want to swap out the lock. So a new key is required and that's called rotation. And when you have a first class secrets manager on all the important credentials, you can basically click one button immediately and it'll rotate every single credential that you have and, and then update your infrastructure to use those new credentials so you don't have any downtime and that's really critical. And so I'll tell you a real story of a customer that came to us. I can't say the name. I guarantee you if I said the name of this customer, you would know them. Most people use their product, they're so big we can't put them on the marketing website as a customer, but that's all. But I guarantee, you know though, they had a malicious actor inside the company, meaning someone was hired that was bad. That actor, that bad actor stole all of their secrets. It was 6,000 secrets. And it got discovered about a couple weeks later. That person got fired. But now the security team had this huge problem. The problem is, well, a bad actor that's going to do bad things with these secrets has access to all of them. So now they're forced to swap out all the locks, they're forced to rotate all the keys. It took their security team six months to rotate all of their secrets with three security engineers dedicated full time to this. So that's six months of wasted energy that's going to someplace that should never have happened. It's six months of exposure where that attacker can keep using those secrets and cost and do an enormous amount of harm in those six months, right? So it's really bad. And now, now that they're on Doppler, they can click a button and they can rotate everything within like, I think it's like two minutes. So it's like you can basically stop the attack dead in its tracks. And that was their primary driver for coming and finding Doppler. And so that's kind of like the big difference there is, like, when a breach does happen, you need to be able to stop it quickly, because if not, it's not just like, oh, we're going to have an immense amount of wasted resources trying to, like, one by one, rotating these secrets. And again, if you don't even know where all your secrets are, that's a really hard problem to solve, Right? You know what, all you may know all the services you're using, but good luck finding all the engineers and all the places those keys are. And so what's going to end up happening and the reason why it takes so long is you need to do it in a way where it doesn't cause an outage. You can't bring yourself down in the process. Like, your code is strongly reliant on these secrets and if you start swapping out the keys, you're going to bring yourself down unless all the infrastructure is updated to use those new secrets, to use those new keys.

So sometimes developers go rogue. They do things quickly without going through red tape. How do you keep track of when someone creates a secret? Like, yeah, oh, we have a tool. Great, I'm not going to tell him. I don't care. Like, I don't know if that's common or not, but, like, how do you even then manage the situation where we have a solution? Use the solution to hold your secrets.

So I don't think you can force developers to use a secrets manager. That's not possible. And I also think it's the exact wrong way to look at it. The right way to look at it is it can't feel punitive to them for using a secrets manager. If they feel like there's a cost that they're bearing by using a secrets manager, like, they lose productivity, then they are definitively not going to use it. They're going to work around it, even at the cost of security. So this is where I think one thing that makes Doppler extremely unique in the marketplace is we, from day one knew this and so we had a different way of looking at the problem we thought, or the way we approach everything in Doplant. It's built into our DNA is to make vegetables taste like candy. Vegetables being the security, candy being the developer productivity. We want developers to save one to two hours per week by using Doppler. They are genuinely more productive by using the security tool. And so they want to use it, they're encouraged to use it. And it prevents a whole bunch of like the worst thing a developer can have is they're developing. All of a sudden they pull down another copy of code and now they're banging their head against the wall because the code doesn't work anymore. All of a sudden. And they're like, shit, why is this thing erroring out? And then they realize, oh, it's because I'm missing a secret in my env file. Now I have to go find the random engineer in this company of 500 engineers and figure out who added that secret so I can keep working again. And this isn't something that I was even responsible for. And now my code is broken and I can't do my job right. We solved that problem day one. If you're using Doppler and, and you should have access to those secrets, the second an engineer adds that secret to their config, they can promote it to roll it out to the rest of the engineering team in one click. And so very easily you kind of have this Dropbox effect of like just by the fact that a secret got added to some to the project you're working on, you immediately have access to that secret. There's no hunting down that person or updating your EMV file. EMV files are now gone. We have a whole other model that's far more secure and real time sync with your entire team in the right way. And so because they're more productive using this tool, they naturally want to use it. And then the company gets all the security benefits from them using it. So you have to allow incentives.

Awesome. Gotcha. So just I had a random thought. I don't know, why is it a good practice, like you just change your keys rather than waiting for them to be hacked? Like once they're in a system every week, do you just. Can you randomly just change them and they'll always fresh, always being regenerated so you can even kind of before anything even happens, you know, someone does anything, it's already changed.

Strong. Strong, yes. So this is called automatic rotation or automated rotation. And the way companies usually do it is they'll set up a policy that says once a month we are going to rotate our Credentials in practice, how this works for any developer who's like whoa, that sounds kind of scary to like rotate our credentials once a month and maybe have an outage. We require a two key system in Doppler. And the way that works is there's always a primary and a secondary. And so what will happen is let's just say you're rotating every 30 days at the 15 day mark the secondary gets removed and a new secondary gets added and that becomes the primary and so the previous primary becomes a secondary. And so then there's another 15 days for the infrastructure to stop using the previous primary, the new secondary and then 15 days later that secondary becomes the new primary once it gets swapped. And so in that case you never have an outage, it's outage free, guaranteed. And you have a guarantee that every 15 days you have a new set of credentials and every 30 days you have a full set of credentials that are, that are new.

Very cool. Once again, could be a dumb question. So AI, you said, you know, hackers are constantly changing, they're constantly using you. Now AI has been introduced, they could probably do other crazy hacking. If someone's using that solution like Doppler, is it aware of maybe known hacks then that have been introduced and then it's able to do like kind of ways to avoid that being hacked. So if someone uses an AI been discovered, you build it into the test your system so that if someone tries that same attack, it's not going to be effective. I don't know, I'm just thinking all out.

Yeah, I think that's a good question. I think the short answer is yes, we don't, we, we are very strongly aware of AI hackers. We're very aware of the, the attack vectors that they normally create. And we're actively putting mitigations in place constantly to make sure that those attackers cannot, you cannot attack us. But, but I think most of the time what you'll find is that if the walls are like a certain level of height, which like a bear, a secrets manager can provide you good example. Secrets are centralized, they are encrypted by default. They are encrypted at rest and in transit. Secrets never touch disk. And you have very strong access controls and you have a very strong auditing story with alerts on those auditing and you have rotation constantly. If you have those set of things, it's very likely that the way the attack is going to happen is not actually through hacking your software, it's through social engineering of some type. They will email you or call you and convince you that they're the CEO and try to convince that engineer to do something wrong. And where this really becomes scary is when you don't have a secrets manager and you don't have that in place, then it's a free for all. Everyone has access to all the credentials. And so a developer may have access to the production environment on their laptop through that production env file. And so when a social engineering attack happens, they get access to all the secrets. But where the story gets different is when you have a secrets manager and you have the right auditing and the right permissions, it gets very hard for a developer to get tricked into giving over the production credentials because they most of the time never have access to it in the first place. And if they did, it's very ephemeral and they lose access immediately once they shouldn't have access anymore. And so a lot of times with these AI attacks, at least right now, it's not really like they're hyper sophisticated attacks that find a zero day. It's more like they have very sophisticated social engineering attacks and they take advantage that because there's no access controls in place at all, no auditing story and no ability to like have your secret centralized, they can very quickly just get get someone to actually forward them an email that had secrets in it or scan a repo for secrets that should never had secrets to begin with, things like that.

Gotcha. So I used to work for large companies, insurance and healthcare and a lot of times we get audited. If there was ever a data breach, you'd be in trouble. So is a solution like Doppler, like required and mandatory for some industries that people may not even know, like oh we could be, if we have a data leak we could lose this amount of money. So this is a no brainer to get this tool or having that audit feature. If you do get audited by the fda, you go hey look, we manage our secrets here type deal.

Yeah. If you're in fintech, health tech, if you're FedRamp required. If you're selling for the military, it's part of NIST as well and it's getting and I think in probably two years it'll be a hard requirement of SOC SOC 2. Right now it's a loose requirement so you should probably get ahead of it. But I think there's even a more fundamental thing which is outside of the forced requirement from a compliance standpoint. You just got to look at the risk standpoint if you get breached which is an if or sorry, it's a when, not an if. When that happens, the first thing any opposing litigator is going to look at the person who's suing you, they're going to say, did you manage the keys to the kingdom? And if you say, no, we didn't do that at all. We did all these other things, but we just didn't manage the keys. We didn't feel we needed to. That's a very clear sign of negligence. And all of a sudden, now you have a very real lawsuit on your hands that's going to be incredibly expensive and damaging to your brand. It's kind of like saying, hey, we have this key to a door, and we just left the door wide open. We didn't manage to lock it at all because the keys are outed and open. And so I think that's really where a lot of the risk is, is in, like, these massive litigation cases that are going to start happening especially as data breaches start getting more and more. They happen more and more frequently. It's like. Like if I'm a litigation lawyer right now and I, like, I literally live off of litigation cases, I would bet there's a whole fleet of litigation lawyers that are looking at this as a very juicy market of, like, every time a data breach happens, I'm going to sue these guys until the sun comes down. Because it's easy money, right? We know very strongly they didn't do the most obvious thing to fucking do. And because of that, they put their customers at risk. Then the attackers exploited that risk, and boom, now you have a very open and closed case.

Awesome. So I may be changing gears here, but I have my notes. I forgot to follow up. What is the Chipotle sales technique?

Yeah, it's. So I don't know how effective it would be today, but what we did is we basically took anyone who would listen to Chipotle and they would be have to be an engineer, and I would tell them, you can get anything you want on the menu, I'll pay for it personally. In exchange, I get a rant at you for two hours about the problem.

Okay.

And that got us some set of customers. The reason why I wouldn't recommend it is because we did talking instead of listening, and that costed us about a year of mistakes. I think if we listened more and talked less, we would have been far more successful faster earlier on.

I guess that would apply to anything a developer does. Listen to the customer before they stop building. So what do you do? You get them on a call and just ask an open ended question, just sit back and record it.

I think it'd be very valuable to actually do it in person. I think build relationships. That way you'll get a lot more candid conversation than you would over Zoom. There's still something that feels fake about being on Zoom. I feel like you just get less genuine conversation and it's more like to the point so they nobody wants to be on a Zoom call. So they hop off. Everyone's cool to just like eat and chill like and lounge around at Chipotle. No one's in a rush to leave that place. And so I would still do it in person, but I would just ask a lot more questions and I would really, really dig deep into the pain points and like why these are pains? Are these pain points that are vitamins where if like I don't have it, nothing really happens or is it like if I don't do this, my hair is going to start falling out and I'm going to have sleepless nights.

Great advice. Okay, Brian, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps secrets efforts? What's the best way to find contact you and learn more about Doppler?

Yeah, I'd say use a Secrets manager. I would. I'm biased. I would recommend Doppler. But you don't have to use Doppler. My job here is not to sell you on using Doppler. It's just tell you that there is a real problem that you are required to solve and you should go solve it because you're putting real people at risk. If you want to use Doppler, you can go to doppler.com, we have a free trial or free account that you can use up to, I think it's like three developers now, maybe four. And then after that once you kind of go into like I am a funded company that has like a team of engineers, you basically have to start paying, which I think is a pretty fair gig. And even then it's very cheap and affordable. So check out doppler.com you can even go to our blog and read more about what I've been talking about. And then if you still feel strongly against using Doppler, that's fine by me. There's tons of other great players in the space. But if you want to check out Doppler, it's Doppler.

Com and for links of everything of value we covered in this DevOps Toolchain Show. Head on over to test guild.com p163 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed with creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community at Test Guild where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With Lutes and Liars the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.